Exploring Amazon S3 with F#


Traditional imperative languages teach developers that program code requires ceremony. You can’t just write a line and expect it to do something. When programming book writers explain how easy is it to display "Hello world" string in their favorite language, they actually start with an explanation that every call should be wrapped in a method, and the method typically takes some arguments etc. And of course they have to let beginners know how to compile the resulting program and how to execute it. Without all these steps the world won’t get its greeting.

All these steps have valid reasons, but as a result development environment based on major imperative languages does not really encourage exploration. It’s best suited towards large scale development, but if a developer wants to have a quick look on something new, and he can’t proceed without creating a new project, then he will often postpone this work until he has enough time or enough reasons to go ahead.

Luckily, there’ve always been alternatives to languages like C# or Java that worked better for rapid prototyping or technology exploration. One of the late additions is F#, and when I recently introduced myself to Amazon Simple Storage Service, I realized that if I need to get a quick overview of its programming facilities, using F# would be much more efficient than if write a test program in C#.

The purpose of this article is to show how fast you can dig into low level details of an unknown technology using F#, and how little code you need to write. Therefore I keep the text of the article short, focusing just on the required steps. If you need more information about F# or Amazon S3, there’s plenty of books and articles available. Just google for them. Instead of numbering steps that I’ve gone through, I’ll provide timing information: I started the exercise at about 17:00 CET on July 15th. It took me about 45 minutes to display a picture extracted from my S3 bucket. And all code was written in F#.

17:00. Getting Amazon Web Services SDK

Amazon Web Services SDK, also known as AWS SDK is available at http://aws.amazon.com/sdkfornet. Once downloaded, it’s installer copies binaries and samples, and optionally registers AWSSDK.dll in the GAC.

17:10. You can’t go wrong with Reflector

Before firing a Visual Studio F# session, I checked the content of the installed binaries and loaded into Red Gate .NET Reflector the only .NET assembly that I found: AWSSDK.dll. I could probably manage without it, but it would take longer time. After I created an S3 client session, I mostly relied on Intellisense support for F# in Visual Studio, but spending few minutes looking at namespaces and classes hinted me about entry points. For example, I assumed that I would need to create an instance of AmazonS3Client and probably use classes from Amazon.S3.Model.


17:15. Opening F# Interactive session

I started Visual Studio and immediately was able to use comparative advantage of F# environment. I didn’t have to create a project – in fact I didn’t know what kind of project I would need to create if I had to: should be a console application? Or maybe a Windows Form program? I didn’t know yet what to expect, all I wanted to get from an IDE is a notepad where I could write code lines and execute them one by one. So F# was a very good fit for my intention: I created a new F# script file and started sending individual lines to an FSI window. The first lines referenced AWSSDK and imported Amazon.S3 namespace that I expected to be useful (thanks Reflector):

open Amazon.S3

Later I imported another namespace (Amazon.S3.Model), but these two lines above were enough to connect to Amazon S3 storage.

17:20. Connecting to Amazon S3 and accessing a bucket

The fun part began. My guess was that since AmazonS3Client class has a constructor with two string parameters, this is the one I should use and assign parameters to AWS key ID and secret key. F# Intellisense confirmed it. So here’s my first call to Amazon S3 in F# (I removed actual access key values).

let s3Client = new AmazonS3Client("KEY-ID", "SECRET-KEY")

After a few seconds F# Interactive came with a response:

val s3Client : AmazonS3Client

Now I had a mighty client with many interesting methods to try. The next logical step would be to list available buckets. I had only one, so I could retrieve only the first element of the list.

let response = s3Client.ListBuckets()

Here’s an FSI output:

val it : System.Collections.Generic.List =
    [Amazon.S3.Model.S3Bucket {BucketName = "abilov.com";
                               CreationDate = "to, 15 jul 2010 10:17:18 GMT";}]

Great! This looked like a real thing. Encouraged with a quick success, I wanted to start retrieving the bucket contents.

17:25. Failed attempt to retrieve bucket objects

The bucket collection was wrapped in a sequence, so I only needed to get it’s head to obtain the only bucket I had at S3:

let bucket = Seq.head response.Buckets

I thought in order to retrieve bucket objects, I had to send the obtained bucket to some method, but I was wrong. Actually I didn’t need to obtain a bucket at all – since I knew the name of the bucket, I could construct a ListObjectRequest from the name and send it to a ListObjects. But I didn’t know about it 5 minutes ago. Anyway, here’re new calls, to retrieve bucket objects – data files with images from our home photo gallery.

let request = new ListObjectsRequest(BucketName = bucket.BucketName)
let objects = s3Client.ListObjects(request)

Bang!!! Exception!

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)

(The rest of the call stack is skipped)

Now when I am writing this article, I am in fact glad that I had this problem. If everything went smooth, you might look at my timing information with a grain of disbelief: the guy probably knew what he was doing. No I didn’t! So I had to check around. Since the error came after so basic steps, I hoped that I wasn’t alone who experienced it, and luckily I was not. Several people complained at Amazon forum, and they were recommended a simple workaround in case they would accept using HTTP instead of HTTPS. And for the purpose of exploration it was just fine.

17:30. Connection to Amazon S3 revisited

Here’s an updated version of connection code, taking now two lines instead of one.

let s3Config = new AmazonS3Config(CommunicationProtocol = Protocol.HTTP)
let s3Client = new AmazonS3Client("KEY-ID", "SECRET-KEY", s3Config)

Then I replayed the other two lines requesting bucket objects. And now I successfully received results:

    {AmazonId2 = "//VkK6Mb7Xe26pNPG7nt40GkYcp4GDyNV45I1E1mfZVCL67aak9/0Txmrk8X+JAE";
     CommonPrefix = seq [];
     CommonPrefixes = seq [];
     Delimiter = null;
     Headers = seq
                 ["x-amz-id-2"; "x-amz-request-id"; "Transfer-Encoding";
                  "Content-Type"; ...];
     IsTruncated = true;
     MaxKeys = "1000";
     Metadata = seq [];
     Name = "abilov.com";
     NextMarker = "private/photo/2005/2005-06-10-12 Elverum/IMG_6803.JPG";
     Prefix = "";
     RequestId = "3A27D1A4495A747A";
     ResponseStream = null;
     S3Objects = seq {…}

     (The rest of results is skipped)

17:35. Retrieving object data

I inspected the response and found the property that I needed: S3Objects. It was a collection of bucket objects representing the photo gallery files I uploaded to S3. Taking a further look at available AmazonS3Client methods I’ve found a method that I should use to retrieve an individual object: GetObject. It took an instance of GetObjectRequest. I first created a GetObjectRequest instance without assigning a Key property. But after receiving exception with error message about missing a key I corrected my mistake. Here’s a code that worked:

let objectRequest = new GetObjectRequest(BucketName=bucket.BucketName, Key="private/photo/2005/2005-01 Kolbotn/IMG_5645.JPG")
let obj = s3Client.GetObject(objectRequest)

The "Key" property is just a file path within the bucket, so it’s easy to figure it out.

I sent "obj" to an FSI window, and it printed it’s properties:

{AmazonId2 = "zUH8IaRglPSPQuC+8m5pzAR59paJTi/L5M1DnxNympp9HcU5RigrntS7NGvpxQQf";
     ContentLength = 1138630L;
     ContentType = "image/jpeg";
     ETag = ""ea02192ee842d3b1987a8de4ac879595"";
     Headers = ?;
     Metadata = seq ["x-amz-meta-cb-modifiedtime"];
     RequestId = "A93CF70A966E4A56";
     ResponseStream = System.Net.ConnectStream;
     ResponseXml = null;
     VersionId = null;}

One of the properties was called ReponseStream. This sounded very promising – perhaps I could try to create an image out of this stream and display it in a Windows Form?

17:40. Displaying an image

What should you do if you want to display an image in a Windows Form in C#? You have to start from creating a new project using Windows Forms Application template. Then you can edit form class and add an Image control to it in a Form designer. How do you do it in F#? You simply create a form with an image right from the script. Here’s the code:

#r "System.Drawing"
#r "System.Windows.Forms"
open System.Windows.Forms
open System.Drawing

let img = Image.FromStream(obj.ResponseStream)
let frm = new Form(ClientSize = img.Size)
frm.Paint.Add(fun e -> e.Graphics.DrawImage(img, new Point(0,0)))

Note that first four lines are just to reference required DLLs and import namespaces. The rest of the code is equally short and complete the task. Look at the Image.FromStream call: I wasn’t quite sure I could do this, but the property name "ResponseStream" was too tempting not to give it a try. And suddenly I saw this window:


This is our house cat Figaro (unfortunately not with us anymore). And no, I didn’t specifically select this photo for my test. This was a random choice, perhaps hinting about what kind of pictures some families tend to take.

17:45. Done!

Yes, we are! Perhaps just to show my environmentalist attitude, I’ll make one last call:



So it was three quarters of exploration. Completely new API, combined with my relatively poor experience with Amazon S3, but a picture from a family photo gallery retrieved from an S3 bucket and shown in a Windows form proves that we managed the full stack of operations. All done in F#. This experiment does not expose other (and far more important) language qualities, but I deliberately wanted to focus on a different topic: language efficiency for the purpose of technology exploration and rapid prototyping. Being .NET language, F# can access any .NET area or feature, and its REPL support ensures that developer’s time is spent in a most optimal way.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s