Benchmarking Amazon EC2 with GHC

My personal computers are pretty old and/or slow. I have an old PowerBook G4 and a newish EEE PC. The PowerBook was top of its class when I got it with most options maxed out. Alas, that was five years ago. The EEE PC is by definition not a top performer, nor does it try to be. I find that when the two machines perform similarly in day-to-day tasks, at least when the EEE PC is in “Super performance” mode.

Truth is that for most of the mundane stuff I do these two machines perform acceptably. I’m obviously not going to be watching any 1080p movies on them, or enjoying the latest games (from a productivity standpoint I’m not sure these are such bad things), but most everything else works fine.

Where I feel the performance does hurt me is when compiling with GHC (or any compiler for that matter, it’s just that GHC is the one I use most). Often I spend way to much of my precious little private developer time waiting for the compiler to finish. This is in stark contrast to the situation at work where I run GHC on a pretty zippy Dell PowerEdge Blade Server.

In the near future I expect to be spending more time developing at home and want to be able to do so more efficiently, preferably on par with the situation at work. I could obviously buy shiny new hardware but being a miser (in case you couldn’t already tell based on my hardware) I’m looking for alternatives that would allow me to avoid or defer a hefty up-front investment. One such alternative I’m considering is to rent compute capacity in the Amazon Elastic Compute Cloud (EC2).

EC2 compute capacity is sold in the form of instances at an hourly rate ranging from $0.10 to $0.80 depending on capacity/performance plus some small-change for bandwidth and persistent storage. While there are a couple of hurdles to overcome in order to leverage EC2 as a development workstation the first thing I want to do is to make sure it is a goal worthy of pursuing in the first place, i.e. will I get the desired performance gains at a reasonable price?

To at least begin to answer this question I’ve done some informal benchmarking of the aforementioned systems, excluding the pricier EC2 instances. All the regular benchmarking caveats apply and to reinforce the unscientificity of it all I’m not going to bother providing complete specs for the systems. Here are the fundamentals:

• Apple PowerBook G4: 1.5 GHz PowerPC G4 processor, 2 GB RAM, 5400 rpm HD.
• Asus EEE PC 900HA: 1.6 GHz Intel Atom processor, 1 GB RAM, 4200 rpm HD.
• Dell PowerEdge 1855 Blade Server: Two single-core 3.2 GHz Xeon processors, 2 GB RAM.
• Amazon EC2 Small Instance: 1 EC2 Compute Unit (1 virtual core), 1.7 GB RAM, $0.10 per hour.
• Amazon EC2 High-CPU Medium Instance: 5 EC2 Compute Units (2 virtual cores), 1.7 GB RAM, $0.20 per hour.

According to Amazon “one EC2 Compute Unit provides the equivalent CPU capacity of a 1.0–1.2 GHz 2007 Opteron or 2007 Xeon processor.”

I figure the results are probably more interesting that the details of the tests so here they are, the systems are ordered by increasing performance which happened to be consistent across the tests:

Benchmark results, times in seconds, shorter times are better.

	astro-tables build	highlighting-kate build	fad test suite
PowerBook G4	292	(848)	28
EEE PC	291	643	18
EC2 Small	171	519	15
Dell PowerEdge	75	260	6
EC2 Medium	55	172	4

The EEE PC was in “Super Performance” mode during the tests and the PowerBook was at its highest CPU speed. All times are the “real” time as measured by the Unix time command and lower numbers are naturally better.

As can be seen an Amazon EC2 Small instance is only marginally faster than the EEE PC. An EC2 High-CPU Medium instance on the other hand is significantly faster than the zippy Dell PowerEdge Blade server. Is either one a good deal? Good question, I think a case could be made either way depending on your priorities but I’m not going to tackle that today.

If you care about the details of the tests read on, if not please move on to your next blog of choice!

astro-tables build

This package currently consists of a single automatically generated 4000-line monster of a module¹. The code is pretty straight-forward but the module takes ages to compile, almost certainly due to me giving the type checker an unnecessarily hard time. I have a trivial rewrite on my todo-list which I expect will shorten the compilation time dramatically but the current form comes in kind of handy for the purposes of this benchmark. The Git repo is git://github.com/bjornbm/astro-tables.git and the commit used in the benchmarking was e63b8978833878526870b2101697197ff64af593. I made sure the dependencies were already installed and ran time cabal install.

highlighting-kate build

From recent memory I knew that John MacFarlane’s highlighting-kate package has a hefty number of modules (the majority of which are also automatically generated) that take a fair amount of time to compile. I downloaded version 0.2.4 from hackage, made sure all dependencies were already installed, and ran time cabal install --flags=executable.

I ran into one snag with this test: the build wouldn’t complete with GHC 6.10.3 on the PowerBook G4 due to some problem with pcre-light (which tends to give me headaches on pretty much every platform). This particular headache² I was unable to resolve and had to run the test using GHC 6.10.1 on the PowerBook G4.

fad test suite

Finally I did a runtime performance (as opposed to compilation) benchmark: running the test suite of the fad library. The Git repo is git://github.com/bjornbm/fad.git and the commit used was cd2965a6741291570930e4bf6e9f8f9ab64ccadd. I ran ghc --make Test and then time ./Test.

---

1. An implementation of the 678 lunisolar terms and 687 planetary terms of the IAU 2000A Precession-Nutation Model. ↩

2. gcc: Internal error: Virtual timer expired (program cc1) ↩

May 2009 HCAR Submissions

Last weekend was the submission deadline for contributions to the May 2009 edition of the Haskell Communities and Activities Report (HCAR). I absolutely love the HCAR; it’s a terrific and comprehensive source of information on all the amazing stuff people are doing with Haskell. I’m very grateful to the editor Janis Voightländer (and Andres Löh before him) for compiling the report, in my opinion he does a great service to the Haskell community. I can’t wait to get my hands on the upcoming edition!

Below are my submissions to the HCAR (free blogging material, yay!) with some additional notes.

dimensional: Statically checked physical dimensions

Report by: Björn Buckwalter
Status: active, mostly stable

Dimensional is a library providing data types for performing arithmetics with physical quantities and units. Information about the physical dimensions of the quantities/units is embedded in their types, and the validity of operations is verified by the type checker at compile time. The boxing and unboxing of numerical values as quantities is done by multiplication and division with units. The library is designed to, as far as is practical, enforce/encourage best practices of unit usage.

The core of dimensional is stable with additional units being added on an as-needed basis. In addition to the SI system of units, dimensional has experimental support for user-defined dimensions and a proof-of-concept implementation of the CGS system of units. I am also experimenting with forward automatic differentiation and rudimentary linear algebra.

The current release is compatible with GHC 6.6.x and above and can be downloaded from Hackage or the project web site. The primary documentation is the literate Haskell source code, but the wiki on the project web site has a few usage examples to help with getting started.

Further Reading

http://dimensional.googlecode.com

Dimensional was largely the project that enticed me to learn Haskell, and is the one that I am most proud of to date. It relies on some pretty tricky type-level hackery, stuff I would never have been able to pull off without the help of many papers ranging from Oleg Kiselyov’s amazing type-hacks to Hudak et al’s A History of Haskell.

Its primary shortcoming at present is the lack of Haddock documentation. Dimensional is written in a literate style and I think anyone reading the source will find the documentation pretty comprehensive and easy to follow (barring the fact that type hacking is pretty tricky business). Alas, with the last few years’ infrastructure improvements and increased (de facto) standardization in the community not having proper haddocks is a wart.

fad: Forward Automatic Differentiation

Report by: Björn Buckwalter
Participants: Barak A. Pearlmutter, Jeffrey Mark Siskind
Status: active

Fad is an attempt to make as comprehensive and usable a forward automatic differentiation (AD) library as is possible in Haskell. Fad (a) attempts to be correct, by making it difficult to accidentally get a numerically incorrect derivative; (b) item provides not only first-derivatives, but also a lazy tower of higher-order derivatives; (c) allows nested use of derivative operators while using the type system to reject incorrect nesting (perturbation confusion); (d) attempts to be complete, in the sense of allowing calculation of derivatives of functions defined using a large variety of Haskell constructs; and (e) tries to be efficient, in the sense of both the defining properties of forward automatic differentiation and in keeping the constant factor overhead as low as possible.

Version 1.0 of fad was uploaded to Hackage on April 3. Recent changes can be found via git clone git://github.com/bjornbm/fad.git

Further Reading

http://github.com/bjornbm/fad
http://flygdynamikern.blogspot.com/2009/04/announce-fad–10-forward-automatic.html

The majority of work on fad (including most of the above report) is actually being done by Barak Pearlmutter, principial of the Hamilton Institute’s Brain and Computation Lab. As you can understand his brain is in a different league than mine when it comes to anything related to computation. My contribution at this stage is mostly reviewing his patches (often struggling to understand them) and providing guidance and support on Haskell infrastructure and conventions.

leapseconds-announced

Report by: Björn Buckwalter
Status: stable, maintained

The leapseconds-announced library provides an easy to use static LeapSecondTable with the leap seconds announced at library release time. It is intended as a quick-and-dirty leap second solution for one-off analyses concerned only with the past and present (i.e. up until the next as of yet unannounced leap second), or for applications which can afford to be recompiled against an updated library as often as every six months.

Version 2009 of leapseconds-announced contains all leap seconds up to 2009–01–01. A new version will be uploaded if/when the IERS announces a new leap second.

Further Reading

http://hackage.haskell.org/cgi-bin/hackage-scripts/package/leapseconds-announced
http://github.com/bjornbm/leapseconds-announced

Not much to add about leapseconds-announced other than that it fills a need I have. I elaborated a little on what it is (and what it doesn’t try to be) in the announcement thread.

That’s it! I hope to have another project or two to report on in the November edition…

ANNOUNCE: fad 1.0 — Forward Automatic Differentiation for Haskell

I’m pleased to announce the initial release of the Haskell fad library, developed by Barak A. Pearlmutter and Jeffrey Mark Siskind. Fad provides Forward Automatic Differentiation (AD) for functions polymorphic over instances of Num. There have been many Haskell implementations of forward AD, with varying levels of completeness, published in papers and blog posts, but alarmingly few of these have made it into hackage — to date Conal Elliott’s vector-spaces package is the only one I am aware of.

Fad is an attempt to make as comprehensive and usable a forward AD package as is possible in Haskell. However, correctness is given priority over ease of use, and this is in my opinion the defining quality of fad. Specifically, Fad leverages Haskell’s expressive type system to tackle the problem of perturbation confusion, brought to light in Pearlmutter and Siskind’s 2005 paper Perturbation Confusion and Referential Transparency. Fad prevents perturbation confusion by employing type-level “branding” as proposed by myself in a 2007 post to haskell-cafe. To the best of our knowledge all other forward AD implementations in Haskell are susceptible to perturbation confusion.

As this library has been in the works for quite some time it is worth noting that it hasn’t benefited from Conal’s ground-breaking work in the area. Once we wrap our heads around his beautiful constructs perhaps we’ll be able to borrow some tricks from him.

As mentioned already, fad was developed primarily by Barak A. Pearlmutter and Jeffrey Mark Siskind. My own contribution has been providing Haskell infrastructure support and wrapping up loose ends in order to get the library into a releasable state. Many thanks to Barak and Jeffrey for permitting me to release fad under the BSD license.

Fad resides on GitHub and hackage and is only a cabal install fad away! What follows is Fad’s README, refer to the haddocks for detailed documentation.

   Copyright  : 2008-2009, Barak A. Pearlmutter and Jeffrey Mark Siskind
   License    : BSD3

   Maintainer : bjorn.buckwalter@gmail.com
   Stability  : experimental
   Portability: GHC only?

Forward Automatic Differentiation via overloading to perform
nonstandard interpretation that replaces original numeric type with
corresponding generalized dual number type.

Each invocation of the differentiation function introduces a
distinct perturbation, which requires a distinct dual number type.
In order to prevent these from being confused, tagging, called
branding in the Haskell community, is used.  This seems to prevent
perturbation confusion, although it would be nice to have an actual
proof of this.  The technique does require adding invocations of
lift at appropriate places when nesting is present.

For more information on perturbation confusion and the solution
employed in this library see:
<http://www.bcl.hamilton.ie/~barak/papers/ifl2005.pdf>
<http://thread.gmane.org/gmane.comp.lang.haskell.cafe/22308/>


Installation
============
To install:
    cabal install

Or:
    runhaskell Setup.lhs configure
    runhaskell Setup.lhs build
    runhaskell Setup.lhs install


Examples
========
Define an example function 'f':

> import Numeric.FAD
> f x = 6 - 5 * x + x ^ 2  -- Our example function

Basic usage of the differentiation operator:

> y   = f 2              -- f(2) = 0
> y'  = diff f 2         -- First derivative f'(2) = -1
> y'' = diff (diff f) 2  -- Second derivative f''(2) = 2

List of derivatives:

> ys = take 3 $ diffs f 2  -- [0, -1, 2]

Example optimization method; find a zero using Newton's method:

> y_newton1 = zeroNewton f 0   -- converges to first zero at 2.0.
> y_newton2 = zeroNewton f 10  -- converges to second zero at 3.0.


Credits
=======
Authors: Copyright 2008,
Barak A. Pearlmutter <barak@cs.nuim.ie> &
Jeffrey Mark Siskind <qobi@purdue.edu>

Work started as stripped-down version of higher-order tower code
published by Jerzy Karczmarczuk <jerzy.karczmarczuk@info.unicaen.fr>
which used a non-standard standard prelude.

Initial perturbation-confusing code is a modified version of
<http://cdsmith.wordpress.com/2007/11/29/some-playing-with-derivatives/>

Tag trick, called "branding" in the Haskell community, from
Bjorn Buckwalter <bjorn.buckwalter@gmail.com>
<http://thread.gmane.org/gmane.comp.lang.haskell.cafe/22308/>

Space News: U.S. Air Force To Widen Access To Detailed Space Surveillance Data

Revisiting my posts on the Iridium 33/Cosmos 2251 collision and data…

The front page article of last week’s Space News (Volume 20, Issue 10, March 9th, 2009) reported that “U.S. Air Force To Widen Access To Detailed Space Surveillance Data” (article quoted in full here). The short story is that:

The U.S. Air Force has agreed to provide wider access to its high-accuracy catalog showing the whereabouts of orbital debris and operational satellites as part of an effort to enable commercial and non-U.S. government satellite operators to better avoid in-orbit collisions, according to U.S. Air Force officials.

The “high-accuracy catalog” would be what I referred to as Special Perturbations (SP) data. The article reiterates the limitations of the Two-Line Elements currently available to the public:

…U.S. Air Force Space Surveillance Network data is published, but only in a form that satellite operators have long said is not useful for space traffic management. This data, called Two-Line Elements (TLEs), has too great a margin of error to permit operators to act.

And says that:

The U.S. Air Force statement suggests that it will furnish more information to the public to enable operators to make a more highly informed decision.

Apparently a policy if forthcoming. The details of the policy are unknown but it should be announced before June.

All this is of course good news and cause for cautious optimism. Why do I say cautious? Here are two quotations from a March 5 roundtable discussion titled Challenges for Space Policy in 2009:

We’re horrible at implementing policy, absolutely horrible.

The problem is not that we have insufficient space policy or a bad space policy, but as we now understand, policy is not self-actualizing. We need to have some other mechanisms that take very sound policy and turn them into action and results.

Let’s hope those mechanisms are in place by June.

While on the topic of the Iridium 33/Cosmos 2251 collision: via Jeff Foust’s spacepolitics.com (good blog, love the tagline) I read that “some have tried to portray last month’s Iridium-Cosmos satellite collision as either a deliberate act by the US or a deliberate act by Russia.” I call bullshit on both of these. The Washington Times editorial is particularly embarrassing in its tin-foil-hattery.

Blogging with Pandoc, literate Haskell, and a bug

I use John MacFarlane’s pandoc for converting my blog entries into HTML. My workflow is roughly:

1. Write my posts vim using (pandoc’s enhanced) Markdown format.
2. Store my posts in a private Git repository on GitHub.
3. Convert posts to pastable HTML using pandoc and paste into Blogger.

pandoc is a great tool. It is a better Markdown than Markdown and also supports a bunch of other markup formats. The feature set is really extensive with plenty of useful options and sensible defaults. Kudos to John and the other contributors.

My previous post on Network.Curl was the first post where I included significant amounts of source code and was thus my first opportunity to exercise pandoc’s syntax high-lighting features. I was fortunate enough that version 1.2 of pandoc was released just days prior to my post with support for literate Haskell. The literate Haskell support meant that I could stop worrying about how to avoid my Haskell code being interpreted as Markdown block quotations (and if I need a block quotation I can always use <blockquote> tags).

By default pandoc will install without support for syntax high-lighting. To enable syntax high-lighting you must supply the highlighting flag when building, e.g. if you are using cabal:

cabal install pandoc --flags=highlighting

This triggers installation of John’s highlighting-kate library which is used for the syntax high-lighting. A snag you might run into here is the dependence on pcre-light which in turn relies on a PCRE dynamic library being available. In my case (on Mac OS 10.4) I had to install PCRE using MacPorts and add /opt/local/lib to $LD_LIBRARY_PATH.

When editing my posts I want to review occasionally and run pandoc with the --standalone option to generate a complete HTML document I can view in my browser. However, when pasting into Blogger I only want a fragment and omit said option. The problem with this is that the CSS style definitions used for the syntax high-lighting isn’t included in the fragment. My solution to this problem was to copy the <style> element from the --standalone output and paste into the <head> of my Blogger template (immediately prior to </head>).

That’s pretty much it. The complete command I use to generate the fragments I paste into Blogger is:

pandoc --smart --to=html+lhs input.lhs

(I pipe the output into e.g. pbcopy or xclip to get it straight into the clipboard.)

The +lhs portion of the --to option tells pandoc to keep the bird-tracks (those ’>’s at the beginning of a line of code). The beauty of this is that readers can copy and paste the entire (as in “Select All”) blog post into a .lhs file and it is valid literate Haskell that e.g. ghc will happily compile. (Well, at least when copied from Firefox. Safari has the nasty habit of omitting the empty line following a code block which doesn’t make for valid literate Haskell… I’ll be investigating work-arounds.)

When writing my previous post I did notice one discrepancy in the syntax high-lighting. The bird-track on Any line following a end-of-line comment (--) wouldn’t be correctly high-lighted. Example:

> -- | Additional options to simulate submitting the login form.
> loginOptions user pass = 
>   CurlPostFields [ "login=" ++ user, "password=" ++ pass ] : method_POST

Depending on your screens calibration it might not stick out like a sore thumb but the middle bird-track there is black while the other two are blueish. This was due to a bug in the syntax high-lighting definition file for literate Haskell (literate-haskell.xml) in the highlighting-kate package currently on Hackage (version 0.2.3). This is no fault of John’s as that file came straight from the KDE subversion repository. A couple of hours ago I submitted a patch to John as well as the KDE developers (who have already applied it to the their repo).

If you can’t wait for John to apply the patch to the highlighting-kate repo you can download it from darcswatch (it’s the one labeled “20090314201157”), apply it to your local copy of John’s repo, cabal install, and finally cabal install pandoc --flags=highlighting --reinstall.

With the patch applied the bird-tracks will render properly:

> -- | Additional options to simulate submitting the login form.
> loginOptions user pass = 
>   CurlPostFields [ "login=" ++ user, "password=" ++ pass ] : method_POST

Update 2009–03–16: John uploaded highlighting-kate 0.2.4 (with patches applied) to Hackage today. Reinstall with cabal (don’t forget to cabal update first).

Extended sessions with the Haskell Curl bindings

I recently needed to automate retrieving protected data from a secure web site. I had to:

1. Log into the website with a POST request.
2. Download the protected data with a GET request.

All this had to be done using SSL, and I suspected I’d need to handle cookies too.

I had read that libcurl had support for sessions and cookies spanning multiple requests, and knew that it could handle SSL. I was aware there is a Haskell binding to libcurl (aptly named “curl” but hereafter referred to as Network.Curl to avoid confusion) on Hackage so I had my tools cut out for me. While I had used the command line curl utility quite a bit I had never programmed against libcurl before and had some learning to do.

It wasn’t entirely clear to me from the haddocks how to use Network.Curl. This may not be a problem if you are familiar with libcurl already (I couldn’t tell) but for me it was quite a hurdle. Googling on the topic I found some blogged examples that got me started but I was unable to find an example demonstrating a multi-request session. However, with the basics from the blogs I was able to return to Network.Curl and figure things out by inspecting its source code. I’ll share an example here for the benefit of others who find themselves in the same situation. I’m using version 1.3.4 of Network.Curl.

As a contrived example let’s assume we want to write a small program that, given a user name and password, fetches the user’s API token from GitHub. Here is the code (literate Haskell, just copy and paste into a .lhs file):

> import Network.Curl
> import System (getArgs)
> import Text.Regex.Posix

> -- | Standard options used for all requests. Uncomment the @CurlVerbose@
> -- option for lots of info on STDOUT.
> opts = [ CurlCookieJar "cookies" {- , CurlVerbose True -} ]

> -- | Additional options to simulate submitting the login form.
> loginOptions user pass = 
>   CurlPostFields [ "login=" ++ user, "password=" ++ pass ] : method_POST

> main = withCurlDo $ do
>   -- Get username and password from command line arguments (will cause
>   -- pattern match failure if incorrect number of args provided).
>   [user, pass] <- getArgs

>   -- Initialize curl instance.
>   curl <- initialize
>   setopts curl opts

>   -- POST request to login.
>   r <- do_curl_ curl "https://github.com/session" (loginOptions user pass)
>     :: IO CurlResponse
>   if respCurlCode r /= CurlOK || respStatus r /= 302
>     then error $ "Failed to log in: "
>                ++ show (respCurlCode r) ++ " -- " ++ respStatusLine r
>     else do
>       -- GET request to fetch account page.
>       r <- do_curl_ curl ("https://github.com/account") method_GET
>         :: IO CurlResponse
>       if respCurlCode r /= CurlOK || respStatus r /= 200
>         then error $ "Failed to retrieve account page: "
>                    ++ show (respCurlCode r) ++ " -- " ++ respStatusLine r
>         else putStrLn $ extractToken $ respBody r

The first thing to note is that we use do_curl_ rather than e.g. curlPost and curlGet. The latter two don’t actually give you access to the response body but instead prints it on stdout! The general process is:

1. Initialize a curl instance.
2. Set options.
3. Call do_curl with the URL and request-specific options.
4. Inspect CurlResponse
5. Repeat from 3 until done.

Note that the all used of libcurl should be wrapped by withCurlDo. In the example I wrapped the entire body of main. Also note that the type of do_curl_ must be specified explicitly unless it can be inferred by later use. The CurlResponse type specified above uses vanilla Strings for everything.

For the POST request I added some CurlPostFields to the method_POST options predefined in the Network.Curl. For the GET request’s options since the predefined method_GET is sufficient.

A GitHub-specific peculiarity here is the error checking after the POST request. GitHub returns a 302 (“Moved Temporarily”) on successful login and a 200 (“OK”) when the credentials are bad. Stuff like this needs to be figured out on a site-by-site basis.

For completeness here is the function that extracts the token from the response body using a regular expression:

> -- | Extracts the token from GitHub account HTML page.
> extractToken body = head' "GitHub token not found" xs
>   where
>     head' msg l = if null l then error msg else head l
>     (_,_,_,xs)  = body =~ "github\\.token (.+)"
>                :: (String, String, String,[String])

If you Load this code in ghci and type :main username password the Octocat will deliver your token.

Programmer Man

While I haven’t tried pair programming I’m intrigued by the idea and would like to give it a shot some time. Rod Hilton recently blogged about his experiences with pair programming and despite vim (+command line) being my primary development environment I have to concede that it makes an interesting as well as entertaining read. As William Pietri noted in the comments the “insights on Programmer Man are both novel and keen” and well worth pondering even if you have no interest in pair programming per se.