Ever since Amazon launched AWS in 2006 cloud computing has been a hot topic of discussion. Startups are pretty much expected to host their infrastructure in the cloud, because it is considered scalable, reliable, and most importantly requires no upfront capital. In my previous startups (Exchange3D and URMyStar) we also used the cloud for these exact reasons. Similarly when we were ready to test Citybot in real life, I spawned an instance on Amazon EC2 and the server was configured and running in less than half an hour. I accepted the cloud as a de facto optimal hosting option so much that it has not even occurred to me to buy or rent a server rack.
Last year we were lucky to win a year of free hosting from Rackspace at http://www.pitchevent.com/ and happily moved our infrastructure from Amazon to Rackspace to save money. As we kept developing and testing Citybot and kept loading the server with more and more data to process I noticed that calculating the same itinerary on the largest Rackspace instance takes 4 times longer than on my MacBook Air.
Looking for a faster server to use during demos and investor pitches, I tested our application on Amazon and Joyent. The performance was better than Racksace, but still slow. Joyent guys were even so nice as to let me use their servers for free for about a month while I was running performance tests and optimizing the code.
Finally, when our free Racksapce service ran out I gave up and rented a dedicated server on LiquidWeb, which was 4 times cheaper than a cloud and twice as fast. Few weeks later I found out about a hybrid StormOnDemand offering from LiquidWeb, which offers both the power of dedicated machines and the flexibility of the cloud at the same time.
To make sure I am choosing the best solution out of all that is available, I decided to rerun my tests again on each of the platforms that I had access to. The following are the results of testing performance of virtual and hybrid instances from 6 different providers:
Each of these providers offers a bunch more services than just virtual or hybrid machines and if you are shopping around for cloud storage, CDN, hosted database, and other IAAS there are quite a few other factors to consider. The results presented below concern only the CPU performance of virtual and hybrid machines instances.
CPU Performance
My tests were performed using our application code. While this is probably not an overall fair performance test, it is the most relevant test for Citybot, since it tests how fast the machine can solve a complicated problem. Since each provider has a different set of options and I could not match them exactly, on each platform I chose the smallest instance that has at least 8 CPU cores. Amazon EC2 offers two 2 types of high-CPU instances with 8 cores: c1.xlarge with 7GB RAM and m2.4xlarge with 68GB RAM. CloudSigma does not have pre-packaged instance types, therefore I created an instance with 8 cores and matched other resources to c1.xlarge EC2 instance. On Rackspace, the only instance that offers 8 cores is the largest one, which comes with 30GB of RAM. Similarly on Windows Azure – only the largest instance type has 8 cores (only 14GB RAM though). On each platform I allocated only 6GB of RAM to the Tomcat instance that was running my application.
Joyent’s claim is that their cloud performs much better at lower cost than Amazon. In our case, Joyent’s performance turned out to be almost as good as EC2, but the cost is indeed much cheaper. You can download Joyent’s own benchmarks here: http://bit.ly/XnSVoG
Flexibility
Amazon EC2
Amazon has more instance types, which gives more room for selecting the amount of resources that you need. This is important when developing and testing, but not as important once you are in production and know what your resource needs are.
Rackspace Cloud
Rackspace has a fair selection of instance types. For us the biggest problem was that their instance selection is linear, i.e. you cannot have more CPU without paying for more RAM. If you want 8 cores, you have to get 30GB of RAM and pay $876.6/month.
Cloudsigma
Thanks to The 12th Founder Showase, we have 6 months of unlimited hosting from CloudSigma. CloudSigma is the most flexible when it comes to tailoring your resources exactly to your needs. CloudSigma has a truly flexible cloud and lets you piece together any configuration you want. Each component is priced separately and you can combine them in any way you want in any amount you want. If you need only 12GB of SSD storage – you can do that and have the rest of your storage on regular drives. If you want an odd number of GBs for RAM – you can do that too. Need 17 cores with 28.5GB RAM – no problem. Assembling instances on CloudSigma is like playing with Legos.
Windows Azure
First of all Kudos to Microsoft for having Linux instances! Overall Azure cloud service offering is very much like Amazon. The only downside is a limited selection of instance types. For some reason the largest instance is only 8 cores/14GB RAM.
I should also mention Windows Azure management portal, which is super nice and unlike the rest of Microsoft’s websites works great in Chrome and Firefox on a Mac.
Joyent
Joyent has a bigger selection of instances types than Azure or Rackspace, but less than Amazon. One critical feature that stopped me from using Joyent for development environment was lack of ability to quickly create custom images of my machines. As far as I can see in the documentation on Joyent.com, if you want to create a custom image you still have to go through a 9 step process culminating with sending a support request. Alternatively you can use Chef to script your machine images. This might not be a big deal for some customers, but for me this was a deal breaker given how simple imaging is on all other platforms. As they say: “laziness drives progress”.
Storm On Demand by LiquidWeb
LiquidWeb has by far the most options of server and instance types. StormOnDemand.com owned by LiquidWeb offers a choice of several dozen different types of servers in 3 categories: Storm, Storm SSD and Bare Metal. With all these options this makes them almost as flexible as CloudSigma. I was testing only the Bare Metal servers since these promise the highest performance while costing as much or cheaper than virtual servers
Support
Overall, each of the providers except Amazon has great support included with the price of the service. In Amazon’s case, you either have to pay for additional support or you are on your own.
Rackspace Cloud
Rackspace support is fanatical, alright. All the techs are very polite, eager to help and highly qualified. However, the problem is that I had to contact them all the time. I would think that about 20% of the time something does not work on Rackspace. Whether I am creating a new machine, scheduling a backup or resizing a machine – every once in awhile I get the cheeky “oops” popup (in the old panel) or the red “Error” (in the new panel) saying that something did not work. When that happens, I have to start by talking to the online rep, who would usually not have enough knowledge or authority to resolve the issue. The scenario almost always would go like this:
1) Try to do something via control panel and get an error
2) I start a chat with online support
3) In 5 – 30 minutes online support figures out that they cannot help me and open a ticket
4) In 1 – 6 hours the ticket gets resolved
On other platforms that I tried everything just works and I almost never had to bother support.
Windows Azure
Microsoft does not make a claim of heroism or fanaticism. However Windows Azure support is awesome! My tickets got resolved within minutes even though I was not a paying customer (Citybot is in BizSpark Plus thanks to The Founder Institute and Microsoft) ! As a side note, I cannot help but notice how much Microsoft is trying lately. Every startup event I attended lately has Microsoft evangelists talking up their new tech, offering devices, software and services for free. I appreciate how they are changing their image from arrogant to humble, which is even more noticeable in contrast with Apple, who used to be the developer’s darling and now is becoming increasingly arrogant, especially to developers and users.
Amazon EC2
On Amazon in my experience support is far behind other providers. However, Amazon offers premium support plans that you can pay for and these offer 15 minutes to 12 hours response times depending on how much you pay. Amazon also has the largest developer community, so most of the time you can search for your problem on developer forums or blogs and figure out that either 1 – your problem does not have a solution yet, 2 – find the DIY solution.
Cloudsigma
Cloudsigma’s support was great. Similar to Rackspace they have online support techs who can resolve some issues within minutes while you are waiting and defer anything that requires more work to next tier.
Joyent
Dealing with Joyent’s support staff was a great experience. Issues were resolved within an hour. Whenever something was not getting resolved over email, they were on the phone right away. One great thing about Joyent support was that they also helped me set up performance monitoring tools and let me play with different instance types for free for several weeks.
LiquidWeb
LiquidWeb calls their support “heroic”. Thankfully, I did not need any heroism from them. I contacted support on several occasions with minor problems and each time the problem was resolved within minutes. The only problem with LiquidWeb’s offerings is the sign up process. It is outright terrible. It was so bad that it made me abandon the idea several times before I finally got an account.
Pricing
Unfortunately, it is not possible to do a fair apples-to-apples comparison of the pricing, because except for CloudSigma each provider offers a set of pre-packaged instances that do not correspond in RAM, storage and CPU to the competitors’ instances. Since my goal was to test the CPU performance, for each provider I chose the cheapest instance that offers at least 8 cores. In Amazon’s case, there are 2 types of instances with 8 cores (c1.xlarge and m2.4xlarge).
Rackspace also has a long comparison chart for Amazon EC2, which manages to not address the cost directly: http://www.rackspace.com/cloud/public/servers/compare/. In our case, due to small selection of instance types, running a CPU hungry application on Rackspace turns out to be much more expensive than anywhere else.
Conclusion
Overall, at this time LiquidWeb’s StormOnDemand seems like by far the best option for a CPU-hungry application being both faster and cheaper than others. For other tasks such as offline data processing and aggregation that require less CPU and RAM Windows Azure is the cheapest option.
Title Image (C) C.Mejia http://asininelife.tumblr.com/
Widgeterian 
Leave a comment