Zettagrid Benchmarked Against Other Cloud Providers

Independent Performance and Test Automation Experts, Altentee benchmarked Australian cloud hosting providers targeting a site hosted by AussieHQ in Canberra and Sydney. The tests were performed between 31-07-2010 and 01-08-2010.

Types of Tests

  1. 10MB file download from a static url hosted on a single web server.
  2. ApacheBench ranging from 10 to 400 concurrent requests to a static url hosted on a single web server.
  3. JMeter load test with 500 concurrent users requesting multiple URLs hosted on the web farm (10 servers) split across Sydney & Canberra.

All benchmarks were run at the same time for comparison. 4 servers at each provider participated in the benchmarks as follows:

Participating Cloud Providers

Provider  Location  Uname OS
Web24 Melbourne 2.6.28-11 Ubuntu
Zettagrid Perth 2.6.32-24 Ubuntu
CloudCentral Canberra 2.6.24-27 Ubuntu
VMVault Brisbane 2.6.18-194 CentOS
AmazoneEC2 US-East 2.6.31-302 Ubuntu

Results

Test 1. 10MB File Download (wget http://…./file.jpg*)

Web24 and VMVault were the fastest, with other providers less than 1MByte/sec. The following chart plots Mbytes per second for each provider.

Test 2. ApacheBench (ab -kc 10..400 -t 60 http://targetsite/)

Zettagrid, VMVault and Amazon were the best performers. CloudCentral and Web24 struggle under concurrent load, which may be an artifact of the kernel patch level. Note: default TCP tuning params have been used on each OS. The following chart plots Mbps for each concurrent user test.

Test 3. JMeter Load Test (500 threads, 5000 samples/min per VM)

Zettagrid, VMVault were the best performers in terms of throughput. CloudCentral and Web24 appear to struggle under concurrent load with general network connectivity issues (unable to find host, timeouts). The following chart plots throughput in Mbps for each provider.

Summary

Based on price alone, Zettagrid and Amazon are the best value for money. VMVault also offer high performance on more dedicated infrastructure. Problems with Web24 and CloudCentral under high load/concurrency would need to be resolved going forward. This may be an artifact of slightly older linux kernels or a TCP tuning configuration. Will keep working to try and improve the performance/throughput with the latter 2 providers.

Thanks for your help to date. If you’d like more info on how the benchmarks were executed please don’t hesitate to contact me.

Regards,
Tim Koopmans
info@altentee.com
Altentee.com
–Performance & Test Automation Experts
This is information is copyright of Altentee.