Cloud Server Benchmarking Part 3: Java, Ruby, Python & PHP

This is the third post in our series on cloud server performance benchmarking. The previous blog posts, What is an ECU? CPU Benchmarking in the Cloud and Disk IO Benchmarking in the Cloud focused on more synthetic (i.e. raw performance numbers with no real-world application) CPU and Disk IO performance. In this post, we'll look at performance using 4 common interpreted (full or byte-code) programming languages: Java, Ruby, Python and PHP.

This post is by no means intended to provide exhaustive and exact metrics for cloud server performance with these languages. The purpose of this and the other benchmarking posts is to provide a quick reference and starting point for further research and evaluation of different cloud providers. There is much variation in performance, features, reliability, support, pricing and other factors of different cloud providers, yet if you were to evaluate those providers based solely on marketing literature, you'd be hard pressed to distinguish one from another. Our goal is to reduce this ambiguity by providing objective, quantifiable measurements for comparing cloud providers.

Benchmark Setup

All benchmarked cloud servers were configured almost identically in terms of OS and software:

• Operating System: CentOS 64-bit (except for IBM Developer Cloud using RHEL 5 32-bit)
• File System: ext3
• Sun JDK 1.6
• Ruby 1.8.5
• Python 2.4.3
• PHP 5.1.6

Benchmark Methodology

We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is Storm on Demand's bare-metal instance on E5506 2.13 hardware (dual processors - 8 cores total), with 4 x 15K RPM SAS drives (Raid 10) and 8GB ram. This is a fairly high-end server with many cores and very fast IO. We assigned this server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 4 language benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Example Score Calculation:

Baseline aggregate score = SPECjvm2008: 100; ruby-benchmark-suite: 100; PyBench: 100; PHPBench 100

Total baseline score = 400

Server X aggregate score = SPECjvm2008: 80/100; ruby-benchmark-suite: 90/100; PyBench: 70/100; PHPBench 95/100

Total baseline score = 335

Server X Score = (335/400) * 100 = 83.75

Benchmarks

The following benchmarks were used in this post:

• SPECjvm (Java Virtual Machine Benchmark) [Higher score is better]: A benchmark suite for measuring the performance of a Java Runtime Environment (JRE), containing several real life applications and benchmarks focusing on core java functionality. The suite focuses on the performance of the JRE executing a single application; it reflects the performance of the hardware processor and memory subsystem, but has low dependence on file I/O and includes no network I/O across machines. The SPECjvm2008 workload mimics a variety of common general purpose application computations. These characteristics reflect the intent that this benchmark will be applicable to measuring basic Java performance on a wide variety of both client and server systems.
• Ruby [Lower score is better]: A suite for measuring the performance of Ruby implementations, including micro-benchmarks that focus on core Ruby functionality, as well as macro-benchmarks that represent a variety of real, common workloads. The project is aimed at providing a useful suite for comparing the performance of the available Ruby implementations and, as a positive side effect, to give VM implementers an additional tool to measure and identity performance related issues. The score for this benchmark is the average of the median times to execute 13 of the macro benchmarks (all macro benchmarks minus bm_hilbert_matrix.rb). Specific times for these benchmarks individually are available in the benchmark source file.
• Python [Lower score is better]: A collection of tests that provides a standardized way to measure the performance of Python implementations. The score for this benchmark is the average of all benchmarks performed by pybench.
• PHP [Higher score is better]: A benchmark suite for PHP. It performs a large number of simple tests in order to benchmark various aspects of the PHP interpreter. PHPBench can be used to compare hardware, operating systems, PHP versions, PHP accelerators and caches, compiler options, etc.

Results

The results are divided into tables separated by provider. If the provider has more than one data center locations, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), PHPBench score, PyBench score, ruby-benchmark-suite score, SPECjvm2008 score, and the aggregate baseline relative score (as described above). Each of the 4 individual scores are linked to the complete results for those specific tests (a single small text file for PHPBench, PyBench, ruby-benchmark-suite and a 3-4MB zip file for SPECjvm2008).

Note: Some smaller server configurations (e.g. EC2 m1.small and c1.medium) are not included in these results due to their inability to run one more more of the individual benchmarks.

Amazon Web Services (AWS) [US East]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
m2.4xlarge	Xeon X5550 2.67 8 cores	68.4	50328	3725	4.12	197.42	138.58
m2.2xlarge	Xeon X5550 2.67 4 cores	34.2	50253	3737	4.08	108.09	115.45
c1.xlarge	Xeon E5410 2.33 8 cores	7	44460	4586	5.46	131.12	105.76
m2.xlarge	Xeon X5550 2.67 2 cores	17.1	50396	3756	4.1	58.15	102.27
m1.xlarge	Xeon E5430 2.66 4 cores	15	38737	5279	6.45	68.09	79.62
m1.large	Xeon E5430 2.66 2 cores	7.5	38625	5324	6.58	38.21	71.29

Amazon Web Services (AWS) [US West]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
m2.4xlarge	Xeon X5550 2.67 8 cores	68.4	50553	3723	4.09	197.83	139.02
m2.2xlarge	Xeon X5550 2.67 4 cores	34.2	50062	3797	4.1	108.53	114.82
c1.xlarge	Xeon E5410 2.33 8 cores	7	44733	4566	5.48	130.26	105.75
m2.xlarge	Xeon X5550 2.67 2 cores	17.1	50832	3724	4.12	58.59	102.81
m1.xlarge	Xeon E5430 2.67 4 cores	15	39001	5323	6.36	67.98	79.79
m1.large	Xeon E5430 2.66 2 cores	7.5	38716	5382	6.46	38.17	71.38

Amazon Web Services (AWS) [EU West]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
m2.4xlarge	Xeon X5550 2.67 8 cores	68.4	50775	3739	4.1	200.09	139.55
m2.2xlarge	Xeon X5550 2.67 4 cores	34.2	50207	3723	4.1	107.32	115.22
c1.xlarge	Xeon E5410 2.33 8 cores	7	44656	4563	5.51	131.52	105.94
m2.xlarge	Xeon X5550 2.67 2 cores	17.1	50275	3732	4.11	58.92	102.55
m1.xlarge	Xeon E5430 2.67 4 cores	15	38387	5259	6.37	69.06	79.94
m1.large	Xeon E5430 2.67 2 cores	7.5	38327	5299	6.39	39.15	71.91

Amazon Web Services (AWS) [APAC]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
m2.4xlarge	Xeon X5550 2.67 8 cores	68.4	50336	3743	4.42	131.72	119.05
m2.2xlarge	Xeon X5550 2.67 4 cores	34.2	50095	3762	4.42	89.86	108.51
m2.xlarge	Xeon X5550 2.67 2 cores	17.1	50042	3799	4.43	50.63	97.92
c1.xlarge	Xeon E5410 2.33 8 cores	7	44992	4608	5.57	63.93	88.13
m1.xlarge	Xeon E5430 2.67 4 cores	15	38793	5314	6.56	54.15	75.62
m1.large	Xeon E5430 2.67 2 cores	7.5	38214	5299	6.49	34.1	70.28

Rackspace Cloud [Dallas]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
4gb	Opteron 2374 HE 2.20 4 cores	4	39210	5465	5.74	63.07	79.76
16gb	Opteron 2374 HE 2.20 4 cores	16	37914	5517	5.75	62.77	78.66
1gb	Opteron 2374 HE 2.20 4 cores	1	38679	5396	5.81	56.22	77.73
2gb	Opteron 2374 HE 2.20 4 cores	2	38868	5454	5.87	41.89	73.71

Rackspace Cloud [Chicago]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
16gb-il	Opteron 2374 HE 2.20 4 cores	16	38729	5397	5.7	63.86	80.08
4gb-il	Opteron 2374 HE 2.20 4 cores	4	39065	5424	5.8	62.14	79.42
8gb-il	Opteron 2374 HE 2.20 4 cores	8	38958	5455	5.66	63.88	78.31
1gb-il	Opteron 2374 HE 2.20 4 cores	1	38443	5434	5.68	56.88	77.99

Storm on Demand [MI, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
48gb	Xeon X5650 2.67 12 cores	45.9	50602	3810	6.17	174.81	124.05
i5-750-4gb-2disks	Intel i5 750 2.67 4 cores	4	45560	4326	3.17	89.03	110.35
i5-750-2gb-1disk	Intel i5 750 2.67 4 cores	2	45620	4352	3.16	82.16	108.53
x3440-8gb-2disks	Xeon X3440 2.53 8 cores	8	43428	4599	3.38	97.18	107.55
8gb	Xeon X3440 2.53 4 cores	7	43861	4565	3.35	86.55	105.51
e5506x2-4gb-1disk	Xeon E5506 2.13 8 cores	8	40577	4774	3.93	108.06	103.52
e5506x2-4gb-2disks	Xeon E5506 2.13 8 cores	8	40136	4766	3.94	105.9	102.67
32gb	Opteron 2378 2.40 8 cores	30.4	40954	5104	3.8	104.01	101.96
4gb	Core i5 750 2.67 2 cores	3.5	45983	4331	3.17	49.71	100.35
e5506x2-8gb-4disks	Xeon E5506 2.13 8 cores	8	40731	4765	4.02	96.18	100
e5506x2-8gb-4disks	Xeon E5506 2.13 8 cores	8	40306	4780	4.01	95.13	99.45
2gb	Core 2 Quad Q9400 2.66 1 core	1.7	49905	4142	3.59	23.91	93.6
amd2350x2-32gb-4disks	Opteron 2350 2.00 8 cores	32	32182	6271	4.96	86.73	81.55
16gb	Opteron 2350 2.00 4 cores	15.2	32240	6187	4.87	53.1	73.48

GoGrid [CA, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
8gb	Xeon E5450 2.99 6 cores	8	55970	3616	3.53	101.76	121.46
4gb	Xeon E5520 2.27 4 cores	4	40513	4806	3.8	74.34	95.42
2gb	Xeon E5520 2.27 2 cores	2	38206	5090	4.09	36.13	80.82
1gb	Xeon E5520 2.27 1 core	1	38215	4945	4.02	20.56	77.89

Voxel [New York]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
14gb-ny	Xeon L5520 2.26 7 cores	14	44463	4214	4.98	119.82	106.89
8gb-ny	Xeon L5520 2.26 4 cores	8	44043	4237	4.91	81.23	96.73
4gb-ny	Xeon L5520 2.26 2 cores	4	40921	4446	4.95	45.71	84.1
2gb-ny	Xeon L5520 2.26 1 core	2	44022	4210	4.94	24.47	82.02

Voxel [Amsterdam]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
14gb-nl	Xeon L5520 2.26 7 cores	14	45069	4221	4.85	124.22	108.9
8gb-nl	Xeon L5520 2.26 4 cores	8	45043	4219	4.9	82.02	97.71
4gb-nl	Xeon L5520 2.26 2 cores	4	44880	4217	4.83	45.52	88.44
2gb-nl	Xeon L5520 2.26 1 core	2	44436	4229	4.86	24.57	82.51

Voxel [Singapore]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
14gb-sg	Xeon L5520 2.26 7 cores	14	42488	4468	5.14	111.08	101.17
8gb-sg	Xeon L5520 2.26 4 cores	8	42355	4473	5.15	75.75	91.84
4gb-sg	Xeon L5520 2.26 2 cores	4	42433	4480	5.15	42.15	83.11
2gb-sg	Xeon L5520 2.26 1 core	2	42153	4482	5.16	23.08	77.93

NewServers [FL, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
fast	Xeon E5450 2.99 4 cores	4	57795	3600	3.24	81.3	120.71
jumbo	Xeon E5504 2.00 8 cores	48	37593	5087	4.2	110.03	99.02
large	Xeon E5405 2.00 4 cores	4	38161	5462	4.59	59.75	82.66
med	Xeon 3.20 2 cores	2	29381	7383	5.89	25.74	57.92
small	Xeon 2.80 1 core	1	25223	8183	6.9	12.37	47.82

Linode VPS Hosting [Atlanta]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
5760	Xeon L5520 2.27 4 cores	5.54	42841	4418	4.67	78.51	95.19
14400	Xeon L5520 2.27 4 cores	14.06	42644	4410	4.65	78.45	95.19
2880	Xeon L5520 2.27 4 cores	2.81	39050	4503	5.02	66.78	87.8
1080	Xeon L5520 2.27 4 cores	1.05	40442	4579	5.12	59.57	85.95
8640	Xeon L5520 2.27 4 cores	8.45	32904	5632	5.28	67.85	78.02

SoftLayer [Dallas]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
8gb	Xeon E5520 2.27 8 cores	8	42414	4439	5.5	123.82	103.33
4gb	Xeon E5520 2.27 4 cores	4	42471	4459	5.51	78.74	91.49
2gb	Xeon X3460 2.80 2 cores	2	47816	4130	5.53	49.59	89.26
1gb	Xeon X3460 2.80 1 core	1	47634	4118	4.61	27.86	87.21

SoftLayer [WDC]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
8gb	Xeon E5520 2.27 8 cores	8	42981	4441	5.48	127.55	104.7
4gb	Xeon X3460 2.80 4 cores	4	47425	4101	4.62	91.07	103.58
2gb	Xeon X3460 2.80 2 cores	2	47478	4108	4.4	48.95	93.7
1gb	Xeon X3460 2.80 1 core	1	47614	4113	4.47	27.61	87.85

SoftLayer [Seattle]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
8gb	Xeon E5520 2.27 8 cores	8	42642	4451	5.54	125.75	103.76
2gb	Xeon X3460 2.80 2 cores	2	48095	4138	4.63	48.7	92.67
1gb	Xeon X3460 2.80 1 core	1	47472	4113	4.61	26.49	86.79

Terremark vCloud Express [FL, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
16gb-8vpu	Opteron 8389 2.91 8 cores	16	48625	4623	3.99	100.42	106.9
1gb	Opteron 8389 2.91 1 core	1	48997	4254	3.18	22	95.4
2gb	Opteron 8389 2.91 1 core	2	48651	4321	3.17	21.87	94.82
4gb-2vpu	Opteron 8389 2.91 2 cores	4	39078	5019	3.72	38.78	84.82
8gb-4vpu	Opteron 8389 2.91 4 cores	8	32959	5388	4.19	70.56	84.67

VPS.NET [Atlanta]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
18node	Xeon E5520 2.27 7 cores	6.6	39447	4762	5.85	96.35	91.45
12node	Xeon E5520 2.27 [5 processors, 5 cores	4.4	42843	5035	5.39	77.12	88.65
6node	Xeon E5520 2.27 [3 processors, 3 cores	2.2	33213	5730	6.8	45.81	67.86
3node	Xeon E5520 2.27 2 cores	1.1	34188	5452	6.54	35.72	67.49

OpSource Cloud [VA, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
2gb	Xeon X7460 2.66 1 core	2	28083	7721	7.93	11.13	48.23
8gb-4cpu	Xeon X7460 2.66 4 cores	8	24639	8758	9.08	9.03	42.14
32gb-4cpu	Xeon X7460 2.66 4 cores	32	23604	8790	8.87	9.04	41.72
16gb-4cpu	Xeon X7460 2.66 4 cores	16	24636	8738	9.43	8.77	41.69

Speedyrails [Quebec]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
4gb	Xeon E5520 2.27 8 cores	4	41819	4735	5.26	100.74	96.12
1gb	Xeon E5520 2.27 8 cores	1	41541	4975	5.28	56.54	83.18

Zerigo [CO, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
4gb	Opteron 2374 HE 2.20 4 cores	4	39030	5400	5.58	55.54	78.46

ReliaCloud [MN, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
4gb	Xeon E5504 2.00 4 cores	4	13287	13209	10.7	3.85	27.56
8gb		8	14159	15567	10.79	3.69	26.62
2gb	Xeon E5504 4.40 2 cores	2	12131	13819	11.36	5.36	26.31

CloudSigma [Switzerland]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
32gb-20gh	Six-Core Opteron 2427 2.21 8 cores	32	37912	5646	4.27	97.2	93.17
16gb-16gh	Six-Core Opteron 2427 2.21 8 cores	16	37599	5654	4.3	98.23	93.05
8gb-8gh	Six-Core Opteron 2427 2.21 4 cores	8	37473	5619	4.2	59.99	83.72
4gb-4gh	Six-Core Opteron 2427 2.21 2 cores	4	37409	5582	4.26	32.54	76.35
1gb-2gh	Six-Core Opteron 2427 2.21 1 core	1	37326	5609	4.27	17.2	72.16
2gb-2gh	Six-Core Opteron 2427 2.21 1 core	2	36561	5789	4.25	17.82	71.3

IBM Development & Test Cloud [NY, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
large	Xeon X5570 2.93 8 cores	3.5	45260	4711	3.6	140.6	117.53

BlueLock [IN, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
8gb-4cpu	Xeon X5550 2.67 4 cores	8	54399	3761	3	94.08	123.02
16gb-8cpu	Xeon X5550 2.67 8 cores	16	48314	3859	3.36	96.07	115.41
4gb-2cpu	Xeon X5550 2.67 2 cores	4	45071	3894	3.21	51.9	103.05
2gb	Xeon X5550 2.67 1 core	2	50021	3739	3.16	26.72	101.31
1gb	Xeon X5550 2.67 1 core	1	49499	3745	3.06	16.01	99.2

Cloud Central [Australia]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
reg	Opteron 2374 HE 2.20 4 cores	4	38725	5469	5.55	60.59	79.41
large	Opteron 2374 HE 2.20 4 cores	8	38921	5439	5.64	60.58	79.36
huge	Opteron 2374 HE 2.20 4 cores	16	37989	5573	6.43	61.86	76.4
small	Opteron 2374 HE 2.20 4 cores	1	37356	5597	6.18	49.6	73.36

RimuHosting [TX, US]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
tx-2gb	Xeon E5506 2.13 2 cores	2	39096	4883	4.89	39.28	79.16

RimuHosting [New Zealand]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
nz-2gb	Xeon E5506 2.13 2 cores	2	39529	4854	5.08	39.22	78.78

ElasticHosts [UK]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
8gb-20gh	Xeon E5420 2.50 6 cores	8	47168	4431	4.1	92.82	104.47
2gb-4gh	Xeon E5420 2.50 2 cores	2	43723	4508	4.11	34.92	86.79
1gb-2gh	Xeon E5420 2.50 1 core	1	44943	4470	4.1	20.94	84.19

Flexiscale [UK]
ID	CPU	Memory	PHP	Python	Ruby	SPECjvm	Score
4gb	Opteron 8220 2.80 2 cores	4	36319	6033	4.03	34.1	69.57
8gb	Opteron 8218 2.62 4 cores	8	33878	6570	4.35	53.46	67.87
2gb	Opteron 8218 2.60 1 core	2	35141	6505	4.16	17.19	63.09
1gb	Opteron 8218 2.60 1 core	1	34497	6590	4.39	16.16	63.05

Summary

We think it is important for these and other cloud performance benchmarks to be made available publicly. With all the hype surrounding cloud computing today, new cloud users may not understand what exactly they are getting in terms of performance and scalability. As these benchmarks show, not all clouds are created equal, and there can be very pronounced performance variations from one provider to another. One of the greatest benefits of the cloud is simplicity and a perceived ability to infinitely scale. However, it should be understood that with that simplicity and abstraction, the cloud is still powered by physical hardware, CPUs, disks and memory, and subject to the same performance limitations that same hardware exhibits in a non-cloud environment.