Cloud Server Benchmarking Part 4: Memory IO

This is the fourth post in our series on cloud server performance benchmarking. In this post, we'll examine memory IO performance using a compilation of 7 different benchmarks. This is the last of the performance posts to focus on synthetic benchmarks. Memory IO performance is particularly important for applications that read and write from memory heavily like caching systems and memory-based data stores such as memcache and redis.

Benchmark Setup

All benchmarked cloud servers were configured almost identically using CentOS 64-bit (or 32-bit in the case of EC2 m1.small, c1.medium, Gandi, and IBM cloud servers).

Benchmark Methodology

We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is the NewServers Jumbo server configured with dual Intel E5504 quad core 2.00 GHz processors and 48GB DDR3 ECC ram. We chose NewServers because they are the only IaaS cloud that does not utilize a virtualization layer that could adversely affect the benchmark results. All NewServers servers run on top of physical hardware. We assigned the baseline server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 7 benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Benchmarks

The following benchmarks were used to calculate the aggregate memory IO performance (MIOP) score displayed in the results tables below. Geekbench and Unixbench measure both CPU and memory IO performance and are included in the aggregate MIOP metric with a weight of 25 points each. To view the raw Geekbench and Unixbench scores for each of the servers in this post see the post What is an ECU? CPU Benchmarking in the Cloud.

• CacheBench [weight=100]: This is a performance test of CacheBench, which is part of LLCbench. CacheBench is designed to test the memory and cache bandwidth performance. 50 points are assigned to each of the benchmarks read and write.
• Geekbench [weight=25]: Geekbench provides a comprehensive set of benchmarks engineered to quickly and accurately measure processor and memory performance. Designed to make benchmarks easy to run and easy to understand, Geekbench takes the guesswork out of producing robust and reliable benchmark results.
• hdparm cached reads [weight=50]: Determines the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.
• RAMspeed [weight=100]: This benchmark tests the system memory (RAM) performance. 33 points are assigned to each of the benchmarks add, copy and scale.
• Redis Benchmark [weight=50]: Redis is an in-memory key-value store. It includes the redis-benchmark utility that simulates SETs/GETs/INCRs/LPUSHs/LPOPs/PINGs/LRANGEs done by N clients at the same time sending M total queries (it is similar to the Apache's ab utility). Our benchmark is performed with 50 simultaneous clients performing 100000 requests (./redis-benchmark -n 100000). The result is the average requests per second for all benchmark actions (gets, sets, etc.). For specific results, download the benchmark output source.
• Stream [weight=100]: This benchmark tests the system memory (RAM) performance. 25 points are assigned to each of the benchmarks add, copy, scale and triad.
• Unixbench [weight=25]: UnixBench provides a basic indicator of the performance of a Unix-like system. Multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system.

Results

The results are divided into tables separated by provider. If the provider has more than one data center location, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), Redis Benchmark score (linked to the complete Redis benchmark results), and the aggregate baseline relative score (as described above). The top performers in this benchmark category were EC2, Storm on Demand (6 servers with scores 116 and above), BlueLock, and IBM Developer Cloud.

Amazon Web Services (AWS) [US East]
ID	CPU	Memory	Redis	MIOP
us-east.m2.4xlarge	Xeon X5550 2.67 [8 processors, 8 cores]	68.4	44330.61	117.88
us-east.m2.2xlarge	Xeon X5550 2.67 [4 processors, 4 cores]	34.2	53150.28	113.04
us-east.m2.xlarge	Xeon X5550 2.67 [2 processors, 2 cores]	17.1	43767.98	102.18
us-east.m1.xlarge	Xeon E5430 2.66 [4 processors, 4 cores]	15	27314.76	63.67
us-east.m1.large	Xeon E5430 2.66 [2 processors, 2 cores]	7.5	28656.14	57.05

Amazon Web Services (AWS) [US West]
ID	CPU	Memory	Redis	MIOP
us-west.m2.4xlarge	Xeon X5550 2.67 [8 processors, 8 cores]	68.4	42603.58	118.12
us-west.m2.2xlarge	Xeon X5550 2.67 [4 processors, 4 cores]	34.2	48163.06	112.11
us-west.m2.xlarge	Xeon X5550 2.67 [2 processors, 2 cores]	17.1	52766.05	106.11
us-west.c1.medium	Xeon E5410 2.33 [1 processor, 2 cores]	1.7	23420.06	70.83
us-west.m1.xlarge	Xeon E5430 2.67 [4 processors, 4 cores]	15	31635.65	65.49
us-west.m1.large	Xeon E5430 2.66 [2 processors, 2 cores]	7.5	28608.91	64.93
us-west.m1.small	Xeon E5430 2.66 [1 processor, 1 core]	1.7	9214.6	31.81

Amazon Web Services (AWS) [EU West]
ID	CPU	Memory	Redis	MIOP
eu-west.m2.4xlarge	Xeon X5550 2.67 [8 processors, 8 cores]	68.4	44084.09	117.27
eu-west.m2.2xlarge	Xeon X5550 2.67 [4 processors, 4 cores]	34.2	48563.97	111.74
eu-west.m2.xlarge	Xeon X5550 2.67 [2 processors, 2 cores]	17.1	46370.88	104.07
eu-west.c1.medium	Xeon E5410 2.33 [1 processor, 2 cores]	1.7	37373.2	73.48
eu-west.m1.xlarge	Xeon E5430 2.67 [4 processors, 4 cores]	15	28149.67	65.94
eu-west.m1.large	Xeon E5430 2.67 [2 processors, 2 cores]	7.5	27864.93	63.22
eu-west.m1.small	Xeon E5430 2.66 [1 processor, 1 core]	1.7	9038.17	33.52

Amazon Web Services (AWS) [APAC]
ID	CPU	Memory	Redis	MIOP
apac.m2.xlarge	Xeon X5550 2.67 [2 processors, 2 cores]	17.1	15859.76	97.48
apac.m2.2xlarge	Xeon X5550 2.67 [4 processors, 4 cores]	34.2	15587.35	95.94
apac.m2.4xlarge	Xeon X5550 2.67 [8 processors, 8 cores]	68.4	31992.3	92.74
apac.c1.medium	Xeon E5410 2.33 [1 processor, 2 cores]	1.7	16492.91	68.46
apac.c1.xlarge	Xeon E5410 2.33 [8 processors, 8 cores]	7	23171.72	66.47
apac.m1.large	Xeon E5430 2.67 [2 processors, 2 cores]	7.5	21667.04	59.67
apac.m1.xlarge	Xeon E5430 2.67 [4 processors, 4 cores]	15	18181.59	59.47
apac.m1.small	Xeon E5430 2.67 [1 processor, 1 core]	1.7	7465.1	31.67

Rackspace Cloud [Dallas]
ID	CPU	Memory	Redis	MIOP
16gb	Opteron 2374 HE 2.20 [1 processor, 4 cores]	16	23844.89	70.09
4gb	Opteron 2374 HE 2.20 [1 processor, 4 cores]	4	16928.56	63.44
2gb	Opteron 2374 HE 2.20 [1 processor, 4 cores]	2	19167	63.04
1gb	Opteron 2374 HE 2.20 [1 processor, 4 cores]	1	15945.68	62.83

Rackspace Cloud [Chicago]
ID	CPU	Memory	Redis	MIOP
16gb-il	Opteron 2374 HE 2.20 [1 processor, 4 cores]	16	25400.7	70.77
8gb-il	Opteron 2374 HE 2.20 [1 processor, 4 cores]	8	25528.68	68.65
2gb-il	Opteron 2374 HE 2.20 [1 processor, 4 cores]	2	24860.47	68.14
1gb-il	Opteron 2374 HE 2.20 [1 processor, 4 cores]	1	25719.31	67.84
4gb-il	Opteron 2374 HE 2.20 [1 processor, 4 cores]	4	25248.93	66.34

Storm on Demand [MI, US]
ID	CPU	Memory	Redis	MIOP
48gb	Xeon X5650 2.67 [1 processor, 12 cores]	45.9	62269.22	136.4
i5-750-4gb-2disks	Core i5 750 2.67 [1 processor, 4 cores]	4	79473.07	124.49
i5-750-2gb-1disk	Core i5 750 2.67 [1 processor, 4 cores]	2	81301.01	122.88
x3440-8gb-2disks	Xeon X3440 2.53 [1 processor, 8 cores]	8	74553.53	121.15
8gb	Xeon X3440 2.53 [1 processor, 4 cores]	7	74183.16	119.81
4gb	Core i5 750 2.67 [1 processor, 2 cores]	3.5	76701.91	116.99
e5506x2-4gb-2disks	Xeon E5506 2.13 [1 processor, 8 cores]	8	49970.66	83.18
e5506x2-4gb-1disk	Xeon E5506 2.13 [1 processor, 8 cores]	8	49760.06	82.8
e5506x2-8gb-4disks	Xeon E5506 2.13 [1 processor, 8 cores]	8	49775.14	76.5
32gb	Opteron 2378 2.40 [4 processors, 8 cores]	30.4	37672.21	75.82
e5506x2-8gb-4disks	Xeon E5506 2.13 [1 processor, 8 cores]	8	49178.7	75.36
2gb	Core 2 Quad Q9400 2.66 [1 processor, 1 core]	1.7	22571.9	65
amd2350x2-32gb	Opteron 2350 2.00 [4 processors, 8 cores]	32	29966.25	59.15
16gb	Opteron 2350 2.00 [2 processors, 4 cores]	15.2	29179.3	52.32

GoGrid [CA, US]
ID	CPU	Memory	Redis	MIOP
4gb	Xeon E5520 2.27 [1 processor, 4 cores]	4	46176.73	96.12
8gb	Xeon E5450 2.99 [1 processor, 6 cores]	8	32590.17	89.74
2gb	Xeon E5520 2.27 [1 processor, 2 cores]	2	39756.91	83.74
1gb	Xeon E5520 2.27 [1 processor, 1 core]	1	23346.48	79.7

Voxel [New York]
ID	CPU	Memory	Redis	MIOP
14gb-ny	Xeon L5520 2.26 [7 processors, 7 cores]	14	30566.6	100.02
8gb-ny	Xeon L5520 2.26 [4 processors, 4 cores]	8	29376.08	89.78
4gb-ny	Xeon L5520 2.26 [2 processors, 2 cores]	4	23520.43	88.62
2gb-ny	Xeon L5520 2.26 [1 processor, 1 core]	2	14434.5	79.8

Voxel [Amsterdam]
ID	CPU	Memory	Redis	MIOP
14gb-nl	Xeon L5520 2.26 [7 processors, 7 cores]	14	27423.78	105.25
8gb-nl	Xeon L5520 2.26 [4 processors, 4 cores]	8	30788.54	101.71
4gb-nl	Xeon L5520 2.26 [2 processors, 2 cores]	4	27979.96	96.63
2gb-nl	Xeon L5520 2.26 [1 processor, 1 core]	2	14691.72	86

Voxel [Singapore]
ID	CPU	Memory	Redis	MIOP
14gb-sg	Xeon L5520 2.26 [7 processors, 7 cores]	14	28547.88	87.16
8gb-sg	Xeon L5520 2.26 [4 processors, 4 cores]	8	27832.58	84.95
4gb-sg	Xeon L5520 2.26 [2 processors, 2 cores]	4	24189.04	77.39
2gb-sg	Xeon L5520 2.26 [1 processor, 1 core]	2	15237.89	75.99

NewServers [FL, US]
ID	CPU	Memory	Redis	MIOP
jumbo	Xeon E5504 2.00 [2 processors, 8 cores]	48	39567.78	100
large	Xeon E5405 2.00 [1 processor, 4 cores]	4	34997.3	72.25
med	Xeon 3.20 [1 processor, 2 cores]	2	22225.77	43.78
small	Xeon 2.80 [1 processor, 1 core]	1	11758.45	31.8

Linode VPS Hosting [Atlanta]
ID	CPU	Memory	Redis	MIOP
14400-atlanta	Xeon L5520 2.27 [2 processors, 4 cores]	14.06	29486.24	99.9
5760-atlanta	Xeon L5520 2.27 [2 processors, 4 cores]	5.54	13240.74	96.33
2880-atlanta	Xeon L5520 2.27 [2 processors, 4 cores]	2.81	20404.32	93.26
1080-atlanta	Xeon L5520 2.27 [2 processors, 4 cores]	1.05	22810.13	87.48
8640-atlanta	Xeon L5520 2.27 [2 processors, 4 cores]	8.45	12350.71	86.39

SoftLayer [Dallas]
ID	CPU	Memory	Redis	MIOP
4gb-dallas	Xeon E5520 2.27 [4 processors, 4 cores]	4	26091.52	98.27
8gb-dallas	Xeon E5520 2.27 [8 processors, 8 cores]	8	25067.69	96.93
2gb-dallas	Xeon X3460 2.80 [2 processors, 2 cores]	2	22067.26	95.74
1gb-dallas	Xeon X3460 2.80 [1 processor, 1 core]	1	14838.57	94.58

SoftLayer [Seattle]
ID	CPU	Memory	Redis	MIOP
8gb-seattle	Xeon E5520 2.27 [8 processors, 8 cores]	8	25503.99	105.11
4gb-seattle	Xeon X3460 2.80 [4 processors, 4 cores]	4	37422.67	103.86
2gb-seattle	Xeon X3460 2.80 [2 processors, 2 cores]	2	34444.9	99.62
1gb-seattle	Xeon X3460 2.80 [1 processor, 1 core]	1	14836.2	93.01

Terremark vCloud Express [FL, US]
ID	CPU	Memory	Redis	MIOP
1gb	Opteron 8389 2.91 [1 processor, 1 core]	1	37476.72	66.78
2gb	Opteron 8389 2.91 [1 processor, 1 core]	2	37367.9	66.06
16gb-8vpu	Opteron 8389 2.91 [1 processor, 8 cores]	16	36601.03	64.74
4gb-2vpu	Opteron 8389 2.91 [1 processor, 2 cores]	4	42788.76	64.03
8gb-4vpu	Opteron 8389 2.91 [1 processor, 4 cores]	8	41525.25	61.08

VPS.NET [Atlanta]
ID	CPU	Memory	Redis	MIOP
12node-atl	Xeon E5520 2.27 [5 processors, 5 cores]	4.4	28676.91	98.17
18node-atl	Xeon E5520 2.27 [7 processors, 7 cores]	6.6	17864.39	88.83
3node-atl	Xeon E5520 2.27 [2 processors, 2 cores]	1.1	25017.57	81
6node-atl	Xeon E5520 2.27 [3 processors, 3 cores]	2.2	23600.54	73.24

OpSource Cloud [VA, US]
ID	CPU	Memory	Redis	MIOP
1gb	Xeon X7460 2.66 [1 processor, 1 core]	1	13552.35	38.13
2gb	Xeon X7460 2.66 [1 processor, 1 core]	2	12775.31	36.56
4gb-2cpu	Xeon X7460 2.66 [1 processor, 2 cores]	4	13772.58	36.52
8gb-4cpu	Xeon X7460 2.66 [1 processor, 4 cores]	8	13538.06	34.62
16gb-4cpu	Xeon X7460 2.66 [1 processor, 4 cores]	16	13131.61	33.88
32gb-4cpu	Xeon X7460 2.66 [1 processor, 4 cores]	32	12260.02	32.99

Speedyrails [Quebec]
ID	CPU	Memory	Redis	MIOP
1gb	Xeon E5520 2.27 [2 processors, 8 cores]	1	20705.27	92.89
4gb	Xeon E5520 2.27 [2 processors, 8 cores]	4	21118.42	87.52
2gb	Xeon E5520 2.27 [2 processors, 8 cores]	2	19086.71	83.5

Zerigo [CO, US]
ID	CPU	Memory	Redis	MIOP
4gb	Opteron 2374 HE 2.20 [2 processors, 4 cores]	4	16428.46	63.81

ReliaCloud [MN, US]
ID	CPU	Memory	Redis	MIOP
4gb	Xeon E5504 2.00 [1 processor, 4 cores]	4	6499.32	34.04
2gb	Xeon E5504 4.40 [1 processor, 2 cores]	2	5559.85	32.21
1gb	Xeon E5504 2.00 [1 processor, 1 core]	1	5300.22	22.42

Gandi.net [France]
ID	CPU	Memory	Redis	MIOP
4gb	Opteron 8346 HE 1.79 [1 processor, 4 cores]	4	33143.29	46.59
6gb	Opteron 8346 HE 1.79 [1 processor, 6 cores]	6	10647.31	43.35
2gb	Opteron 8346 HE 1.79 [1 processor, 2 cores]	2	10713.03	36.56
1gb	Opteron 8346 HE 1.79 [1 processor, 1 core]	1	11255.61	33.6

CloudSigma [Switzerland]
ID	CPU	Memory	Redis	MIOP
16gb-16gh	Six-Core Opteron 2427 2.21 [1 processor, 8 cores]	16	37227.13	72.03
32gb-20gh	Six-Core Opteron 2427 2.21 [1 processor, 8 cores]	32	36628.6	67.26
4gb-4gh	Six-Core Opteron 2427 2.21 [1 processor, 2 cores]	4	37030.85	60.68
2gb-2gh	Six-Core Opteron 2427 2.21 [1 processor, 1 core]	2	21457.67	56.7
1gb-2gh	Six-Core Opteron 2427 2.21 [1 processor, 1 core]	1	21682.06	55.96

IBM Development & Test Cloud [NY, US]
ID	CPU	Memory	Redis	MIOP
dev-large	Xeon X5570 2.93 [1 processor, 8 cores]	3.5	62280.44	124.37
dev-med	Xeon X5570 2.93 [1 processor, 4 cores]	1.75	46424.68	95.5

BlueLock [IN, US]
ID	CPU	Memory	Redis	MIOP
16gb-8cpu	Xeon X5550 2.67 [1 processor, 8 cores]	16	57578.66	117.88
8gb-4cpu	Xeon X5550 2.67 [1 processor, 4 cores]	8	48148.66	115.64
4gb-2cpu	Xeon X5550 2.67 [1 processor, 2 cores]	4	57390.96	107.8
2gb	Xeon X5550 2.67 [1 processor, 1 core]	2	29641.05	103.73
1gb	Xeon X5550 2.67 [1 processor, 1 core]	1	24710.52	91.9

Cloud Central [AU]
ID	CPU	Memory	Redis	MIOP
large	Opteron 2374 HE 2.20 [4 processors, 4 cores]	8	18585.76	60.91
reg	Opteron 2374 HE 2.20 [4 processors, 4 cores]	4	19328.25	59.37
huge	Opteron 2374 HE 2.20 [4 processors, 4 cores]	16	13726.38	57.93
med	Opteron 2374 HE 2.20 [4 processors, 4 cores]	2	18325.34	57.23
small	Opteron 2374 HE 2.20 [4 processors, 4 cores]	1	17260.76	56.05

RimuHosting [TX, US]
ID	CPU	Memory	Redis	MIOP
tx-2gb	Xeon E5506 2.13 [1 processor, 2 cores]	2	24771.2	71.62

RimuHosting [NZ]
ID	CPU	Memory	Redis	MIOP
nz-2gb	Xeon E5506 2.13 [1 processor, 2 cores]	2	22446.26	71.22

ElasticHosts [UK]
ID	CPU	Memory	Redis	MIOP
4gb-8gh	Xeon E5420 2.50 [1 processor, 4 cores]	4	38376.45	74.91
1gb-2gh	Xeon E5420 2.50 [1 processor, 1 core]	1	17456.46	67.03
2gb-4gh	Xeon E5420 2.50 [1 processor, 2 cores]	2	24852.55	66.4

Flexiscale [UK]
ID	CPU	Memory	Redis	MIOP
8gb	Opteron 8218 2.62 [1 processor, 4 cores]	8	20853.61	44.9
4gb	Dual-Core Opteron 8220 2.80 [1 processor, 2 cores]	4	22004.85	44.51
2gb	Opteron 8218 2.60 [1 processor, 1 core]	2	17149.07	42.86
1gb	Opteron 8218 2.60 [1 processor, 1 core]	1	16410.28	37.14