Encoding Performance: Comparing Zencoder, Encoding.com, Sorenson & Panda

A few months ago we were approached by Zencoder to conduct a sponsored performance comparison of 4 encoding services including their own. The purpose was to validate their claims of faster encoding times using an independent, credible external source. This was a new frontier for us. Our primary focus has been performance analysis of infrastructure as a service (IaaS). However, we are curious about all things related to cloud and benchmarking and we felt this could be useful data to make available publicly, so we accepted.

Testing Methodology

This is a description of the methodology we used for conducting this performance analysis.

Source Media

Following discussions with Zencoder, we opted to test encoding performance using 4 distinct media types. We were tasked with finding samples for each media type, they were not provided by Zencoder. All source media was stored in AWS S3 using the US East region (the same AWS region each of the 4 encoding services are hosted from). The 4 media types we used for testing are:

• HD Video: We chose an HD 1080P trailer for the movie Avatar. This file was 223.1 MB in size and 3 mins, 30 secs in duration.
• SD Video: We chose a 480P video episode from a cartoon series. The file was 519.2 MB in size and about 23 mins in duration.
• Mobile Video: We created a 568x320 video using an iPhone (source here). The file was 2.9 MB in size, 30 secs in duration.
• MP3 Audio: We used an MP3 file we found on the web about Yoga (source here). The file was 42.2 MB in size, 58 mins 41 secs in duration.

Encode Settings

We used the same encode options across all of the services tested. The following is a summary of the encode options used for each corresponding media type:

Media Type	Video Codec	Video Bitrate	Audio Codec	Audio Bitrate	Encode Passes
HD Video	H.264	3000 Kb/s	AAC	96 Kb/s	2
SD Video	H.264	1000 Kb/s	AAC	96 Kb/s	2
Mobile Video	H.264	500 Kb/s	AAC	96 Kb/s	2
MP3 Audio	NA	NA	AAC	96 Kb/s	2

Test Scheduling

Testing was conducted during a span of 1 week. We built a test harness that integrated with the APIs of each of the 4 encoding services. The test harness invoked 2 test iterations daily with each service. Testing included both single request and 4 parallel requests. Each test iteration consisted of the following 8 test scenarios:

• Single HD Video Request
• Single SD Video Request
• Single Mobile Video Request
• Single MP3 Audio Request
• 4 Parallel HD Video Requests
• 4 Parallel SD Video Requests
• 4 Parallel Mobile Video Requests
• 4 Parallel MP3 Audio Requests

The order of the test scenarios was randomized, but the same tests were always requested at the same time for each service. Each test was run to completion on all services before the next test was invoked. The start times for 2 daily test iterations were separated by about 12 hours and incremented by 100 minutes each day. The end result was a distribution of test scenarios during many different times of the day. A total of 112 tests were performed during the 1 week test span, followed by an additional 24 test scenarios on encoding.com to test different combinations of service specific encoding options (described below).

Performance Metrics

During testing, we captured the following metrics:

• Encode Time: The amount of time in seconds required to encode the media (excludes transfer time)
• Transfer Time: The amount of time in seconds required to transfer the source media from AWS S3 into the service processing queue
• Failures: When a service failed to complete a job

Test Results

The following is a summary of the results for each of the 8 test scenarios. Result tables have the following columns:

• Avg Encode Time: The mean (average) encoding time in seconds for all jobs in this test scenario
• Standard Deviation %: Standard deviation as percentage of the mean. A lower values indicates more consistency in performance
• Median Encode Time: The median encoding time in seconds for all jobs in this test scenario
• Avg Total Time: The mean (average) total job time in seconds. The total time is the sum of transfer (source files were hosted in AWS S3 US East), queue and encoding times, essentially the total turnaround time from the moment the job was submitted. Sorenson and Panda may cache source media files, thus reducing transfer time for future requests

A graph is displayed below each results table depicting sorted average encode and total times using a dual series horizontal bar chart. Also depicted on the graph is a line indicating the actual duration of the source media. Encode time bars that terminate to the left of this line signify faster than realtime encoding performance.

encoding.com offers a few different encoding job options that can affect encode performance. These options include the following:

• instant: using the instant option, encoding.com will begin encoding before the source media has been fully downloaded. For larger source files, this can decrease encode times
• twin turbo: this setting causes jobs to be delegated to faster servers in exchange for paying a $2/GB premium for encoding. encoding.com states that this option will deliver 6-8X faster encoding time over standard servers

Test Results: Single HD Video Request

The following are the results from a total of 14 HD video encode jobs submitted at various times of the day over a period of 1 week:

Service	Avg Encode Time	Standard Deviation %	Median Encode Time	Avg Total Time
zencoder	141.08	6	137	155.08
encoding.com 2 (max+TT)	213.38	16	193	244.53
encoding.com 1 (max+TT+instant)	213.5	11	213.5	238
encoding.com 4 (plus+TT+instant)	226.75	11	235.5	258.75
encoding.com 3 (max)	618.75	12	643.5	653.25
sorenson	974.31	2	975	982.46
panda	1246.31	4	1247	1255.93

Failures - None

Test Results: Single SD Video Request

The following are the results from a total of 14 SD video encode jobs submitted at various times of the day over a period of 1 week:

Service	Avg Encode Time	Standard Deviation %	Median Encode Time	Avg Total Time
zencoder	229.14	9	225.5	281.64
encoding.com 4 (plus+TT+instant)	453	1	453.5	718.75
encoding.com 2 (max+TT)	454.07	11	458.5	689.21
encoding.com 1 (max+TT+instant)	474	7	474	732.5
encoding.com 3 (max)	1137.75	12	1162	1429
panda	1649.36	7	1640.5	1673.93
sorenson	2087.23	8	2052	2101

Failures

• Sorenson: 1

Test Results: Single Mobile Video Request

The following are the results from a total of 14 mobile video encode jobs submitted at various times of the day over a period of 1 week. NOTE: During our testing, encoding.com jobs would occasionally experience excessive queue times with the status "Waiting for encoder". This is the reason for the long green section representing the transfer/queue time delta on the graph below.

Service	Avg Encode Time	Standard Deviation %	Median Encode Time	Avg Total Time
zencoder	19.38	12	20	30.23
panda	25.69	4	25	26.31
encoding.com 4 (plus+TT+instant)	35	7	34	122.75
encoding.com 3 (max)	40.25	9	40.5	51.25
sorenson	47	6	46	68.31
encoding.com 2 (max+TT)	76.92	139	31	88.23
encoding.com 1 (max+TT+instant)	97	67	97	112

Failures - None

Test Results: Single MP3 Audio Request

The following are the results from a total of 14 audio encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.

Service	Avg Encode Time	Standard Deviation %	Median Encode Time	Avg Total Time
zencoder	120.31	11	115	131
sorenson	189.38	11	182	196.23
panda	218.58	3	217	222.16
encoding.com 1 (max+TT+instant)	221	0	221	255
encoding.com 4 (plus+TT+instant)	224.75	9	225	248.75
encoding.com 3 (max)	240.5	7	238	355.75
encoding.com 2 (max+TT)	328.36	52	254	360.45

Failures

• encoding.com max+TT+instant: 1
• encoding.com max+TT: 2

Test Results: 4 Parallel HD Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) HD video encode jobs submitted at various times of the day over a period of 1 week:

Service	Avg Encode Time	Standard Deviation %	Median Encode Time	Avg Total Time
zencoder	141.23	4	140	153.71
encoding.com 1 (max+TT+instant)	236.88	19	250	280.13
encoding.com 2 (max+TT)	292.21	46	259.5	321.96
encoding.com 4 (plus+TT+instant)	298.69	37	256.5	354.44
encoding.com 3 (max)	723.44	25	751.5	796.38
panda	1322.19	17	1249	1336.9
sorenson	1751.44	3	1745	1769.71

Failures - None

Test Results: 4 Parallel SD Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) SD video encode jobs submitted at various times of the day over a period of 1 week:

Service	Avg Encode Time	Standard Deviation %	Median Encode Time	Avg Total Time
zencoder	240.48	17	224.5	327.98
encoding.com 2 (max+TT)	439.5	21	445	700.15
encoding.com 4 (plus+TT+instant)	461.56	14	459.5	757.87
encoding.com 1 (max+TT+instant)	484.88	10	459	770.01
encoding.com 3 (max)	1274.63	11	1274	1552.69
panda	1680.96	10	1648.5	1710.9
sorenson	3777.54	10	3835	3808.94

Failures - None

Test Results: 4 Parallel Mobile Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) mobile video encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.

Service	Avg Encode Time	Standard Deviation %	Median Encode Time	Avg Total Time
zencoder	20.4	15	20	27.27
panda	26.65	7	26	27.21
encoding.com 1 (max+TT+instant)	34	8	34	49.75
sorenson	55.2	15	53	77.54
encoding.com 4 (plus+TT+instant)	68	60	44.5	106.63
encoding.com 2 (max+TT)	87.46	100	37	104.92
encoding.com 3 (max)	115.88	197	46	149.32

Failures

• sorenson: 8

Test Results: 4 Parallel MP3 Audio Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) audio encode jobs submitted at various times of the day over a period of 1 week:

Service	Avg Encode Time	Standard Deviation %	Median Encode Time	Avg Total Time
zencoder	115.62	6	114	126.16
sorenson	187.24	5	184	204.57
panda	225.15	17	214	227.92
encoding.com 4 (plus+TT+instant)	234.38	17	228.5	266.63
encoding.com 2 (max+TT)	265.43	19	261	313.56
encoding.com 1 (max+TT+instant)	273.67	17	274	344
encoding.com 3 (max)	285.69	31	259	317.15

Failures

• sorenson: 3
• encoding.com max+TT+instant: 5
• encoding.com max+TT: 5
• encoding.com max: 3

Encoding Service Accounts

The following encoding services are included in this analysis: Zencoder, encoding.com, Sorenson Media, and Panda. Each service offers different account and pricing options. We setup an account with each service (including Zencoder) using their standard signup process. Because 4 parallel job requests were part of the testing, we opted for service plans that would limit the effects of queue time under such conditions. The pricing data below is for informational purposes only. The purpose of this post is not to compare service pricing, as there are simply too many variations between services to be able to do so. The following is a summary of the account options we selected and pricing with each service:

Service	Plan Used	Plan Cost	Plan Usage Included	Encoding Cost	Total Test Costs
Zencoder	Launch	$40/mo	1000 Encoding Mins	$0.08/min HD; $0.04/min SD; $0.01/min audio	$91
encoding.com	Max	$299/mo	75GB Encoded Media	NA	$299
Sorenson Media	Squeeze Managed Server	$199/mo	1200 Encoding Mins	$8 per 60 Mins extra	$375 (incl addl encoding mins)
Panda	4 dedicated encoders	$396	Unlimited encoding - $0.15/GB to upload encoded media	NA	$436 (incl bandwidth)

Disclaimer

This comparison was sponsored by Zencoder. In order to sustain our ability to provide useful, free & publicly accessible analysis, we frequently take on paid engagements. However, in doing so, we try to maintain our credibility and objectivity by using reliable tests, being transparent about our test methods, and attempting to represent the data in a fair way.

Summary

In order to maintain our objectivity and independence, we generally do not recommend one service over another. We prefer to simply present the data as it stands, and let readers draw their own conclusions.