Testing Methodology
This is a description of the methodology we used for conducting this performance analysis.Source Media
Following discussions with Zencoder, we opted to test encoding performance using 4 distinct media types. We were tasked with finding samples for each media type, they were not provided by Zencoder. All source media was stored in AWS S3 using the US East region (the same AWS region each of the 4 encoding services are hosted from). The 4 media types we used for testing are:- HD Video: We chose an HD 1080P trailer for the movie Avatar. This file was 223.1 MB in size and 3 mins, 30 secs in duration.
- SD Video: We chose a 480P video episode from a cartoon series. The file was 519.2 MB in size and about 23 mins in duration.
- Mobile Video: We created a 568x320 video using an iPhone (source here). The file was 2.9 MB in size, 30 secs in duration.
- MP3 Audio: We used an MP3 file we found on the web about Yoga (source here). The file was 42.2 MB in size, 58 mins 41 secs in duration.
Encode Settings
We used the same encode options across all of the services tested. The following is a summary of the encode options used for each corresponding media type:Media Type | Video Codec | Video Bitrate | Audio Codec | Audio Bitrate | Encode Passes |
---|---|---|---|---|---|
HD Video | H.264 | 3000 Kb/s | AAC | 96 Kb/s | 2 |
SD Video | H.264 | 1000 Kb/s | AAC | 96 Kb/s | 2 |
Mobile Video | H.264 | 500 Kb/s | AAC | 96 Kb/s | 2 |
MP3 Audio | NA | NA | AAC | 96 Kb/s | 2 |
Test Scheduling
Testing was conducted during a span of 1 week. We built a test harness that integrated with the APIs of each of the 4 encoding services. The test harness invoked 2 test iterations daily with each service. Testing included both single request and 4 parallel requests. Each test iteration consisted of the following 8 test scenarios:- Single HD Video Request
- Single SD Video Request
- Single Mobile Video Request
- Single MP3 Audio Request
- 4 Parallel HD Video Requests
- 4 Parallel SD Video Requests
- 4 Parallel Mobile Video Requests
- 4 Parallel MP3 Audio Requests
Performance Metrics
During testing, we captured the following metrics:- Encode Time: The amount of time in seconds required to encode the media (excludes transfer time)
- Transfer Time: The amount of time in seconds required to transfer the source media from AWS S3 into the service processing queue
- Failures: When a service failed to complete a job
Test Results
The following is a summary of the results for each of the 8 test scenarios. Result tables have the following columns:- Avg Encode Time: The mean (average) encoding time in seconds for all jobs in this test scenario
- Standard Deviation %: Standard deviation as percentage of the mean. A lower values indicates more consistency in performance
- Median Encode Time: The median encoding time in seconds for all jobs in this test scenario
- Avg Total Time: The mean (average) total job time in seconds. The total time is the sum of transfer (source files were hosted in AWS S3 US East), queue and encoding times, essentially the total turnaround time from the moment the job was submitted. Sorenson and Panda may cache source media files, thus reducing transfer time for future requests
A graph is displayed below each results table depicting sorted average encode and total times using a dual series horizontal bar chart. Also depicted on the graph is a line indicating the actual duration of the source media. Encode time bars that terminate to the left of this line signify faster than realtime encoding performance.
encoding.com offers a few different encoding job options that can affect encode performance. These options include the following:
- instant: using the instant option, encoding.com will begin encoding before the source media has been fully downloaded. For larger source files, this can decrease encode times
- twin turbo: this setting causes jobs to be delegated to faster servers in exchange for paying a $2/GB premium for encoding. encoding.com states that this option will deliver 6-8X faster encoding time over standard servers
Test Results: Single HD Video Request
The following are the results from a total of 14 HD video encode jobs submitted at various times of the day over a period of 1 week:Service | Avg Encode Time | Standard Deviation % | Median Encode Time | Avg Total Time |
---|---|---|---|---|
zencoder | 141.08 | 6 | 137 | 155.08 |
encoding.com 2 (max+TT) | 213.38 | 16 | 193 | 244.53 |
encoding.com 1 (max+TT+instant) | 213.5 | 11 | 213.5 | 238 |
encoding.com 4 (plus+TT+instant) | 226.75 | 11 | 235.5 | 258.75 |
encoding.com 3 (max) | 618.75 | 12 | 643.5 | 653.25 |
sorenson | 974.31 | 2 | 975 | 982.46 |
panda | 1246.31 | 4 | 1247 | 1255.93 |
Failures - None
Test Results: Single SD Video Request
The following are the results from a total of 14 SD video encode jobs submitted at various times of the day over a period of 1 week:Service | Avg Encode Time | Standard Deviation % | Median Encode Time | Avg Total Time |
---|---|---|---|---|
zencoder | 229.14 | 9 | 225.5 | 281.64 |
encoding.com 4 (plus+TT+instant) | 453 | 1 | 453.5 | 718.75 |
encoding.com 2 (max+TT) | 454.07 | 11 | 458.5 | 689.21 |
encoding.com 1 (max+TT+instant) | 474 | 7 | 474 | 732.5 |
encoding.com 3 (max) | 1137.75 | 12 | 1162 | 1429 |
panda | 1649.36 | 7 | 1640.5 | 1673.93 |
sorenson | 2087.23 | 8 | 2052 | 2101 |
Failures
Test Results: Single Mobile Video Request
The following are the results from a total of 14 mobile video encode jobs submitted at various times of the day over a period of 1 week. NOTE: During our testing, encoding.com jobs would occasionally experience excessive queue times with the status "Waiting for encoder". This is the reason for the long green section representing the transfer/queue time delta on the graph below.Service | Avg Encode Time | Standard Deviation % | Median Encode Time | Avg Total Time |
---|---|---|---|---|
zencoder | 19.38 | 12 | 20 | 30.23 |
panda | 25.69 | 4 | 25 | 26.31 |
encoding.com 4 (plus+TT+instant) | 35 | 7 | 34 | 122.75 |
encoding.com 3 (max) | 40.25 | 9 | 40.5 | 51.25 |
sorenson | 47 | 6 | 46 | 68.31 |
encoding.com 2 (max+TT) | 76.92 | 139 | 31 | 88.23 |
encoding.com 1 (max+TT+instant) | 97 | 67 | 97 | 112 |
Failures - None
Test Results: Single MP3 Audio Request
The following are the results from a total of 14 audio encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.Service | Avg Encode Time | Standard Deviation % | Median Encode Time | Avg Total Time |
---|---|---|---|---|
zencoder | 120.31 | 11 | 115 | 131 |
sorenson | 189.38 | 11 | 182 | 196.23 |
panda | 218.58 | 3 | 217 | 222.16 |
encoding.com 1 (max+TT+instant) | 221 | 0 | 221 | 255 |
encoding.com 4 (plus+TT+instant) | 224.75 | 9 | 225 | 248.75 |
encoding.com 3 (max) | 240.5 | 7 | 238 | 355.75 |
encoding.com 2 (max+TT) | 328.36 | 52 | 254 | 360.45 |
Failures
Test Results: 4 Parallel HD Video Requests
The following are the results from a total of 56 (14 sets of 4 parallel requests) HD video encode jobs submitted at various times of the day over a period of 1 week:Service | Avg Encode Time | Standard Deviation % | Median Encode Time | Avg Total Time |
---|---|---|---|---|
zencoder | 141.23 | 4 | 140 | 153.71 |
encoding.com 1 (max+TT+instant) | 236.88 | 19 | 250 | 280.13 |
encoding.com 2 (max+TT) | 292.21 | 46 | 259.5 | 321.96 |
encoding.com 4 (plus+TT+instant) | 298.69 | 37 | 256.5 | 354.44 |
encoding.com 3 (max) | 723.44 | 25 | 751.5 | 796.38 |
panda | 1322.19 | 17 | 1249 | 1336.9 |
sorenson | 1751.44 | 3 | 1745 | 1769.71 |
Failures - None
Test Results: 4 Parallel SD Video Requests
The following are the results from a total of 56 (14 sets of 4 parallel requests) SD video encode jobs submitted at various times of the day over a period of 1 week:Service | Avg Encode Time | Standard Deviation % | Median Encode Time | Avg Total Time |
---|---|---|---|---|
zencoder | 240.48 | 17 | 224.5 | 327.98 |
encoding.com 2 (max+TT) | 439.5 | 21 | 445 | 700.15 |
encoding.com 4 (plus+TT+instant) | 461.56 | 14 | 459.5 | 757.87 |
encoding.com 1 (max+TT+instant) | 484.88 | 10 | 459 | 770.01 |
encoding.com 3 (max) | 1274.63 | 11 | 1274 | 1552.69 |
panda | 1680.96 | 10 | 1648.5 | 1710.9 |
sorenson | 3777.54 | 10 | 3835 | 3808.94 |
Failures - None
Test Results: 4 Parallel Mobile Video Requests
The following are the results from a total of 56 (14 sets of 4 parallel requests) mobile video encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.Service | Avg Encode Time | Standard Deviation % | Median Encode Time | Avg Total Time |
---|---|---|---|---|
zencoder | 20.4 | 15 | 20 | 27.27 |
panda | 26.65 | 7 | 26 | 27.21 |
encoding.com 1 (max+TT+instant) | 34 | 8 | 34 | 49.75 |
sorenson | 55.2 | 15 | 53 | 77.54 |
encoding.com 4 (plus+TT+instant) | 68 | 60 | 44.5 | 106.63 |
encoding.com 2 (max+TT) | 87.46 | 100 | 37 | 104.92 |
encoding.com 3 (max) | 115.88 | 197 | 46 | 149.32 |
Failures
Test Results: 4 Parallel MP3 Audio Requests
The following are the results from a total of 56 (14 sets of 4 parallel requests) audio encode jobs submitted at various times of the day over a period of 1 week:Service | Avg Encode Time | Standard Deviation % | Median Encode Time | Avg Total Time |
---|---|---|---|---|
zencoder | 115.62 | 6 | 114 | 126.16 |
sorenson | 187.24 | 5 | 184 | 204.57 |
panda | 225.15 | 17 | 214 | 227.92 |
encoding.com 4 (plus+TT+instant) | 234.38 | 17 | 228.5 | 266.63 |
encoding.com 2 (max+TT) | 265.43 | 19 | 261 | 313.56 |
encoding.com 1 (max+TT+instant) | 273.67 | 17 | 274 | 344 |
encoding.com 3 (max) | 285.69 | 31 | 259 | 317.15 |
Failures
Encoding Service Accounts
The following encoding services are included in this analysis: Zencoder, encoding.com, Sorenson Media, and Panda. Each service offers different account and pricing options. We setup an account with each service (including Zencoder) using their standard signup process. Because 4 parallel job requests were part of the testing, we opted for service plans that would limit the effects of queue time under such conditions. The pricing data below is for informational purposes only. The purpose of this post is not to compare service pricing, as there are simply too many variations between services to be able to do so. The following is a summary of the account options we selected and pricing with each service:Service | Plan Used | Plan Cost | Plan Usage Included | Encoding Cost | Total Test Costs |
---|---|---|---|---|---|
Zencoder | Launch | $40/mo | 1000 Encoding Mins | $0.08/min HD; $0.04/min SD; $0.01/min audio | $91 |
encoding.com | Max | $299/mo | 75GB Encoded Media | NA | $299 |
Sorenson Media | Squeeze Managed Server | $199/mo | 1200 Encoding Mins | $8 per 60 Mins extra | $375 (incl addl encoding mins) |
Panda | 4 dedicated encoders | $396 | Unlimited encoding - $0.15/GB to upload encoded media | NA | $436 (incl bandwidth) |