Monday, October 17, 2011

Encoding Performance: Comparing Zencoder, Encoding.com, Sorenson & Panda

A few months ago we were approached by Zencoder to conduct a sponsored performance comparison of 4 encoding services including their own. The purpose was to validate their claims of faster encoding times using an independent, credible external source. This was a new frontier for us. Our primary focus has been performance analysis of infrastructure as a service (IaaS). However, we are curious about all things related to cloud and benchmarking and we felt this could be useful data to make available publicly, so we accepted.

Testing Methodology

This is a description of the methodology we used for conducting this performance analysis.

Source Media

Following discussions with Zencoder, we opted to test encoding performance using 4 distinct media types. We were tasked with finding samples for each media type, they were not provided by Zencoder. All source media was stored in AWS S3 using the US East region (the same AWS region each of the 4 encoding services are hosted from). The 4 media types we used for testing are:
  • HD Video: We chose an HD 1080P trailer for the movie Avatar. This file was 223.1 MB in size and 3 mins, 30 secs in duration.
  • SD Video: We chose a 480P video episode from a cartoon series. The file was 519.2 MB in size and about 23 mins in duration.
  • Mobile Video: We created a 568x320 video using an iPhone (source here). The file was 2.9 MB in size, 30 secs in duration.
  • MP3 Audio: We used an MP3 file we found on the web about Yoga (source here). The file was 42.2 MB in size, 58 mins 41 secs in duration.

Encode Settings

We used the same encode options across all of the services tested. The following is a summary of the encode options used for each corresponding media type:
Media Type Video Codec Video Bitrate Audio Codec Audio Bitrate Encode Passes
HD Video H.264 3000 Kb/s AAC 96 Kb/s 2
SD Video H.264 1000 Kb/s AAC 96 Kb/s 2
Mobile Video H.264 500 Kb/s AAC 96 Kb/s 2
MP3 Audio NA NA AAC 96 Kb/s 2

Test Scheduling

Testing was conducted during a span of 1 week. We built a test harness that integrated with the APIs of each of the 4 encoding services. The test harness invoked 2 test iterations daily with each service. Testing included both single request and 4 parallel requests. Each test iteration consisted of the following 8 test scenarios:
  • Single HD Video Request
  • Single SD Video Request
  • Single Mobile Video Request
  • Single MP3 Audio Request
  • 4 Parallel HD Video Requests
  • 4 Parallel SD Video Requests
  • 4 Parallel Mobile Video Requests
  • 4 Parallel MP3 Audio Requests
The order of the test scenarios was randomized, but the same tests were always requested at the same time for each service. Each test was run to completion on all services before the next test was invoked. The start times for 2 daily test iterations were separated by about 12 hours and incremented by 100 minutes each day. The end result was a distribution of test scenarios during many different times of the day. A total of 112 tests were performed during the 1 week test span, followed by an additional 24 test scenarios on encoding.com to test different combinations of service specific encoding options (described below).

Performance Metrics

During testing, we captured the following metrics:
  • Encode Time: The amount of time in seconds required to encode the media (excludes transfer time)
  • Transfer Time: The amount of time in seconds required to transfer the source media from AWS S3 into the service processing queue
  • Failures: When a service failed to complete a job

Test Results

The following is a summary of the results for each of the 8 test scenarios. Result tables have the following columns:
  • Avg Encode Time: The mean (average) encoding time in seconds for all jobs in this test scenario
  • Standard Deviation %: Standard deviation as percentage of the mean. A lower values indicates more consistency in performance
  • Median Encode Time: The median encoding time in seconds for all jobs in this test scenario
  • Avg Total Time: The mean (average) total job time in seconds. The total time is the sum of transfer (source files were hosted in AWS S3 US East), queue and encoding times, essentially the total turnaround time from the moment the job was submitted. Sorenson and Panda may cache source media files, thus reducing transfer time for future requests

A graph is displayed below each results table depicting sorted average encode and total times using a dual series horizontal bar chart. Also depicted on the graph is a line indicating the actual duration of the source media. Encode time bars that terminate to the left of this line signify faster than realtime encoding performance.

encoding.com offers a few different encoding job options that can affect encode performance. These options include the following:

  • instant: using the instant option, encoding.com will begin encoding before the source media has been fully downloaded. For larger source files, this can decrease encode times
  • twin turbo: this setting causes jobs to be delegated to faster servers in exchange for paying a $2/GB premium for encoding. encoding.com states that this option will deliver 6-8X faster encoding time over standard servers

Test Results: Single HD Video Request

The following are the results from a total of 14 HD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 141.08 6 137 155.08
encoding.com 2 (max+TT) 213.38 16 193 244.53
encoding.com 1 (max+TT+instant) 213.5 11 213.5 238
encoding.com 4 (plus+TT+instant) 226.75 11 235.5 258.75
encoding.com 3 (max) 618.75 12 643.5 653.25
sorenson 974.31 2 975 982.46
panda 1246.31 4 1247 1255.93

Failures - None

Test Results: Single SD Video Request

The following are the results from a total of 14 SD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 229.14 9 225.5 281.64
encoding.com 4 (plus+TT+instant) 453 1 453.5 718.75
encoding.com 2 (max+TT) 454.07 11 458.5 689.21
encoding.com 1 (max+TT+instant) 474 7 474 732.5
encoding.com 3 (max) 1137.75 12 1162 1429
panda 1649.36 7 1640.5 1673.93
sorenson 2087.23 8 2052 2101

Failures

  • Sorenson: 1

Test Results: Single Mobile Video Request

The following are the results from a total of 14 mobile video encode jobs submitted at various times of the day over a period of 1 week. NOTE: During our testing, encoding.com jobs would occasionally experience excessive queue times with the status "Waiting for encoder". This is the reason for the long green section representing the transfer/queue time delta on the graph below.
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 19.38 12 20 30.23
panda 25.69 4 25 26.31
encoding.com 4 (plus+TT+instant) 35 7 34 122.75
encoding.com 3 (max) 40.25 9 40.5 51.25
sorenson 47 6 46 68.31
encoding.com 2 (max+TT) 76.92 139 31 88.23
encoding.com 1 (max+TT+instant) 97 67 97 112

Failures - None

Test Results: Single MP3 Audio Request

The following are the results from a total of 14 audio encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 120.31 11 115 131
sorenson 189.38 11 182 196.23
panda 218.58 3 217 222.16
encoding.com 1 (max+TT+instant) 221 0 221 255
encoding.com 4 (plus+TT+instant) 224.75 9 225 248.75
encoding.com 3 (max) 240.5 7 238 355.75
encoding.com 2 (max+TT) 328.36 52 254 360.45

Failures

  • encoding.com max+TT+instant: 1
  • encoding.com max+TT: 2

Test Results: 4 Parallel HD Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) HD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 141.23 4 140 153.71
encoding.com 1 (max+TT+instant) 236.88 19 250 280.13
encoding.com 2 (max+TT) 292.21 46 259.5 321.96
encoding.com 4 (plus+TT+instant) 298.69 37 256.5 354.44
encoding.com 3 (max) 723.44 25 751.5 796.38
panda 1322.19 17 1249 1336.9
sorenson 1751.44 3 1745 1769.71

Failures - None

Test Results: 4 Parallel SD Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) SD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 240.48 17 224.5 327.98
encoding.com 2 (max+TT) 439.5 21 445 700.15
encoding.com 4 (plus+TT+instant) 461.56 14 459.5 757.87
encoding.com 1 (max+TT+instant) 484.88 10 459 770.01
encoding.com 3 (max) 1274.63 11 1274 1552.69
panda 1680.96 10 1648.5 1710.9
sorenson 3777.54 10 3835 3808.94

Failures - None

Test Results: 4 Parallel Mobile Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) mobile video encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 20.4 15 20 27.27
panda 26.65 7 26 27.21
encoding.com 1 (max+TT+instant) 34 8 34 49.75
sorenson 55.2 15 53 77.54
encoding.com 4 (plus+TT+instant) 68 60 44.5 106.63
encoding.com 2 (max+TT) 87.46 100 37 104.92
encoding.com 3 (max) 115.88 197 46 149.32

Failures

  • sorenson: 8

Test Results: 4 Parallel MP3 Audio Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) audio encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 115.62 6 114 126.16
sorenson 187.24 5 184 204.57
panda 225.15 17 214 227.92
encoding.com 4 (plus+TT+instant) 234.38 17 228.5 266.63
encoding.com 2 (max+TT) 265.43 19 261 313.56
encoding.com 1 (max+TT+instant) 273.67 17 274 344
encoding.com 3 (max) 285.69 31 259 317.15

Failures

  • sorenson: 3
  • encoding.com max+TT+instant: 5
  • encoding.com max+TT: 5
  • encoding.com max: 3

Encoding Service Accounts

The following encoding services are included in this analysis: Zencoder, encoding.com, Sorenson Media, and Panda. Each service offers different account and pricing options. We setup an account with each service (including Zencoder) using their standard signup process. Because 4 parallel job requests were part of the testing, we opted for service plans that would limit the effects of queue time under such conditions. The pricing data below is for informational purposes only. The purpose of this post is not to compare service pricing, as there are simply too many variations between services to be able to do so. The following is a summary of the account options we selected and pricing with each service:
Service Plan Used Plan Cost Plan Usage Included Encoding Cost Total Test Costs
Zencoder Launch $40/mo 1000 Encoding Mins $0.08/min HD; $0.04/min SD; $0.01/min audio $91
encoding.com Max $299/mo 75GB Encoded Media NA $299
Sorenson Media Squeeze Managed Server $199/mo 1200 Encoding Mins $8 per 60 Mins extra $375 (incl addl encoding mins)
Panda 4 dedicated encoders $396 Unlimited encoding - $0.15/GB to upload encoded media NA $436 (incl bandwidth)

Disclaimer

This comparison was sponsored by Zencoder. In order to sustain our ability to provide useful, free & publicly accessible analysis, we frequently take on paid engagements. However, in doing so, we try to maintain our credibility and objectivity by using reliable tests, being transparent about our test methods, and attempting to represent the data in a fair way.

Summary

In order to maintain our objectivity and independence, we generally do not recommend one service over another. We prefer to simply present the data as it stands, and let readers draw their own conclusions.
Posted by at
Labels: , ,

12 comments:

  1. YvoOctober 18, 2011 at 3:05 AM

    Interesting test; I'm missing heywatch.com which has been around for a decade (well in internet time) and would make the comparison more authorative imho.

    ReplyDelete
  2. Compression XpertOctober 18, 2011 at 12:07 PM

    This is not a accurate presentation of Encoding.com's Instant encoding feature. If all source video files were on Amazon's US East S3 network than the transfer time to the provider is negligible and does not travel over the public internet. For the vast majority of user's source video content is not already on the cloud and Encoding.com's Instant Encoding.com will produce a huge reduction on overall encoding time by reducing the transit time from source video location to cloud processing center.

    ReplyDelete
  3. UnknownOctober 18, 2011 at 2:06 PM

    Compression Xpert: this test is specifically designed to highlight transcoding time and not file transfer time, which is why S3 was used. For a cloud encoding provider, this test is still important, because a huge (and growing) number of media publishers use Amazon S3 as intermediate hosting.

    A general-purpose test that includes public internet transfer speed would be impossible, because every server and every file will have a different transfer profile. But you could certainly benchmark something like "Transcoding a file delivered via FTP at 100Mbps" and "Transcoding a file deliviered via FTP at 20Mbps". Encoding.com's instant transfer feature would eliminate either (1) the transfer time, or the (2) first-pass transcoding time, whichever is shorter.

    In fact, based on the tests above, we can actually profile what would happen at each provider via FTP given different bitrates. In theory, Encoding.com would be no slower when moving to FTP up to a limit of about 1/3 of encoding time (because the Instant Encoding is limited by the first pass of the encoding process).

    At Zencoder, the average download bandwidth via FTP (not S3) is 80Mbps. At 80Mbps, the HD job would take 21 seconds longer for Zencoder to process and 0 seconds longer for Encoding.com, so times would be 176 seconds at Zencoder and 238 at Encoding.com. Once bandwidth drops to 20Mbps or so, we're tied, and at lower bitrates, we're affected equally.

    The long-form SD video would take 50 seconds longer at Zencoder at 80Mbps and 0 seconds longer at Encoding.com, which make it 331 at Zencoder and 718 at Encoding.com. At about 25Mbps, Zencoder takes 431 seconds, and Encoding.com still takes 718. Below 25Mbps, inbound transfer becomes the bottleneck, and so both companies slow down linearly at that point.

    So when bandwidth is low, Instant Encoding makes a big speed difference. We're big fans of that feature and you'll probably see it at every cloud encoding provider before too long. But when bandwidth is high, transfer time ends up being a minor part of the encoding process, and transcoding speed + queue time is more important.

    ReplyDelete
  4. Compression XpertOctober 19, 2011 at 12:39 PM

    Can you provide more details on the encoding parameters of the test. For example are you using H.264 baseline or high?

    ReplyDelete
  5. AnantJanuary 23, 2012 at 3:56 AM

    Thanks so much for this. This is awesome post I ever seen on internet. This is rare to find that’s why difficult to understand. Anyway, you are definitely someone that has something to say that people need to hear. Keep up the good work. Keep on inspiring the people.
    For more information regards:
    IT Service Provider Benchmarking

    ReplyDelete
  6. AnantJanuary 23, 2012 at 3:57 AM

    Thanks so much for this. This is awesome post I ever seen on internet. This is rare to find that’s why difficult to understand. Anyway, you are definitely someone that has something to say that people need to hear. Keep up the good work. Keep on inspiring the people.
    For more information regards:
    IT Service Provider Benchmarking

    ReplyDelete
  7. AnantJanuary 23, 2012 at 3:57 AM

    Thanks so much for this. This is awesome post I ever seen on internet. This is rare to find that’s why difficult to understand. Anyway, you are definitely someone that has something to say that people need to hear. Keep up the good work. Keep on inspiring the people.
    For more information regards:
    IT Service Provider Benchmarking

    ReplyDelete
  8. TimApril 19, 2012 at 8:21 PM

    There are faster encoding services out there..
    You should include Gomi Cloud Video Encoding Platform in your comparison.

    ReplyDelete
  9. UnknownJanuary 25, 2013 at 11:23 PM

    Do you also compare video quality?

    ReplyDelete
  10. UnknownJanuary 25, 2013 at 11:23 PM

    Do you also compare video quality?

    ReplyDelete
  11. HeyWatchFebruary 5, 2013 at 6:34 AM

    Thanks for this benchmark even though we are not in it! Actually, it is a little bit odd as HeyWatch has been the pioneer in Cloud Video Encoding since 2006. I guess you should start this benchmark again considering all the providers so that this test could 100% objective and independent. Is it possible? Please let me know.

    ReplyDelete
  12. Better EncodingMarch 3, 2013 at 9:19 AM

    I totally agree with HeyWatch.
    They definitely should be there and it's not the first time they are being ignored on these kind of tests, and with no legitimate reason.
    I want to add 2 more services that weren't included:

    http://www.BetterEncoding.com
    and
    http://www.GridVid.me


    ReplyDelete

Subscribe to: Post Comments (Atom)