Monday, May 6, 2013

DNS Marketshare - Alexa 10,000 + Fortune 500 - May 2013

The following are updated managed DNS marketshare statistics for May 2013. This is a follow up to our previous posts in March and April 2013.

Alexa Top 10,000 DNS Marketshare - May 6, 2013

The following table provides a comparison of DNS provider marketshare between Apr 8 and May 6 2013 for the top 10,000 Alexa websites. Alexa rankings change frequently. This analysis is based on an Alexa rankings snapshot taken on April 5 2013.

AWS Route 53 and CloudFlare continue to grow significantly in this market segment.

Akamai's marketshare growth is primarily being driven by migrations off the Cotendo platform which is scheduled to be shutdown sometime this year.

ProviderRankWebsites (out of 10,000)MarketshareMarketshare Change
DynECT14404.4%+6 / +1.382%
AWS Route 5323813.81%+14 / +3.815%
UltraDNS33613.61%-2 / -0.551%
DNSPod43363.36%+5 / +1.511%
CloudFlare53143.14%+23 / +7.904%
GoDaddy DNS62872.87%-10 / -3.367%
DNS Made Easy72462.46%0
Akamai82172.17%+10 / +4.831%
Rackspace Cloud DNS91561.56%-2 / -1.266%
Verisign DNS101061.06%+5 / +4.95%
Softlayer DNS11790.79%0
Namecheap12760.76%0
easyDNS13760.76%-1 / -1.299%
Enom DNS14660.66%-1 / -1.493%
Cotendo Advanced DNS15470.47%-11 / -18.966%
Savvis16420.42%0
Nettica17300.3%0
ZoneEdit18290.29%0
Internap19270.27%0
ClouDNS20210.21%+3 / +16.667%
DNS Park21170.17%+1 / +6.25%
No-IP22120.12%0
Zerigo DNS23100.1%0
EuroDNS2470.07%0
Worldwide DNS2550.05%-1 / -16.667%
DTDNS2620.02%0
CDNetworks DNS2720.02%+1 / +100%


Alexa Top 1,000 DNS Marketshare - May 6, 2013

The following table provides a comparison of DNS provider marketshare between Apr 8 and May 6 2013 for the top 1,000 Alexa websites. Alexa rankings change frequently. This analysis is based on an Alexa rankings snapshot taken on April 5 2013.

This market segment is much less volatile month-to-month relative to the top 10,000.

ProviderRankWebsites (out of 1,000)MarketshareMarketshare Change
DynECT1797.9%0
UltraDNS2636.3%+1 / +1.613%
Akamai3484.8%0
AWS Route 534343.4%-1 / -2.857%
DNSPod5323.2%0
DNS Made Easy6212.1%0
GoDaddy DNS7141.4%0
Cotendo Advanced DNS8111.1%-1 / -8.333%
Verisign DNS9101%0
easyDNS10101%0
CloudFlare1180.8%+1 / +14.286%
Rackspace Cloud DNS1270.7%0
Namecheap1360.6%0
Softlayer DNS1450.5%0
Enom DNS1550.5%0
Internap1630.3%0
Savvis1730.3%0
Nettica1820.2%0
ClouDNS1920.2%0
ZoneEdit2020.2%0
DTDNS2110.1%0
EuroDNS2210.1%0
No-IP2310.1%0
Worldwide DNS2410.1%0


Fortune 500 DNS Marketshare - May 6, 2013

The following table provides a comparison of DNS provider marketshare between Apr 8 and May 6 2013 for the 2012 US Fortune 500 Companies. Again, very little movement in this market segment.

ProviderRankWebsites (out of 500)MarketshareMarketshare Change
UltraDNS1367.2%+1 / +2.857%
Verisign DNS2244.8%0
Akamai3132.6%0
DynECT481.6%0
DNS Made Easy561.2%0
Savvis640.8%0
GoDaddy DNS740.8%0
Internap840.8%0
Rackspace Cloud DNS920.4%0
AWS Route 531020.4%0
easyDNS1110.2%0
No-IP1210.2%0
Enom DNS1310.2%0
ZoneEdit1410.2%0


Top 20 Alexa Site Changes - Apr 8 to May 6 2013

As part of the marketshare analysis, we track which Alexa 10,000 sites have switched DNS providers since the last snapshot.

Interesting to note that indiatimes.com migrated back to UltraDNS after having switched to from Ultra to Dyn just one month ago.

HostnameRankNew DNS ProviderPrevious DNS Provider
indiatimes.com128UltraDNSDynECT
goodreads.com312UltraDNSGoDaddy DNS
lifehacker.com586DynECTCotendo Advanced DNS
gawker.com1138DynECTCotendo Advanced DNS
elwatannews.com1307CloudFlareDynECT
payoneer.com1418AkamaiCotendo Advanced DNS
speedbit.com1500AkamaiCotendo Advanced DNS
kotaku.com1600DynECTCotendo Advanced DNS
shop.com1650Verisign DNSUltraDNS
cafemom.com1660AWS Route 53GoDaddy DNS
io9.com1875DynECTCotendo Advanced DNS
tokyo-porn-tube.com2014AWS Route 53GoDaddy DNS
jezebel.com2106DynECTCotendo Advanced DNS
r2games.com2383DNSPodGoDaddy DNS
nomorerack.com2688UltraDNSAWS Route 53
tokyo-tube.com2721AWS Route 53GoDaddy DNS
socialmediabar.com2884CloudFlareRackspace Cloud DNS
unfranchise.com.tw2896Verisign DNSUltraDNS
sevenforums.com3110GoDaddy DNSDynECT
tasteofhome.com3233AWS Route 53UltraDNS

Monday, April 8, 2013

DNS Marketshare - Alexa 10,000 + Fortune 500 - April 2013

This is a followup from last month's DNS marketshare post. In this post we use the same top 10,000 Alexa websites from last month (based on a 10 Mar 2013 Alexa snapshot). We then mined the DNS servers for each of these websites again using the same technique. The tables below represent the changes in marketshare and DNS hosting occurring during this time period (about 30 days).

New to this post is marketshare analysis for 2012 US Fortune 500 companies. This is a list published annually by Fortune Magazine ranking public US companies by gross revenue. For this analysis, we used the corporate websites listed for these companies on the CNN.com website. This marketshare analysis shows a very different provider makeup relative to top Alexa websites.

Notable during the past 30 days was a large scale DNS DDoS attack against spam blacklist provider spamhaus. To avert this attack, spamhaus switched DNS hosting to CloudFlare who successfully worked internally and with upstream providers to defend spamhaus against the attack as documented on their blog. The resulting positive press, appears to have attracted many new customers with their marketshare increasing by nearly 9% (22 Alexa 10,000 websites).

Fortune 500 DNS Marketshare - Apr 8, 2013

The following table provides a comparison of DNS provider marketshare between Mar 5 and Apr 8 2013 for the 2012 US Fortune 500 Companies.

ProviderRankWebsites (out of 500)MarketshareMarketshare Change
UltraDNS1357%+1 / +2.941%
Verisign DNS2244.8%0
Akamai3132.6%0
DynECT481.6%0
DNS Made Easy561.2%0
Savvis640.8%0
GoDaddy DNS740.8%0
Internap840.8%0
Rackspace Cloud DNS920.4%0
AWS Route 531020.4%+1 / +100%

Alexa Top 1,000 DNS Marketshare - Apr 8, 2013

The following table provides a comparison of DNS provider marketshare between Mar 5 and Apr 8 2013 for the top 1,000 Alexa websites.

ProviderRankWebsites (out of 1,000)MarketshareMarketshare Change
DynECT1787.8%+5 / +6.849%
UltraDNS2626.2%0
Akamai3494.9%+3 / +6.522%
AWS Route 534363.6%-1 / -2.703%
DNSPod5282.8%0
DNS Made Easy6212.1%0
GoDaddy DNS7131.3%0
Cotendo Advanced DNS8111.1%-1 / -8.333%
Verisign DNS9111.1%+1 / +10%
easyDNS10101%0
Rackspace Cloud DNS1180.8%0
CloudFlare1280.8%+2 / +33.333%
Enom DNS1370.7%0
Namecheap1460.6%0
Softlayer DNS1550.5%0
Internap1640.4%0
Savvis1730.3%0
Nettica1820.2%0
ClouDNS1920.2%0
ZoneEdit2020.2%0
DTDNS2110.1%0
EuroDNS2210.1%0
Worldwide DNS2310.1%0
No-IP2410.1%0

Alexa Top 10,000 DNS Marketshare - Mar 5, 2013

The following table provides a comparison of DNS provider marketshare between Mar 5 and Apr 8 2013 for the top 10,000 Alexa websites. Notable here is AWS Route 53 replacing UltraDNS in the #2 slot and CloudFlare's large marketshare increase.

ProviderRankWebsites (out of 10,000)MarketshareMarketshare Change
DynECT14354.35%+13 / +3.081%
AWS Route 5323703.7%+10 / +2.778%
UltraDNS33663.66%-1 / -0.272%
DNSPod43293.29%+4 / +1.231%
GoDaddy DNS52892.89%0
CloudFlare62822.82%+22 / +8.462%
DNS Made Easy72472.47%+3 / +1.23%
Akamai82142.14%+6 / +2.885%
Rackspace Cloud DNS91561.56%-2 / -1.266%
Verisign DNS101001%+4 / +4.167%
Softlayer DNS11800.8%-1 / -1.235%
easyDNS12740.74%-4 / -5.128%
Namecheap13720.72%0
Enom DNS14700.7%0
Cotendo Advanced DNS15570.57%-2 / -3.39%
Savvis16430.43%0
ZoneEdit17330.33%0
Internap18290.29%0
Nettica19280.28%+1 / +3.704%
ClouDNS20180.18%+2 / +12.5%
DNS Park21170.17%0
Zerigo DNS22100.1%0
No-IP23100.1%0
EuroDNS2490.09%+1 / +12.5%
Worldwide DNS2560.06%0
DTDNS2620.02%-1 / -33.333%

Top 20 Alexa Site Changes - Mar 5 to Apr 8 2013

As part of the marketshare analysis, we track which Alexa 10,000 sites have switched DNS providers since the last snapshot.

HostnameRankNew DNS ProviderPrevious DNS Provider
indiatimes.com133DynECTUltraDNS
iminent.com308AkamaiCotendo Advanced DNS
sergey-mavrodi.com452CloudFlareAWS Route 53
myntra.com906UltraDNSAWS Route 53
bookmyshow.com1930DynECTUltraDNS
apserver.net1952AWS Route 53easyDNS
template-help.com2051CloudFlareeasyDNS
clubpenguin.com2125DynECTeasyDNS
prntscr.com2202CloudFlareGoDaddy DNS
naturalnews.com2302Softlayer DNSGoDaddy DNS
freemp3x.com2497CloudFlareClouDNS
iptorrents.com2671CloudFlareNamecheap
ycombinator.com2888CloudFlareeasyDNS
dawanda.com3050CloudFlareCotendo Advanced DNS
yoast.com3364AWS Route 53Softlayer DNS
gratisjuegos.org4259CloudFlareSoftlayer DNS
cucirca.com4299AWS Route 53CloudFlare
frontdoor.com4374AWS Route 53Verisign DNS
awwwards.com4393DNS Made EasyRackspace Cloud DNS
hotklix.com4557UltraDNSDynECT

Wednesday, March 6, 2013

DNS Provider Marketshare Summary - March 2013

Every few months we mine the DNS servers used by the top 10,000 Alexa sites. We then use various techniques including hostname and IP class C matching, and DNS lookup validations to correlate the DNS servers with specific DNS services like UltraDNS, Dyn and AWS Route 53. We call this point-in-time analysis a DNS marketshare snapshot. Finally we compare current and prior DNS marketshare snapshots to identify market trends and changes.

Steady DNS Outsourcing Trend

Outsourcing of DNS hosting to managed providers has grown consistently for Alexa 10,000 sites since we began capturing marketshare snapshots last year. Our analysis shows an increase of 6% in the past 4 months, and 15% in the past 11 months. Such growth appears to be favoring some services over others, with many even loosing marketshare. The following table provides a summary of managed DNS marketshare changes for the past 11 months. Only services with greater than 1% marketshare are displayed. The table is sorted by current Alexa 10,000 marketshare.

DNS Service Marketshare Change: Apr 2012 - Mar 2013
DNS ServiceService TypeAlexa 1,000 Marketshare ChangeAlexa 10,000 Marketshare Change
DynEnterprise5.1% to 7.1%+39%2.72% to 3.91%+44%
UltraDNSEnterprise5.7% to 6.2%+9%3.73% to 3.7%-1%
AWS Route 53Self Serve1.8% to 3.8%+111%1.68% to 3.56%+112%
DNSPodSelf Serve2.6% to 2.2%-15%3.4% to 3.1%+9%
GoDaddySelf Serve1.5% to 1.3%-15%2.89% to 2.89%+0%
CloudFlareSelf Serve0.3% to 0.6%+100%0.92% to 2.56%+178%
DNS Made EasySelf Serve2% to 2.1%+5%2.47% to 2.43%-2%
AkamaiEnterprise4.2% to 4.7%+12%1.84% to 2.06%+12%
Rackspace Cloud DNSSelf Serve1.3% to 0.7%-86%1.48% to 1.61%+9%
VerisignEnterprise0.7% to 1%+43%0.7% to 0.97%+39%
easyDNSSelf Serve0.6% to 1%+67%0.82% to 0.79%-4%
Cotendo (acquired by Akamai)Enterprise2.1% to 1.2%-75%0.76% to 0.6%-27%

Alexa Top 1,000 DNS Marketshare - Mar 5, 2013

The following table provides current marketshare analysis for the top 1,000 Alexa website based on a snapshot taken on March 5, 2013.

ProviderRankNumber of SitesMarketshare Change
Nov 2012 to Mar 2013
Market Share %
Dyn Enterprise171+107.1%
UltraDNS262+36.2%
Akamai347+64.7%
AWS Route 53438+133.8%
DNSPod52202.2%
DNS Made Easy621-42.1%
GoDaddy713-11.3%
Cotendo812-61.2%
EasyDNS910+21%
Versign1010+21%
Rackspace Cloud117-20.7%
CloudFlare126+10.6%
eNom136-30.6%
Namecheap145+10.5%
SoftLayer15500.5%
Internap164+10.4%
Savvis173-10.3%
Nettica182-20.2%
ClouDNS19200.2%
ZoneEdit202-30.2%
EuroDNS22100.1%

Alexa Top 10,000 DNS Marketshare - Mar 5, 2013

The following table provides current marketshare analysis for the top 10,000 Alexa website based on a snapshot taken on March 5, 2013.

ProviderRankNumber of SitesMarketshare Change
Nov 2012 to Mar 2013
Market Share %
Dyn Enterprise1391+523.91%
UltraDNS2370-173.7%
AWS Route 533356+573.56%
DNSPod4310-23.1%
GoDaddy5289+112.89%
CloudFlare6256+772.56%
DNS Made Easy7243-42.43%
Akamai8206+252.06%
Rackspace Cloud9161+31.61%
Versign1097+190.97%
SoftLayer1182+30.82%
Easy1279-10.79%
Namecheap1373+140.73%
eNom1470-100.7%
Cotendo1560-70.6%
Savvis164300.43%
ZoneEdit1734-60.34%
DynDNS1830-20.3%
Internap1929-10.29%
Nettica2028-30.28%
ClouDNS2119+30.19%
Zerigo DNS231200.12%
No-IP249-60.09%
EuroDNS258+10.08%

Top 20 Alexa Site Changes - Nov 1 2012 to Mar 5 2013

As part of the marketshare analysis, we track which Alexa 10,000 sites have switched managed DNS providers since the last snapshot.

WebsiteAlexa RankNew Provider - 2013-03-05Previous Provider
adf.ly79CloudFlareZoneEdit
adcash.com111Dyn EnterpriseGoDaddy
dropbox.com123AWS Route 53Nettica
addthis.com170Dyn EnterpriseAkamai
sergey-mavrodi.com384AWS Route 53ZoneEdit
agoda.com393VersignUltraDNS
zillow.com497Dyn EnterpriseCotendo
bleacherreport.com510AWS Route 53Dyn Enterprise
swagbucks.com520AWS Route 53Cotendo
odesk.com540AkamaiDNS Made Easy
retailmenot.com560UltraDNSAWS Route 53
adscale.de564AkamaiCotendo
timeanddate.com589AkamaiCotendo
kickstarter.com774AWS Route 53DNS Made Easy
twitch.tv846VersignAkamai
twitch.tv846Dyn EnterpriseAkamai
dpreview.com856AWS Route 53UltraDNS
piriform.com935AWS Route 53GoDaddy
docstoc.com994Dyn EnterpriseDynDNS
bidorbuy.co.za1003Dyn EnterpriseUltraDNS

Confirmed Provider Changes - Nov 1 2012 to Mar 5 2013

Because the makeup of the Alexa top 10,000 sites changes frequently, the stats above may not reflect actual marketshare changes. This table lists Alexa 10,000 marketshare gains or losses our analysis has confirmed for the past 4 months.

ProviderGain/Loss% of Market Share
Dyn Enterprise+27+6.91%
AWS Route 53+18+5.06%
Versign+18+18.56%
CloudFlare+12+4.69%
Akamai+3+1.46%
eNom00%
Easy DNS00%
DNS Made Easy-1-0.41%
EuroDNS-1-12.5%
SoftLayer-1-1.22%
Savvis-1-2.33%
No-IP-2-22.22%
Nettica-2-7.14%
Namecheap-2-2.74%
ZoneEdit-5-14.71%
GoDaddy-10-3.46%
Cotendo-13-21.67%
UltraDNS-36-9.73%

Tuesday, August 21, 2012

Comparison and Analysis of Managed DNS Providers

Introduction

This blog post is the culmination of year's effort researching and developing methods for analyzing and comparing managed DNS services. During this time, we provided access to our analysis and solicited feedback from any DNS provider that would listen. We would like to thank our contacts with UltraDNS, Cotendo, Amazon Web Services and NetDNA for the feedback they provided. As always, it is our intent to provide objective, fair and actionable analysis. We have not been paid by anyone to conduct this testing or write this post. If you have feedback, we'd like to hear it.

This blog post is also intended as an introduction to a new monthly report entitled State of the Cloud - DNS available for purchase on our website. The full version of the August 2012 edition of this report is available for free in html format or pdf format (10MB). Future editions of this report will include both free and premium editions where the free version will include everything but some of the more advanced content such as marketshare analysis. This blog post provides an introduction to and summary of the August 2012 edition of this report.

View Full Report - August 2012

What is DNS?

Domain Name System or DNS for short, is the part of the Internet that lets users access websites (and other Internet services) using easy to remember words and phrases called hostnames like amazon.com or google.com. Without DNS, users would be required to use cryptic numeric-based identifiers called IP addresses (e.g. 72.21.194.1). When a user types a hostname into their browser address bar, one of the first steps undergone is to translate the hostname to an IP address. This translation process involves querying a DNS server that has been assigned responsibility for that hostname. These are called authoritative DNS servers. If the authoritative DNS server is not accessible, the browser will be unable to resolve the IP address and display the website.

DNS server software is freely available and is not overly complex to setup or run. The core functionality of a DNS server is simple… the translation of hostnames to IP addresses. It requires only minimal bandwidth and CPU resources to maintain a DNS server. Many organizations host their own DNS servers without much effort.

Managed DNS

Managed DNS is a service that allows organizations to outsource DNS to a third party provider. There many reasons why an organization may elect to outsource DNS hosting... here are a few:

  • Simplicity Organizations don't have to worry about setting up and maintaining their own DNS servers. Management of DNS records is also easier because providers enable this using a simple browser-based GUI or API
  • Performance Providers that specialize in DNS have often invested significant time and capital setting up global networks of servers that can respond quickly to DNS queries regardless of a user's location
  • Availability Managed DNS providers employ dedicated staff to monitor and maintain highly available DNS services and are often better equipped to handle service anomalies like DDOS attacks
  • Advanced Features Managed DNS providers often offer features that are not part of the standard DNS stack such as integrated monitoring and failover and geographic load balancing

Whatever the reasons are, managed DNS is a fast growing sector in the cloud.

Enterprise versus Self Service

Managed DNS providers can be generally divided into two categories:

  • Enterprise providers typically offer more advanced features, personalized support and account management, and often have larger DNS server networks. These providers typically utilize a formal sales and contract negotiation process for new customers where pricing is variable depending on the customer's negotiating prowess, usage volume and term commitment. Pricing is typically orders of magnitude higher than self service providers. Some enterprise providers offer low volume, low cost introductory packages that are lead-ins to their standard service offerings
  • Self Service providers typically offer simple, contract free, self management DNS services. Pricing is often catered more towards smaller organizations with limited budgets. Self service providers usually (but not always) have smaller DNS server networks and offer fewer advanced features. Based on our analysis, these services are generally as reliable as enterprise services

After speaking with multiple enterprise providers, it is our impression that they generally consider self service providers as non-competitors targeting a different customer demographic.

Comparing Managed DNS Services

Comparing DNS services is not as simple as running a few benchmarks and calling it good. There are multiple criteria where comparisons may be drawn. In this post, we'll present some criteria we believe to be relevant, the techniques we have used to implement them, and the resulting analysis. The following DNS providers are included:

  • Neustar UltraDNS is one of the oldest managed DNS providers founded in 1999. Their network includes 16 DNS POPs (points of presence) on 6 continents. UltraDNS is a leading provider in marketshare with 403 of the Alexa top 10,000 sites according to our recent analysis
  • Dyn has evolved over the years from offering various free DNS services to its current form as an enterprise DNS provider. Although they still support a self service DNS under the DynDNS brand, our analysis includes only their enterprise service. The Dyn network consists of 17 DNS POPs in 4 continents. Dyn's enterprise service is slightly behind UltraDNS in marketshare with 319 of the Alexa top 10,000 sites according to our analysis.
  • Cotendo/Akamai Cotendo was acquired by Akamai in 2012. The Cotendo DNS network consists of 29 DNS POPs in 5 continents. Combining Akamai and Cotendo DNS makes them the leading provider in marketshare for Alexa top 1,000 sites according to our analysis. Akamai's DNS service, Enhanced DNS currently utilizes different DNS infrastructure from Cotendo and is presented separately in this post
  • AWS Route 53 is part of the Amazon Web Services suite of cloud services. It launched in 2011 and is the newest service included in this post. Route 53 uses a self-service, low cost model. The DNS network consists of 33 DNS POPs in 5 continents. Route 53 marketshare has grown significantly in 2012 according to our analysis. It currently lacks many of the more advanced features offered by enterprise providers including DNSSEC and integrated monitoring
  • easyDNS is a smaller, self-service provider founded in 1998. Their network consists of of 16 DNS POPs in 3 continents
  • DNS Made Easy is another smaller, self-service DNS provider founded in 2002. Their network consists of 12 DNS POPs in 3 continents

End-User Performance

There are many factors that affect DNS performance. When a user types a hostname into a browser address bar the path taken for resolving that hostname to an IP address varies between users. Generally, the first point is the user's ISP DNS resolver. These are specialized DNS servers that cache lookups and are used only for DNS resolving. If this DNS server does not have an answer, it will query the next level up (often another ISP DNS server), and this process continues until the authoritative server is queried. This process is referred to as a recursive DNS lookup. From an end-user's perspective, DNS performance is the total lookup time including the entire recursive chain. To improve performance, managed DNS providers typically deploy mnay DNS servers globally and use an Anycast network to reduce latency and thus reduce lookup times. A well designed DNS network will generally provide better and more consistent performance for end-users globally. However, because DNS lookups are almost always cached, after the first lookup, DNS generally has little impact on website performance.

To measure end-user DNS performance, we developed a browser based test that measures the time difference between downloading a small (4 byte) file using both cached and non-cached hostnames. We use a special type of DNS record called a wildcard name that allows the test to generate random hostnames that are guaranteed to require an authoritative DNS lookup. Multiple measurements are taken during each test and the median is recorded. We've run this test using thousands of unique users globally every month. We also utilize a Geo IP database to determine where the user running the test is located which allows us to generate the region specific analysis for each DNS service show below.

The following charts display the results of our end-user DNS performance analysis for July 2012. The bars in each chart represent a managed DNS service and a specific geographic region. The vertical axis is the median DNS lookup time in milliseconds (1000 milliseconds equals 1 second) for all users and tests in that region (typically hundreds or thousands of unique users). The line spanning horizontally on the chart represents the median lookup time for all regions. In general, DNS lookup times in the 50-200 millisecond range are very good.

End-User DNS Performance - North America July 2012
In North America, bandwidth and connectivity are relatively simple and affordable. All providers performed generally well in these regions with limited variation between them.

End-User DNS Performance - Europe July 2012
Performance in Europe regions was only marginally slower than North America and we observed slightly higher performance variation between providers.

End-User DNS Performance - Other July 2012
Bandwidth and connectivity are significantly more costly and complex in Asia, Oceania, and South America. We observed slower performance and higher variation between providers in these regions.

Synthetic Performance

Another method of measuring DNS performance is to query authoritative DNS servers directly (bypassing recursive lookup chains). This method utilizes test agents located in data centers and provides a more repeatable, consistent and controlled testing environment. However, because these tests are conducted by just a handful of servers located in data centers, and bypasses recursive lookup chains, it is less relevant to the performance an end user would experience. For our testing, we are utilized a network of 110 servers (57 US, 28 EU, 25 AsiaPAC and other locations) conducting tests every 5 minutes. The charts below use the same format as the end-user charts.

Synthetic DNS Performance - North America July 2012
Due to the lower cost and easier deployment of bandwidth and connectivity, performance in North American regions was generally good and showed minimal variation between providers. Akamai uses a combination of Anycast and Unicast (non latency minimizing) networks which is likely the reason for their lower performance in this analysis

Synthetic DNS Performance - Europe July 2012
Performance in Europe was also generally good. Many of our test agents are likely located in close proximity to the DNS servers (i.e. in the same or a nearby data center)

Synthetic DNS Performance - Other July 2012
In Asia and Oceania, where bandwidth and connectivity is more expensive and complex to setup, we observed lower performance and higher performance variation between providers

Availability

Managed DNS provider networks consist of many DNS POPs (a single POP consists of 1 or more DNS servers) distributed globally. When used in combination with Anycast routing, these DNS servers can be configured to automatically failover to other POPs. Additionally, the DNS protocol can include more than one authoritative server and incorporates retry logic such that DNS clients will try to query multiple DNS servers until a response is received. These factors enable DNS to provide continual availability as long as at least 1 DNS POP is available.

To calculate availability, we utilized the 110 monitoring agents querying provider DNS servers every 5 minutes throughout the month. Every authoritative DNS server was queried during each test. Two metrics are were captured... availability of at least 1 DNS server and availability of all DNS servers. The former is the more important as it reflects what an actual user would generally experience. All providers offered very high availability. The following table represents the results on this analysis for July 2012:

DNS Provider Availability July 2012

DNS Propagation Latency

Another comparison criterion is the time required to push a DNS record update to a provider's DNS network. We refer to this metric as DNS Propagation Latency. This criterion is more relevant to an organization making frequent or automated DNS updates, where those updates are critical to some functionality. To measure DNS propagation latency, we used 110 global test agents. At the exact moment a DNS change was submitted, those agents were instructed to directly query provider DNS servers and record the amount of time that change took to complete. Testing was conducted for both primary and secondary DNS (secondary DNS is a replication-only service). The tables below show the results of this analysis (Propagation Latency is a median value for all 110 test agents and all provider DNS POPs):

Primary DNS Propagation Latency - July 2012

Secondary DNS Propagation Latency - July 2012
Secondary DNS is a replication service, DNS propagation times are longer because the secondary service has to wait for the primary service to replicate first. During our testing, DNS Made Easy was the primary DNS service. Not all providers offer secondary DNS service.

DNS Provider Marketshare

Marketshare is a good indicator of the robustness of a provider DNS network. Providers that manages DNS for popular websites have a demonstrated capability to effectively support very high DNS volume.

To capture DNS provider marketshare we determined which DNS providers are used for the top 10,000 Alexa sites (Alexa publishes a list of the most popular websites). Some DNS providers allow customers to mask their DNS servers using custom hostnames (a featured referred to as vanity DNS servers). To include these, we correlate vanity DNS servers using IP address matching (matching class C IP addresses to those of known provider DNS servers). To verify correlations, we perform authoritative lookups of the hostnames using actual provider DNS servers.

We also track provider marketshare distribution changes by comparing provider marketshare this month to that of the previous month. The following are the results of this analysis for the month of July 2012:

Alexa Top 1000 DNS Marketshare - Aug 2012
Alexa Top 10000 DNS Marketshare - Aug 2012

Top 20 Provider Alexa 10,000 Changes - July 2012

Top 20 Provider Alexa 10,000 Changes - July 2012

Confirmed Alexa Top 10,000 Changes - July 2012

Confirmed Alexa Top 10,000 Changes - July 2012
Because the makeup of the top 10,000 Alexa websites changes from month to month, the marketshare change analysis may not represent actual provider change. The change metrics above, represent the number of actual confirmed provider changes between July 1 and Aug 1 2012. During this time, we observed significant usage growth for both Route 53 and Dyn.

Features

Advanced features are one of the biggest ways that DNS providers distinguish their services. The following is an overview of a few common features and their associated support with each DNS provider:

Health Checks - DNS Failover

DNS Health Checks - FailoverDNS failover involves dynamic DNS resolution based on the availability of target hosts. These hosts are monitored continually by DNS providers using ICMP (ping) or more advanced methods such as HTTP content monitoring. If a primary target host fails a health check, DNS resolution automatically change to a backup target host.

UltraDNSDynCotendoRoute 53DNS Made EasyeasyDNS
YesYesYesNoYesYes

Health Checks - DNS Load Balancing

DNS Health Checks - FailoverLike DNS failover, DNS load balancing monitors target hosts. However, with load balancing there target hosts are not considered active or failover - instead, all hosts receive a even (or weighted) distribution of traffic. If a target host goes down, the DNS service will stop sending traffic to it by ceasing to resolve that IP address.

UltraDNSDynCotendoRoute 53DNS Made EasyeasyDNS
YesYesYesNoNoNo

Location Based Routing (Geo IP)

DNS Location Based Routing (Geo IP)Location based DNS routing allows a DNS hostname to resolve dynamically depending on the geographic location of the user (or more specifically, location of the user's DNS resolver). To accomplish this, the geographic location is determined using Geo IP databases like Neustar IP Intelligence (formerly Quova) or MaxMind. This location is then run through custom, user-defined DNS rules that may affect the IP address the hostname resolves to. For example, a hostname might resolve to a server in Singapore for users in Asia, and a server in the US for others thereby improving webpage load times.

UltraDNSDynCotendoRoute 53DNS Made EasyeasyDNS
YesYesYesYes1NoNo

1Route 53 provides a unique feature called Latency Based Routing, where DNS resolves to a target host with the presumed lowest latency to the end user (the target host must be in one of 7 AWS data center regions)

Zone Based Routing (Anycast)

DNS Zone Based Routing (Anycast)Zone Based Routing is functionally similar to Location Based Routing, but instead of using a Geo IP database to determine the user's geographic location, it uses the DNS server that the user is querying. In Anycast networks, this DNS server will typically reside in the same general geographic region as the DNS client. In practice, this limits the number of location specific rules to the number of Anycast zones in the provider's network (typically in the single digits). For example, Dyn's network consists of 7 Anycast zones, thus allowing up to 7 location specific target hosts. Because of this, zone based routing is more limited relative to location based routing. Additionally, zone based routing can be problematic in some geographic regions where Anycast networking is less predictable (i.e. Asia) or when POPs are taken down for maintenance

UltraDNSDynCotendoRoute 53DNS Made EasyeasyDNS
NoYes (7 regions)NoNoYes (4 regions)Yes (4 regions)

DNSSEC

DNSSEC DNSSEC (Domain Name System Security Extensions) is a specification for securing DNS records. DNSSEC was designed to protect clients from forged DNS responses by digitally signing DNS responses. By checking the digital signature, DNS clients can verify the authenticity of those responses. Usage of DNSSEC is growing, but due to complexity and lack of support is still relatively low.
Provider or User Managed DNSSEC
Manual generation and management DNSSEC certificates can be cumbersome. Some providers simplify this by generating and deploying certificates automatically thereby eliminating many of the administrative complexities.

UltraDNSDynCotendoRoute 53DNS Made EasyeasyDNS
Yes (provider managed)Yes (provider managed)NoNoNoYes (user managed)

Pricing

Enterprise DNS providers, generally do not disclose pricing publicly. Part of this likely has to do with the negotiable nature of their pricing. In order to determine pricing for these services (where it was not available publicly), we contacted each provider for a pricing quote. Actual pricing may vary depending on a customer's ability to negotiate.

DNS Query Pricing (monthly)

Provider1 million10 million100 million1 billion10 billion
AWS Route 53$0.50$5$50$500$2,750
UltraDNS$50-$1951$865-$1,2002$2,200-$3,0002$5,1252$17,5002
CotendoNot offered3$500$1,000$5,000$10,0000
Dyn$604$295 (10 QPS)$600 (40 QPS)$2,250 (400 QPS)$5,495 (4000 QPS)
DNS Made Easy$2.505$55$2187$1,5207$7,3707
easyDNS$9.956$20$2008$2,0008$20,0008
  1. $50 plan includes US and EU DNS POPs only
  2. UltraDNS discounts query pricing by up to 50% when bundled with advanced features list below. The prices provided here are based on some bundling
  3. The lowest usage tier for Cotendo DNS is 10 million queries/mo
  4. Dyn Enterprise DNS Lite - includes 1.2 million queries/mo
  5. Must prepay annually
  6. Enterprise Plan - Pricing is for 5 million queries/mo
  7. Based on $1500/yr corporate membership (includes 50 million queries/mo)
  8. Based on Enterprise plan and published overage rate - discounts may be available

Advanced Feature Pricing (monthly)

ProviderHealth Checks - FailoverHealth Checks - Load BalancingLocation Based Routing (Geo IP)Zone Based Routing (Anycast)DNSSEC
AWS Route 53NANA$0.25/million queries10NANA
UltraDNS$2251$5632$5003NAIncluded
Cotendo$1304$1304IncludedIncludedNA
Dyn$1005$2005$4006$2006Included
DNS Made Easy$0.427NANA$558NA
easyDNSIncluded9NANAIncluded9Included
  1. UltraDNS refers to this feature as Sitebacker. Price is based on 10 million query bundle pricing (25% discount) with 2 monitored target hosts. Each additional target host is $113/mo ($150 without bundle discount) up to 5, then $90 up to 10, then $68
  2. UltraDNS refers to this feature as Traffic Controller. Price is based on 10 million query bundled pricing (25% discount) with 3 monitored target hosts. Each additional IP is $188/mo ($250 without bundle discount)
  3. Pricing based on 10 million query bundled pricing (25% discount) with up to 5 target hosts. Each additional target host is $100/mo
  4. Per hostname with up to 10 monitored target hosts
  5. Priced based on # of monitor samples per month, $3 per 1000 samples - pricing provided is based on 3 target hosts, 1 monitoring node and 3 minute monitoring intervals. Pricing reduces to $0.30/1000 for 10 million samples/month
  6. Per hostname
  7. Must prepay annually - the Business and Corporate plans include 3 and 10 failover hostnames respectively and up to 5 monitored target hosts using 2-4 minute monitoring intervals
  8. Requires $60/yr business plan. $1500/yr Corporate plan includes 1 geo-targeted hostname at no additional charge
  9. Only 15 minute monitoring intervals are supported
  10. Query surcharge for latency based routing - only target hosts in AWS data centers are supported. Query pricing reduces to $0.125/ million queries for volume above 1 billion queries/mo

Summary

There is a lot to consider when comparing managed DNS providers. We've included what we believe to be a few relevant, objective and comparable evaluation criteria in this post and our new DNS report. Organizations should consider the criteria that are most relevant to them when evaluating providers. If advanced features are needed, an enterprise provider may be the best choice. If just plain DNS is needed, a self service provider like AWS Route 53 may be a better and more cost effective service. The most important factor is to make your selection based on relevant and objective criteria, not marketing spin.

View Full Report - August 2012

Thursday, November 17, 2011

Is Joyent Really 14X Faster than EC2 and Azure the "Fastest Cloud"? Questions to Ask About Benchmark Studies

Many are skeptical of claims that involve benchmarks. Over the years benchmarks have been manipulated and misrepresented. Benchmarks aren't inherently bad or created in bad faith. To the contrary, when understood and applied correctly, benchmarks can often provide useful insight for performance analysis and capacity planning. The problem with benchmarks is they are often misunderstood or misrepresented, frequently resulting in bold assertions and questionable claims. Oftentimes there are also extraneous factors involved such as agenda-driven marketing organizations. In fact, the term "benchmarketing" was coined to describe questionable marketing-driven, benchmark-based claims. This post will discuss a few questions one might consider when reading benchmark-based claims. We'll then apply these questions to 2 recent cloud related, benchmark-based studies.

Questions to consider

The following are 7 questions one might ask when considering benchmark-based claims. Answering these questions will help to provide a clearer understanding on the validity and applicability of the claims.
  1. What is the claim? Typically the bold-face, attention grabbing headline like Service Y is 10X faster than Service Z
  2. What is the claimed measurement? Usually implied by the headline. For example the claim Service Y is 10X faster than Service Z implies a measurement of system performance
  3. What is the actual measurement? To answer this question, look at the methodology and benchmark(s) used. This may require some digging, but can usually be found somewhere in the article body. Once found, do some research to determine what was actually measured. For example, if Geekbench was used, you would discover the actual measurement is processor and memory performance, but not disk or network IO
  4. Is it an apples-to-apples comparison? The validity of a benchmark-based claim ultimately depends on the fairness of the testing methodology. Claims involving comparisons should compare similar things. For example, Ford could compare a Mustang Shelby GT500 (top speed 190 MPH) to a Chevy Aveo (top speed 100 MPH) and claim their cars are nearly twice as fast, but the Aveo is not a comparable vehicle and therefore the claim would be invalid. A more fair, apples-to-apples comparison would be a Mustang GT500 and a Chevy Camaro ZL1 (top speed 186).
  5. Is the playing field level? Another important question to ask is whether or not there are any extraneous factors that provided an unfair advantage to one test subject over another. For example, using the top speed analogy, Ford could compare a Mustang with 92 octane fuel and a downhill course to a Camaro with 85 octane fuel and an uphill course. Because there are extraneous factors (fuel and angle of the course) which provided an unfair advantage to the Mustang, the claim would be invalid. To be fair, the top speeds of both vehicles should be measured on the same course, with the same fuel, fuel quantity, driver and weather conditions.
  6. Was the data reported accurately? Benchmarking often results in large datasets. Summarizing the data concisely and accurately can be challenging. Things to watch out for include lack of good statistical analysis (i.e. reporting average only), math errors, and sloppy calculations. For example, if large, highly variable data is collected, it is generally a best practice to report the median value in place of mean (average) to mitigate the effects of outliers. Standard deviation is also a useful metric to include to identify data consistency.
  7. Does it matter to you? The final question to ask is, assuming the results are valid, does it actually mean anything to you? For example, purchasing a vehicle based on a top speed comparison is not advisable if fuel economy is what really matters to you.

Case Study #1: Joyent Cloud versus AWS EC2

In this case study, Joyent sponsored a third party benchmarking study to compare Joyent Cloud to AWS EC2. The study utilized our own (CloudHarmony) benchmarking methodology to compare 3 categories of performance: CPU, Disk IO and Memory. The end results of the study are published on the Joyent website available here. In the table below, we'll apply the questions listed above to this study. Answers will be color coded green where the study provided a positive response to the question, and red where the results are misleading or misrepresented.
Questions & Answers

What is the claim? Joyent Cloud is 3x - 14x Faster than AWS EC2
The claims are broken down by measurement type (CPU, Disk IO, Memory), and OS type (SmartOS/Open Solaris, Linux). The resulting large, colorful icons on the Joyent website claim that Joyent Cloud is faster than EC2 by a margin of 3x - 14x
What is the claimed measurement? CPU, Disk IO, Memory Performance
Our benchmarking methodology was used to measure these different categories of performance. This methodology consists of running multiple benchmarks per category and creating a composite measurement based on a summary of the results for all benchmarks in each category. The methodology is described in more detail on our blog here (CPU), here (Disk IO) and here (Memory).
What is the actual measurement? CPU, Disk IO, Memory Performance
Is it an apples-to-apples comparison? Dissimilar instance types were compared
In the Linux comparison, Joyent claims 5x faster CPU, 3x faster Disk IO, and 4x faster memory. Based on the report details, it appears those ratios originate from comparing a 1GB Joyent VM to an EC2 m1.small. This selection provided the largest performance differential and hence the biggest claim. While price-wise, these instance types are similar (disregarding m1.small spot and reserve pricing where it is 1/2 the cost), that is where the similarities stop. At the time of this report, m1.small was the slowest EC2 instance with a single core and older CPU, while Joyent's 1GB instance type has 2 burstable cores and a newer CPU. The m1.small is not intended for compute intensive tasks. For that type of workload EC2 offers other options with newer CPUs and more cores. To provide an apples-to-apples comparison on performance, the claim should be based on 2 instance types that are intended for such a purpose (e.g. an EC2 m2 or cc1).
Is the playing field level? Operating system and storage type were different
The study compares Joyent Cloud VMs running SmartOS or Ubuntu 10.04 to AWS EC2 VMs running CentOS 5.4. Joyent's SmartOS is based on Open Solaris and highly optimized for the Joyent environment. Ubuntu 10.04 uses Linux Kernel 2.6.32 (release date: Dec 2009) which is over 3 years newer than the 2.6.18 kernel (release date: Sep 2006) in CentOS 5.4. Newer and more optimized operating systems will almost always perform better for similar tasks on identical hardware. This provided an advantage to the Joyent VMs from the offset.

Additionally, the tests compared EC2 instances running on networked storage (EBS) to Joyent instances running on local storage, which also provided an advantage to the Joyent VMs for the disk IO benchmarks.
Was the data reported accurately? Mistakes were made in calculations
This study was based on a cloud performance comparison methodology we (CloudHarmony) developed for a series of blog posts in 2010. For CPU performance, we developed an algorithm that combined the results of 19 different CPU benchmarks to provide a single performance metric that attempts to approximate the AW ECU (Elastic Compute Unit). To do so, we utilized EC2 instances and their associated ECU value as a baseline. We called this metric CCU and the algorithm for producing it was described in this blog post. Part of the algorithm involved calculating CCU when performance exceeded the largest baseline EC2 instance type, the 26 ECU m2.4xlarge. In our algorithm we used the performance differential ratio between an m1.small (1 ECU) and m2.4xlarge (26 ECUs). The third party, however, used the ratio between an m2.2xlarge (13 ECUs) and m2.4xlarge (26 ECUs). Because m2s run on the same hardware type, the performance difference between an m2.2xlarge and an m2.4xlarge is not very great, but the difference in ECUs is very high. The end results was their calculations producing a very high CCU value for the Joyent instances (in the range of 58-67 CCUs). Had the correct algorithm been used, the reported CCUs would have been much lower.
Does it matter to you? Probably not
There isn't much value or validity to the data provided in these reports. The bold headlines which state Joyent Cloud is 3X - 14X faster than EC2 are based on very shaky grounds. In fact, with Joyent's approval, we recently ran our benchmarks in their environment resulting in the following CPU, disk IO and memory performance metrics: CloudHarmony Generated Joyent/EC2 Performance Comparison
CPU Performance: AWS EC2 vs Joyent
View Full Report
Provider Instance Type Memory Cost CCU
EC2 cc1.4xlarge 23 GB $1.30/hr 33.5
Joyent XXXL 48GB (8 CPU) 48 GB $1.68/hr 28.44
EC2 m2.4xlarge 68.4 GB $2.00/hr 26
EC2 m2.2xlarge 34.2 GB $1.00/hr 13
Joyent XL 16GB (3 CPU) 16 GB $0.64/hr 10.94
Joyent XXL 32GB (4 CPU) 32 GB $1.12/hr 6.82
EC2 m2.xlarge 17.1 GB $0.50/hr 6.5
Joyent Large 8GB (2 CPU) 8 GB $0.36/hr 6.19
Joyent Medium 4GB (1 CPU) 4 GB $0.24/hr 5.53
Joyent Medium 2GB (1 CPU) 2 GB $0.17/hr 5.45
Joyent Small 1GB (1 CPU) 1 GB $0.085/hr 4.66
EC2 m1.large 7.5 GB $0.34/hr 4
EC2 m1.small 1.7 GB $0.085/hr 1
Disk IO Performance: AWS EC2 vs Joyent
View Full Report - Note: the EC2 instances labeled EBS utilized a single networked storage volume - better performance may be possible using local storage or multiple EBS volumes. All Joyent instances utilized local storage (networked storage is not available).
Provider Instance Type Memory Cost IOP
EC2 cc1.4xlarge (local storage - raid 0) 23 GB $1.30/hr 212.06
EC2 cc1.4xlarge (local storage) 23 GB $1.30/hr 194.29
Joyent XXXL 48GB (8 CPU) 48 GB $1.68/hr 187.38
Joyent XL 16GB (3 CPU) 16 GB $0.64/hr 144.71
Joyent XXL 32GB (4 CPU) 32 GB $1.12/hr 142.19
Joyent Large 8GB (2 CPU) 8 GB $0.36/hr 130.84
Joyent Medium 4GB (1 CPU) 4 GB $0.24/hr 110.78
Joyent Medium 2GB (1 CPU) 2 GB $0.17/hr 109.2
EC2 m2.2xlarge (EBS) 34.2 GB $1.00/hr 87.58
EC2 m2.xlarge (EBS) 17.1 GB $0.50/hr 83.62
EC2 m2.4xlarge (EBS) 68.4 GB $2.00/hr 82.79
EC2 m1.large (EBS) 7.5 GB $0.34/hr 56.82
Joyent Small 1GB (1 CPU) 1 GB $0.085/hr 56.08
EC2 m1.small (EBS) 1.7 GB $0.085/hr 27.08
Memory Performance: AWS EC2 vs Joyent
View Full Report
Provider Instance Type Memory Cost CCU
EC2 cc1.4xlarge 23 GB $1.30/hr 137.2
EC2 m2.2xlarge 34.2 GB $1.00/hr 109.41
EC2 m2.4xlarge 68.4 GB $2.00/hr 109.14
EC2 m2.xlarge 17.1 GB $0.50/hr 103.35
Joyent XL 16GB (3 CPU) 16 GB $0.64/hr 100.87
Joyent XXXL 48GB (8 CPU) 48 GB $1.68/hr 92.5
Joyent XXL 32GB (4 CPU) 32 GB $1.12/hr 90.79
Joyent Large 8GB (2 CPU) 8 GB $0.36/hr 90.37
Joyent Medium 2GB (1 CPU) 2 GB $0.17/hr 84.2
Joyent Small 1GB (1 CPU) 1 GB $0.085/hr 78.51
Joyent Medium 4GB (1 CPU) 4 GB $0.24/hr 76.04
EC2 m1.large 7.5 GB $0.34/hr 61.8
EC2 m1.small 1.7 GB $0.085/hr 22.24

Case Study #2: Microsoft Azure Named Fastest Cloud Service

In October 2011, Compuware published a blog post related to cloud performance. This post was picked up by various media outlets resulting in the following headlines:
Here's how the test worked in a nutshell:
  • Two sample e-commerce web pages were created. The first with items description and 40 thumbnails (product list page), and the second with a single 1.75 MB image (product details page)
  • These pages were made accessible using a Java application server (Tomcat 6) running in each cloud environment. The exception to this is Microsoft Azure and Google AppEngine (platform-as-a-service/PaaS environments) which required the pages to be bundled and deployed using their specific technology stack
  • 30 monitoring servers/nodes were instructed to request these 2 pages in succession every 15 minutes and record the amount of time it took to render both in their entirety (including the embedded images)
  • The 30 monitoring nodes are located in data centers in North America (19), Europe (5), Asia (3), Australia (1) and South America (2) - they are part of the Gomez Performance Network (GPN) monitoring service
  • After 1 year an average response time was calculated for each service (response times above 10 seconds were discarded)
Now lets dig a little deeper...
Questions & Answers

What is the claim? Microsoft Azure is the "fastest cloud"
What is the claimed measurement? Overall performance (it's fastest)
What is the actual measurement? Network Latency & Throughput
Rendering 2 html pages and some images is not CPU intensive and as such is not a measure of system performance. The main bottleneck is network latency and throughput, particularly to distant monitoring nodes (e.g. Australia to US)
Is it an apples-to-apples comparison? Types of services tested are different (IaaS vs PaaS) and the instance types are dissimilar
Microsoft Azure and Google AppEngine are platform-as-a-service (PaaS) environments, very different from infrastructure-as-a-service (IaaS) environments like EC2 and GoGrid. With PaaS, users must package and deploy applications using custom tools and more limited capabilities. Applications are deployed to large clustered, multi-tenant environments. Because of the greater structure and more limited capabilities of PaaS, providers are able to better optimize and scale those applications, often resulting in better performance and availability when compared to a single server IaaS deployment. Not much information is disclosed regarding the sizes of instances used for the IaaS services. With some IaaS providers, network performance can vary depending on instance size. For example, with Rackspace Cloud, a 256MB cloud server is capped with a 10 Mbps uplink. With EC2, bandwidth is shared across all instances deployed to a physical host. Smaller instance sizes generally have less, and more variable bandwidth. This test was conducted using the nearly smallest EC2 instance size, an m1.small.
Is the playing field level? Services may have unfair advantage due network proximity and uplink performance
Because network latency is the main bottleneck for this test, and only a handful of monitoring nodes were used, the results are highly dependent on network proximity and latency between the services tested and the monitoring nodes. For example, the Chicago monitoring node might be sitting in the same building as the Azure US Central servers giving Azure and unfair advantage in the test. Additionally, the IaaS services where uplinks are capped on smaller instance types would be at a disadvantage to uncapped PaaS and IaaS environments.
Was the data reported accurately? Simple average was reported - no median, standard deviation or regional breakouts were provided
The CloudSleuth post provided a single metric only… the average response time for each service across all monitoring nodes. A better way to report this data would involve breaking the data down by region. For example, average response time for eastern US monitoring nodes. Reporting median, standard deviation and 90th percentile statistical calculations would also be very helpful in evaluating the data.
Does it matter to you? Probably not
Unless your users are sitting in the same 30 data centers as the GPN monitoring nodes, this study probably means very little. It does not represent a real world scenario where static content like images would be deployed to a distributed content delivery network like CloudFront or Edgecast. It attempts to compare two different types of cloud services, PaaS and IaaS. It may use IaaS instance types like the EC2 m1.small that represent the worst case performance scenario. The 30 node test population is also very small and not indicative of a real end user population (end users don't sit in data centers). Finally, reporting only a single average value ignores most statistical best practices.

Monday, October 17, 2011

Encoding Performance: Comparing Zencoder, Encoding.com, Sorenson & Panda

A few months ago we were approached by Zencoder to conduct a sponsored performance comparison of 4 encoding services including their own. The purpose was to validate their claims of faster encoding times using an independent, credible external source. This was a new frontier for us. Our primary focus has been performance analysis of infrastructure as a service (IaaS). However, we are curious about all things related to cloud and benchmarking and we felt this could be useful data to make available publicly, so we accepted.

Testing Methodology

This is a description of the methodology we used for conducting this performance analysis.

Source Media

Following discussions with Zencoder, we opted to test encoding performance using 4 distinct media types. We were tasked with finding samples for each media type, they were not provided by Zencoder. All source media was stored in AWS S3 using the US East region (the same AWS region each of the 4 encoding services are hosted from). The 4 media types we used for testing are:
  • HD Video: We chose an HD 1080P trailer for the movie Avatar. This file was 223.1 MB in size and 3 mins, 30 secs in duration.
  • SD Video: We chose a 480P video episode from a cartoon series. The file was 519.2 MB in size and about 23 mins in duration.
  • Mobile Video: We created a 568x320 video using an iPhone (source here). The file was 2.9 MB in size, 30 secs in duration.
  • MP3 Audio: We used an MP3 file we found on the web about Yoga (source here). The file was 42.2 MB in size, 58 mins 41 secs in duration.

Encode Settings

We used the same encode options across all of the services tested. The following is a summary of the encode options used for each corresponding media type:
Media Type Video Codec Video Bitrate Audio Codec Audio Bitrate Encode Passes
HD Video H.264 3000 Kb/s AAC 96 Kb/s 2
SD Video H.264 1000 Kb/s AAC 96 Kb/s 2
Mobile Video H.264 500 Kb/s AAC 96 Kb/s 2
MP3 Audio NA NA AAC 96 Kb/s 2

Test Scheduling

Testing was conducted during a span of 1 week. We built a test harness that integrated with the APIs of each of the 4 encoding services. The test harness invoked 2 test iterations daily with each service. Testing included both single request and 4 parallel requests. Each test iteration consisted of the following 8 test scenarios:
  • Single HD Video Request
  • Single SD Video Request
  • Single Mobile Video Request
  • Single MP3 Audio Request
  • 4 Parallel HD Video Requests
  • 4 Parallel SD Video Requests
  • 4 Parallel Mobile Video Requests
  • 4 Parallel MP3 Audio Requests
The order of the test scenarios was randomized, but the same tests were always requested at the same time for each service. Each test was run to completion on all services before the next test was invoked. The start times for 2 daily test iterations were separated by about 12 hours and incremented by 100 minutes each day. The end result was a distribution of test scenarios during many different times of the day. A total of 112 tests were performed during the 1 week test span, followed by an additional 24 test scenarios on encoding.com to test different combinations of service specific encoding options (described below).

Performance Metrics

During testing, we captured the following metrics:
  • Encode Time: The amount of time in seconds required to encode the media (excludes transfer time)
  • Transfer Time: The amount of time in seconds required to transfer the source media from AWS S3 into the service processing queue
  • Failures: When a service failed to complete a job

Test Results

The following is a summary of the results for each of the 8 test scenarios. Result tables have the following columns:
  • Avg Encode Time: The mean (average) encoding time in seconds for all jobs in this test scenario
  • Standard Deviation %: Standard deviation as percentage of the mean. A lower values indicates more consistency in performance
  • Median Encode Time: The median encoding time in seconds for all jobs in this test scenario
  • Avg Total Time: The mean (average) total job time in seconds. The total time is the sum of transfer (source files were hosted in AWS S3 US East), queue and encoding times, essentially the total turnaround time from the moment the job was submitted. Sorenson and Panda may cache source media files, thus reducing transfer time for future requests

A graph is displayed below each results table depicting sorted average encode and total times using a dual series horizontal bar chart. Also depicted on the graph is a line indicating the actual duration of the source media. Encode time bars that terminate to the left of this line signify faster than realtime encoding performance.

encoding.com offers a few different encoding job options that can affect encode performance. These options include the following:

  • instant: using the instant option, encoding.com will begin encoding before the source media has been fully downloaded. For larger source files, this can decrease encode times
  • twin turbo: this setting causes jobs to be delegated to faster servers in exchange for paying a $2/GB premium for encoding. encoding.com states that this option will deliver 6-8X faster encoding time over standard servers

Test Results: Single HD Video Request

The following are the results from a total of 14 HD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 141.08 6 137 155.08
encoding.com 2 (max+TT) 213.38 16 193 244.53
encoding.com 1 (max+TT+instant) 213.5 11 213.5 238
encoding.com 4 (plus+TT+instant) 226.75 11 235.5 258.75
encoding.com 3 (max) 618.75 12 643.5 653.25
sorenson 974.31 2 975 982.46
panda 1246.31 4 1247 1255.93

Failures - None

Test Results: Single SD Video Request

The following are the results from a total of 14 SD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 229.14 9 225.5 281.64
encoding.com 4 (plus+TT+instant) 453 1 453.5 718.75
encoding.com 2 (max+TT) 454.07 11 458.5 689.21
encoding.com 1 (max+TT+instant) 474 7 474 732.5
encoding.com 3 (max) 1137.75 12 1162 1429
panda 1649.36 7 1640.5 1673.93
sorenson 2087.23 8 2052 2101

Failures

  • Sorenson: 1

Test Results: Single Mobile Video Request

The following are the results from a total of 14 mobile video encode jobs submitted at various times of the day over a period of 1 week. NOTE: During our testing, encoding.com jobs would occasionally experience excessive queue times with the status "Waiting for encoder". This is the reason for the long green section representing the transfer/queue time delta on the graph below.
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 19.38 12 20 30.23
panda 25.69 4 25 26.31
encoding.com 4 (plus+TT+instant) 35 7 34 122.75
encoding.com 3 (max) 40.25 9 40.5 51.25
sorenson 47 6 46 68.31
encoding.com 2 (max+TT) 76.92 139 31 88.23
encoding.com 1 (max+TT+instant) 97 67 97 112

Failures - None

Test Results: Single MP3 Audio Request

The following are the results from a total of 14 audio encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 120.31 11 115 131
sorenson 189.38 11 182 196.23
panda 218.58 3 217 222.16
encoding.com 1 (max+TT+instant) 221 0 221 255
encoding.com 4 (plus+TT+instant) 224.75 9 225 248.75
encoding.com 3 (max) 240.5 7 238 355.75
encoding.com 2 (max+TT) 328.36 52 254 360.45

Failures

  • encoding.com max+TT+instant: 1
  • encoding.com max+TT: 2

Test Results: 4 Parallel HD Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) HD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 141.23 4 140 153.71
encoding.com 1 (max+TT+instant) 236.88 19 250 280.13
encoding.com 2 (max+TT) 292.21 46 259.5 321.96
encoding.com 4 (plus+TT+instant) 298.69 37 256.5 354.44
encoding.com 3 (max) 723.44 25 751.5 796.38
panda 1322.19 17 1249 1336.9
sorenson 1751.44 3 1745 1769.71

Failures - None

Test Results: 4 Parallel SD Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) SD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 240.48 17 224.5 327.98
encoding.com 2 (max+TT) 439.5 21 445 700.15
encoding.com 4 (plus+TT+instant) 461.56 14 459.5 757.87
encoding.com 1 (max+TT+instant) 484.88 10 459 770.01
encoding.com 3 (max) 1274.63 11 1274 1552.69
panda 1680.96 10 1648.5 1710.9
sorenson 3777.54 10 3835 3808.94

Failures - None

Test Results: 4 Parallel Mobile Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) mobile video encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 20.4 15 20 27.27
panda 26.65 7 26 27.21
encoding.com 1 (max+TT+instant) 34 8 34 49.75
sorenson 55.2 15 53 77.54
encoding.com 4 (plus+TT+instant) 68 60 44.5 106.63
encoding.com 2 (max+TT) 87.46 100 37 104.92
encoding.com 3 (max) 115.88 197 46 149.32

Failures

  • sorenson: 8

Test Results: 4 Parallel MP3 Audio Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) audio encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 115.62 6 114 126.16
sorenson 187.24 5 184 204.57
panda 225.15 17 214 227.92
encoding.com 4 (plus+TT+instant) 234.38 17 228.5 266.63
encoding.com 2 (max+TT) 265.43 19 261 313.56
encoding.com 1 (max+TT+instant) 273.67 17 274 344
encoding.com 3 (max) 285.69 31 259 317.15

Failures

  • sorenson: 3
  • encoding.com max+TT+instant: 5
  • encoding.com max+TT: 5
  • encoding.com max: 3

Encoding Service Accounts

The following encoding services are included in this analysis: Zencoder, encoding.com, Sorenson Media, and Panda. Each service offers different account and pricing options. We setup an account with each service (including Zencoder) using their standard signup process. Because 4 parallel job requests were part of the testing, we opted for service plans that would limit the effects of queue time under such conditions. The pricing data below is for informational purposes only. The purpose of this post is not to compare service pricing, as there are simply too many variations between services to be able to do so. The following is a summary of the account options we selected and pricing with each service:
Service Plan Used Plan Cost Plan Usage Included Encoding Cost Total Test Costs
Zencoder Launch $40/mo 1000 Encoding Mins $0.08/min HD; $0.04/min SD; $0.01/min audio $91
encoding.com Max $299/mo 75GB Encoded Media NA $299
Sorenson Media Squeeze Managed Server $199/mo 1200 Encoding Mins $8 per 60 Mins extra $375 (incl addl encoding mins)
Panda 4 dedicated encoders $396 Unlimited encoding - $0.15/GB to upload encoded media NA $436 (incl bandwidth)

Disclaimer

This comparison was sponsored by Zencoder. In order to sustain our ability to provide useful, free & publicly accessible analysis, we frequently take on paid engagements. However, in doing so, we try to maintain our credibility and objectivity by using reliable tests, being transparent about our test methods, and attempting to represent the data in a fair way.

Summary

In order to maintain our objectivity and independence, we generally do not recommend one service over another. We prefer to simply present the data as it stands, and let readers draw their own conclusions.