Since February visitors of our website have the possibility to measure "website asset delivery speed" of 24 CDN and cloud providers using their own internet connection (see the blog post). The results for each user are stored in our database and we now have gathered the performance results of 340,000 requests.
There is something special about our test: We have "real surfer" measurements for the 24 providers because we have actually used the "real world" Internet connections of our website visitors to run the test requests. Most CDN comparison tests that we have seen published on the Internet were created by using just a few measurement points, and often - to make things even more unrealistic - the measurements were taken out of professional data centers (which, in most cases, have high speed connectivity anyway). 24% of the tests were run from inside the USA, 9% each from Germany and the UK, the rest came from countries all over the world. So now it is time to dive into the data and draw some conclusions.
Introduction: How Providers Were Selected
First we have created accounts with a selection of 24 CDN and cloud computing providers. We chose those who did not require entry barriers and offered pay-as-you-go pricing. We did this together with our friends from cloudharmony.com (we leverage each other's accounts for CDN testing). Thank you, guys! We wanted to know how much faster it is to use dedicated CDNs for website assets compared to serving these assets directly from the website's web server. That is the reason we not only tested CDN providers but also cloud computing (PaaS) providers.
A Word of Caution
The Javascript-based test at CloudClimate.com runs inside the user's browser window. In order to trigger 10 downloads, we add a unique "cachebreaker" URL parameter to each download URL. This causes some of the CDN providers to pull the images from their origin for each of the 10 requests (instead of pulling it once and then delivering it from the "edge server"). This of course skews results for the 2nd and all subsequent requests, but measurements for the first request are not changed. Especially Akamai CDN, Voxcast CDN, Azure CDN and RackSpaceCloud CDN seem to have this issue.
Lesson #1: Using Any CDN is Better Than Not Using One
When we compared the CDN sites with the cloud computing based sites we found that almost all CDNs we have tested are faster than all servers hosted by cloud computing providers. Even if you are using the fastest cloud server in our test, using a CDN for website assets will likely slash asset load times by up to 50%, especially for users outside the U.S. Using a CDN is the second recommendation in Yahoo's "Best Practices for Speeding Up Your Web Site".
Lesson #2: First Requests Are Much Slower
When we looked at the data we found that for all sites the first request is always the slowest. The initial requests took up to twice as long as the 9 subsequent requests. Reasons for this behavior are:
- The hostname must be resolved into an IP using the DNS system (and this can take even longer for a CDN than for normal systems) before the server can be contacted for the first time
- Some CDNs support http/1.1 keep alives (so for subsequent requests no new http connection must be created)
- The edge servers of the CDNs in most cases have to fetch the file from the origin server (they are then cached for subsequent requests)
The following graphs shows the average times for 10 requests we made to each CDN provider (from a global perspective and for the USA only:
It is obvious that all providers need more time for the initial request.
Our Actual Measurements
If you have a website with heavy traffic from one region then your assets are likely already cached in the edge servers of a CDN because they were recently requested by a previous website visitor (most CDNs keep a file in the cache for 24h). For websites with only little traffic from one region the chance to have website assets already cached on the edge server closest to you becomes smaller and smaller. This must be taken into account when we compare the average request times of the CDN providers: The list of the fastest providers look different for "fastest initial request" and "fastest average request". First the results from a global perspective:
Secondly, the results from the USA only:
Lesson #3: Which Provider Wins The Comparison?
Well, to some extent it depends on the amount of traffic your site receives from its target region. If your site has enough traffic to have your assets likely stored in the CDNs' edge servers for most users, look at the upper graphs. If you have less traffic and most users will likely cause the CDN to pull the asset from the origin server, look at the lower graph which is sorted by the request times for the first request. Anyway, the following 4 providers all hold solid positions in the top 10 of all four lists. They provide the best overall performance from a global and a north American perspective and I would consider all of them solid choices to serve website assets through a CDN:
- CacheFly CDN (#1, #2, #1, #4)
- SoftLayer/Internap CDN (#3, #3, #2, #4)
- GoGrid/Edgecast CDN (#4, #2, #10, #3)
- Google Appspot (#1, #6, #2, #8)
Lesson #4: The Performance Winner Is... CacheFly CDN
We found that CacheFly CDN showed the most solid performance in all 4 categories (1st and average request time, globally and USA only). SoftLayer/Internap and GoGrid/Edgecast are close behind CacheFly though. A honorable mention goes to the cloud server running in the Terremark cloud which showed up in the upper half of our lists between the dedicated CDN providers (#10, #7, #7, #5).
Lesson #5: The Budget Winner Is: Google
This one is a little bit of a surprise, because it is basically a cloud computing offering which has not been mainly designed to be a CDN: The impressive performance of Google's AppSpot (hosted on Google’s distributed architecture) is quite interesting because it is actually free for up to 1.3 Mio requests and 1 gigabyte per day. For web sites with little to medium traffic this seems to be a great way to speed up the website without paying any extra money for hosting. Finally, here are the raw numbers behind our graphs: