Monday, March 31, 2014

Netflix's benchmark is wrong... ...for games and large downloads

Reloaded Technologies February 2014 ISP performance benchmark for large game and content downloads.

[jump directly to the results]

Did Netflix just say "DSL is as good as Fiber?"

The Netflix ISP-speed index has been an interesting, and occasionally head-scratching, look at ISP performance in the era of consumer video streaming.

At first glance, the index has some strange results; providers using fiber-only networks rank at the same speed as DSL-only networks that use only "good old" copper phone wires to deliver data. For anyone who has been shopping around for a new ISP this may come as a bit of a surprise that providers who run fiber to the home are not consistently top performers in Netflix's benchmark.

As a counter-point to the Netflix benchmark, we at Reloaded Technologies have decided to release some of our internal performance data gathered anonymously from millions of game downloads performed across thousands of global ISPs.

Game downloads are a completely different type of network activity, since they consist of games that average 5GB in size, where players download a massive amount of data upfront, followed by a long period of play, where small quantities of data is exchanged between the game and the game server in each play session.

Our technology "Super Node Delivery" is used by hundreds of the most popular free2play games in the world, and this technology splits games into thousands of elements that get spread across our massive multi-CDN and multi-node backend, before it's downloaded and reassembled by our client technology designed to take maximum advantage of each end-user's download capacity. The goal of our end-to-end technology is to ensure the fastest and most reliable download for every individual user. Especially since broken or inefficient downloads lead directly to lost customers.

Of course as an infrastructure provider in most cases users are not really paying attention to the fact that they are benefiting from a solution powered by our widely adopted service system (which leads to discussions like this one).

What does the Netflix benchmark actually measure?

By measuring "all Netflix streams" in a month, the index ends up really measuring how ISPs treat their super-consumers of Netflix content. Every hour of Netflix viewing downloads approximately 1GB in standard definition video and 3GB in HD. Therefore super-consumers who stream 10 hours of HD video per day could be downloading 1,000GB of data in any given month, and that means their experiences have the largest impact on the index. Moderate consumers of content might actually have a better experience than the index would indicate, because the measurement really favors those who stream the most.

This explains the really strange result that Verizon FIOS (number 7 in the February 2014 index at 1.76Mbps), a 100% fiber optic network with up to 500Mbps speeds, rates on par with Windstream (number 10 at 1.75Mbps), a regional DSL provider with max speeds of only 12Mbps.

That odd result (1.76Mbps for FIOS vs 1.75Mbps for Windstream) means the index ends up measuring mostly the quality of business agreements between Netflix' and various ISPs, and it gives a big boost to ISPs that agree to put Netflix content servers inside their own networks.

As a Netflix user, I clearly applaud Netflix for sharing this information, but the benchmark tells us nothing about other types of content you may want to consume at home; in particular large digital downloads.

On the opposite side of the spectrum are the artificial speed benchmarks, most notably, which, unlike the Netflix benchmark, creates a single artificial test for your connection. Unfortunately that benchmark has little real-world value when it comes to predicting your download performance for large files from many different download sources.

In short; there have been very few benchmarks that measure real-world practical capacity of your network connection. We hope to help change that by publishing a new ISP benchmark for digital downloads.

RESULTS for Large Downloads: Preliminary ISP speed results

Since our entire focus is optimizing the speed of downloads, we of course get vastly different results in our ISP performance benchmark than those who measure streaming performance. Our benchmark truly gives the edge to ISPs that have built great last-mile networks. From fastest-to-slowest the following are the preliminary February 2014 results for average ISP performance:

Real world ISP performance data for large scale downloads as measured by more than 2,000,000 completed game installs managed by the network in the month of February 2014, including average download time in minutes for an average 5GB game.

Avg Speed
Avg Speed
Avg 5GB file
download time
1 Verizon FiOS 27.22 Mbps 3402.7 KB/sec
24 min
2 Comcast Cable 24.41 Mbps 3051.0 KB/sec
27 min
3 Optimum Online 23.73 Mbps 2965.8 KB/sec
28 min
4 Bright House Networks 21.99 Mbps 2748.3 KB/sec
30 min
5 Charter Communications 21.85 Mbps 2731.8 KB/sec
31 min
6 Cox Communications 20.64 Mbps 2579.8 KB/sec
32 min
7 Shaw Communications 19.56 Mbps 2444.7 KB/sec
34 min
8 Time Warner (Road Runner) 17.27 Mbps 2159.2 KB/sec
39 min
9 Suddenlink Communications 16.42 Mbps 2052.9 KB/sec
41 min
10 Verizon DSL 13.35 Mbps 1669.1 KB/sec
50 min
11 AT&T U-verse 10.74 Mbps 1342.4 KB/sec
62 min
12 CenturyLink 10.45 Mbps 1305.6 KB/sec
64 min
13 Frontier Communications 8.66 Mbps 1082.9 KB/sec
77 min
14 Windstream Communications 6.99 Mbps 873.5 KB/sec
95 min
15 AT&T DSL 6.01 Mbps 751.6 KB/sec
111 min

Conclusion - DSL and Fiber are clearly not equal.

So while the Netflix speed index indicates Verizon FIOS and Windstream are equivalent providers (at 1.76Mbps and 1.75Mbps), when measuring real-world performance for large-size downloads, an average Verizon customer will complete his/her download of a 5GB game in 24 minutes at an average speed of 27Mbps, whereas the average Windstream customer will have to wait an hour and a half, taking nearly four times as long!

Is this a fair benchmark?

Well, just like the Netflix benchmark, the "Super Node Benchmark" is strictly meant to measure only one aspect of your online experience. In our case that one aspect is the average performance per ISP when downloading a very large (5GB) digital content file using technology that has been incredibly optimized to take advantage of the complete user pipe.

In coming months we will refine this benchmark, and start tracking performance trends in other markets. From our point of view, we are clearly interested in helping ISPs continually improve download performance for all its customers through a combination of improved real world throughput, and cheaper higher-bandwidth tiers.

Since we don't yet have a way to know what type of speed tier the customer 'really' is paying for today, we may also enable a more detailed mode for end users who are interested in sharing data about their connection, so we can get a sense of what type of efficiency is offered by each ISP (i.e. if I purchase a 50Mbps connection, do I actually recieve 50Mbps of download performance in a real-world example like a game download when the download is extremely optimized across dozens of servers and independent networks?). Anecdotally when comparing our results to the artificial tests at, we are seeing that most ISPs appear to be between 80%-90% "efficient" in giving users the actual 'tiers' that they have been paying for. But more data is needed before we can draw any firm conclusions.

Inaugural and preliminary data

We are calling this a "Preliminary" benchmark, and over the next few months our team will continue to refine the types of data we will be sharing publicly. Our mission is to encourage ISPs to improve its quality of service to consumers, since the whole online ecosystem benefits when last-mile providers are reliable, dependable and un-congested.

There are also some entertaining findings that too small to be statistically significant, but interesting nevertheless; players University of Calgary can sustain more than 400Mbps when downloading games from our network, and the Cal State system appears to have a better average experience for game downloads (169Mbps) than the University of California system (110Mbps). One clear finding is that educational networks are fantastic for grabbing games fast, though that is probably not what the regents had in mind when they created these great networks.

Cheers, and Happy Downloading!


No comments:

Post a Comment