Reprinted from the Any2 Exchange Quarterly Newsletter, November, 2007.
A nice bonus to consulting on strategy for the Any2 project is that after spending several years working on infrastructure elsewhere in the world, I’ve been getting to do some thinking about what the US peering infrastructure should look like in a couple of years. Most predictions of the future turn out in hindsight to have been laughably wrong, and this will probably be no exception. That said, here are some random thoughts.
As I talk to backbone engineers and attend operational conferences, I hear a sense of doom. For the last decade, we’ve kept up with amazing amounts of traffic growth on the Internet by indiscriminately adding capacity. If a connection filled, we just replaced it with a faster one. We’ve gone from T1s to DS3s, OC3s, OC12s, OC48s, OC192s, and now OC768s. On the Ethernet side, we’ve gone from 10 megabits per second to 100 megabits per second, 1 gigabit per second, 10 gigabits per second, and now 40 gigabits per second. And now, from the complaints I’m hearing, we’re stuck. The optical interfaces needed to increase capacity are unavailable. It seems unlikely that we’ll be stuck at 40 gigabits per second for very long – after all, faster optics have always come along eventually and there’s no reason to think they’ll stop being invented – but we may well start seeing a situation where supply outstrips demand.
The situation reminds me a lot of urban planning models of the 1950s. We’ve probably all seen the displays in museums – the futuristic cities with elevated freeways whisking drivers quickly from the suburbs to their destinations. And we’ve all experienced the results, sitting on those completely jammed freeways, sometimes for hours, trying in vain to get somewhere. We’ve seen the highway engineers’ attempts to deal with the situation – building wider and wider roads only to see them get clogged again. Among urban planners, that vision has been discredited. “Traffic is a gas: it expands to fill the available space,” an urban planner told me recently. Instead, they talk about “smart growth.” Rather than trying to build enough road capacity to move everybody between arbitrary locations miles away, they try to put people close to their likely destinations – in effect moving back to the development pattern that was forced before World War II by the lack of cars.
On the Internet – particularly the American Internet – we’ve got a similar history and a similar problem. We’ve gone from a telecom world where everything was local – where making a long distance phone call was a big deal – to one where we’re frequently told, “distance is dead.” In many ways this is good. It gives us access to literally a world of information. It lets us easily stay in touch with far away friends. But when I ask backbone engineers what all this traffic that’s causing them such agony is, they tell me it’s mostly American video being shipped from China to individual American viewers because the file sharing applications make no effort to distinguish local and distant sources. The backbone operators can keep adding capacity, but the videos are going to get bigger too, and the number of viewers seems likely to explode as people become accustomed to the idea of video on demand. It would make much more sense to apply some of the principles of “smart growth” to this.
That means putting the content close to the users. There will be some economies of scale involved in determining how close, but it’s worth noting that just 20,000 people downloading iTunes video at its 2 Mb/s viewing rate would fill one of our currently state of the art 40 gigabit per second backbone links. How many simultaneous primetime TV viewers are there in a typical medium-sized city?
In much of the world, particularly in places where long distance capacity has long been in short supply, exchange points can be easily categorized as big international exchanges or local exchanges. The international exchanges are often only a few per continent – think LINX, AMS-IX, and DE-CIX in Europe, or HKIX and the various exchanges in Singapore and Tokyo in Asia. Keeping the number of global exchanges small allows international networks to build out to a limited number of locations, and be assured of getting fairly full connectivity. Local exchanges serve a very different purpose – allowing networks in a specific geographic area to hand off traffic to other networks in the same area, without the cost in money or performance of hauling the traffic to an international exchange. It’s the local exchanges where the most popular video content presumably belongs. In the US, we’ve either blurred that distinction or done away with the local exchanges altogether.
The US has some of the world’s most important exchanges for international traffic, mostly on the East and West Coasts where international exchanges would be expected to be. The few in the middle – mainly Chicago and Dallas – serve such wide areas that they can hardly be thought of as local exchanges either. While local networks do connect to exchanges in their areas and peer with each other, pricing designed to squeeze the maximum amount of money out of those with the resources to operate big global networks makes those exchanges increasingly hard for local networks to justify.
This is poised to be an exciting time for US exchange points. After several years of having only two major exchange operators, there are now several investment firms buying up telecom buildings and planning new exchange points. CRG West is one of these, and after some considerable success with the first Any2 exchange in Los Angeles they’re launching exchanges in several more locations. Boston is the one of those I find exciting because it’s a previously unserved market. But the other locations will be interesting as well. Even in areas with well-established exchange points, CRG West’s pricing model should enable local peering for those who can’t cost-justify it at the existing exchanges. And as other real estate companies see what’s possible and follow suit, I’m hopeful that we’ll start seeing more exchanges in more places, bringing us closer to what should be a sustainable model for the American Internet as it continues to grow.