Paid Internet Peering on the Rise, Disputes Possible

"I think of these disputes mostly as growing pains,” said Constantine Dovrolis, associate professor of computer science at Georgia Tech. Dovrolis, who has published several papers on the evolution of peering agreements, thinks settlement-free peering is a vestige from the Internet’s early history. “We really need some new ways to think about peering,” he said. “Paid peering may actually end up being the norm, rather than the exception."

For several years, interconnections between ISPs were either transit -- paying for access to the entire Internet -- or settlement-free peering, where different ISPs exchanged traffic between their networks for free. A 2011 survey by Packet Clearing House examined 142,000 peering agreements and found 0.49 percent were formalized in written contracts (http://bit.ly/12qQuaS). Tier 1 ISPs often had written agreements and policies, officials say. But the vast majority of peering arrangements were built on handshake agreements, the report said. Akamai has thousands of peering relationships, and most are handshake agreements, Chief Architect Patrick Gilmore told us. “I have signed one peering agreement in the entire time at Akamai,” he said. “One piece of paper."

That trust is degrading, said network operators. Fewer service providers are engaging in handshake agreements. Many big ones, including Verizon, Comcast, Charter Communications and AT&T, all list detailed peering policies online. In “all the discussions I've been in, they've been discussed as contractual,” said Charlie Baker, director-product management at caching company PeerApp. In the “good old days” when there were “five or six peering sites across the country,” peering agreements were based on loose handshake deals, said Bob Stovall, vice president-network operations and engineering at Merit, an Ann Arbor, Mich.-based nonprofit regional ISP with customers including universities. Now, they're frequently spelled out in contracts -- contracts that increasingly involve payments, he said.

"More and more ISPs are starting to charge for transit,” said Vishal Misra, associate professor of computer science at Columbia University. Misra said he predicted the change in academic papers a few years back. When peering arrangements were established years ago, “the Internet was a very symmetric looking entity,” he said, with largely similar amounts of traffic traveling between universities. As the Internet has become more commercial, it has partitioned into different entities, with content providers like Netflix and Amazon on one side, and “eyeballs” on the other side. And traffic has become very asymmetric, he said. Verizon’s David Young, vice president-federal regulatory affairs, cited the “unbalanced” traffic load when he accused Cogent in June of violating “one of the basic and long-standing requirements” of settlement-free peering arrangements (http://vz.to/19VpvsU).

As the Internet has become more commercial, the traditional roles of various Internet entities have become less clear, researchers said. The roles of access ISPs, transit or backbone ISPs, content providers and content delivery networks used to be fairly distinct, Dovrolis said. Over the last few years, those distinctions have become more and more blurry, he said. “Everybody’s basically trying to play all of these roles all the time.” This increases the likelihood of disputes, Dovrolis said: Before, the various Internet companies used to complement each other, he said. Now, they're offering the same services, said Dovrolis. “This makes these disputes even stronger."

"I don’t think settlement-free peering is going away,” said a Tier 1 ISP executive. What’s changing is that new charging agreements are becoming available, he said. Paid peering is one of them, but there are others that fall between the extremes of free peering and paying for transit, he said. “It goes beyond just paid peering. The Internet interconnection market is evolving. It’s been evolving from day one, and it’s going to continue to evolve in order to accommodate the changing requirements of the network."

The shifts in business model reveal an industry that’s trying to work out its future, said Craig Labovitz, CEO of the cloud-computing firm DeepField. “For a long time, it was just that the technology was driving the evolution,” he said. “Now, as a maturing industry, you really are seeing the business models drive, literally, redesigning how the Internet is built."

Settlement-Free Peering ‘Unstable'?

A huge spike in consumer demand for over-the-top content has made agreements increasingly contentious, said network operators, academic researchers and other stakeholders. “Nobody thought there would be disputes, because nobody thought there would be so much incoming as opposed to outgoing,” Baker said. “The Internet is just a lot of consumption."

That consumption leads to asymmetries in traffic, and those disputes “really can get ugly,” said Michael Miller, chief marketing officer at Renesys, which analyzes peering relationships. “When they're not equitable like that, the whole idea of not paying for the service starts to fall apart.” The problem is compounded when some providers use peering points to carry less of a load than they might otherwise have to -- essentially “gaming the system,” Miller said. “There are providers who will try to push traffic off of their network as quickly as possible, so they don’t have to carry the traffic,” he said. “They're trying to get their peers to carry as much traffic as possible, so that they don’t have to carry it on their network, because that’s [savings] they can use for their customers versus their noncustomers. That’s usually where the tension comes from."

The question many in the industry are working through is how to measure the value of the traffic carried. Some connections will simply never be symmetrical, since last-mile networks always receive a lot more data than they send, said Mark Taylor, Level 3 vice president-media and Internet Protocol services. It should be a “simple engineer-to-engineer discussion,” he said, but it’s getting complicated by business and strategic issues. Level 3 bases its peering policies on the distance a bit has traveled, or bit-miles, he said: Traffic loads can be asymmetrical as long as the distance they are carried is relatively analogous.

It makes sense that Level 3 is pushing a bit-mile metric, said William Norton, executive director of DrPeering International, a consulting firm focused on Internet peering. “You would too if you ran a long-haul network.” That’s just one measure, he said: Some ISPs won’t peer unless the out:in ratio is less than 2.5:1. Verizon’s settlement-free interconnection policy states that the ratio of traffic “shall be roughly balanced and shall not exceed 1.8:1” (http://vz.to/14f1ok5). It also requires the aggregate traffic volume exchange in each direction over all interconnection links “shall equal or exceed 1500 Mbps of traffic” for Verizon.

Research predicts “there will be more disputes,” Misra said. Fifty years of study into cooperative game theory shows that what each partner in a coalition gets is proportional to what it brings to the table, he said: It’s not surprising that Verizon “wants a part” of what Netflix is paying to Cogent, Misra said, because in Verizon’s role as an access ISP, it’s bringing the eyeballs. “I'm not saying that Verizon is wrong, but we'll see more and more of this happening,” he said. “As the Internet has become more and more asymmetric, the current scenario is unstable."

Eyeballs aren’t the only source of leverage in trying to convert formerly free relationships into paying ones, said researchers and engineers. Wholesale networks have an incentive to convert as many of their peered relationships into paying relationships as possible, Akami’s Gilmore said. Networks that serve on-demand content -- like Netflix videos -- could threaten to cut off that connection without payment, if they know it would have a detrimental effect on another network’s end-users, he said. Bill Woodcock, research director at Packet Clearing House, called such tactics “blackmail.”

"It happens surprisingly a lot around Cogent, which is why people find them objectionable,” Woodcock said. “More Internet-oriented companies generally don’t bother with this, because it’s all getting in the way of actually serving their customers. Using other peoples’ customers as hostages is blackmail, and using your own customers by abusing them -- by not providing them the service they paid for and hoping they don’t claim a refund -- it’s just bad business practice.” Cogent CEO Dave Schaeffer told us recently that his company gets a bad rap in the industry because it regularly undercuts competitors on Internet rates. He said Cogent doesn’t de-peer other ISPs, but acknowledged it has been involved in a number of high-profile peering disputes. The company had no comment for this story.

Silent, or ‘Quite the Spectacle'

As ISPs seek to protect their interests, the potential for conflicts rise, said many experts. When negotiations reach an impasse, the result can be a silent de-peer, or a public de-peer, Norton said. “The silent de-peers happen all the time. The session is turned down and like dating, networks don’t want to tell others that they were dumped for fear of how it will make them look: Weak and undesirable.” De-peering can also happen in response to errant cable cuts or weather emergencies. Woodcock estimates 500 to 1,000 small de-peerings occur daily.

Public de-peerings get media coverage. “The case is made to the press, blame is targeted and everyone on the Internet gets a chance to chime in on various blogs with comments enabled,” Norton said. “It is quite the spectacle and everyone is loudly sure that one side is evil and the other is the victim. The court of public opinion is intended to put pressure on the other side."

It’s becoming more common for parties to a dispute to try to leverage their subscribers to advantage their side of the debate, said Baker. Peering disputes are “being used now in the public to try to act as a lever,” he said, citing the Netflix ISP Speed Index as an example (http://nflx.it/14f0BiZ). “It has nothing to do with selling more subscriptions,” Baker said. “It’s purely done by Netflix so that there’s a public outcry. If your ISP is not rated as one of the top ISPs in their Netflix ranking, Netflix wants to use that as leverage to get better agreements and more gear and things going in that network. And they want public perception to be ‘[Your ISP] is not a good Netflix network, you need to call them to make Netflix better.'” Netflix declined to comment.

Some say peering disputes are still rare, just increasingly public. Big disputes happen closer to once a year than once a month, Gilmore said. “They've happened before. Maybe they just weren’t as public,” he told us. “It doesn’t seem like they're happening much more frequently.”

Not everyone agrees the rise in paid peering is significant. Woodcock called paid peering a “fiction,” pushed by network salespeople selling transit that they've rebranded as peering. “There are people who understand that peering exists, but haven’t figured out how to do it for themselves, and somebody comes along and says, ‘Here, let me sell it to you,'” he said. “It’s no more true now than it ever has been in the past. There is no massive groundswell towards people desiring to give [these networks] their money. Just not going to happen. Although [those] salespeople would love to have people believe that that’s the trendy, hot new thing."

Stakeholders disagree over how much harm consumers face as a result of peering disputes. Akamai’s Gilmore said that if large backbone providers actually de-peer, end-users of one ISP could lose their connection to end-users of another. Level 3’s Taylor said the Web had enough other peering points to prevent such an extreme case, but cautioned that de-peering could significantly slow connection speeds for end-users. “The route gets more and more congested, and the quality of the bits get more and more degraded,” he said. Miller agreed. Without a direct peering relationship, traffic will flow into the Internet either via another peered relationship, or a paid transit relationship, he said. “It still may make it to its destination, but it may go in a roundabout way, and impact people’s performance. When you're a content provider like Comcast, those things become very important very quickly in the performance your customers are getting out to the Internet."

It is difficult to predict what impact a de-peering will have, Merit’s Stovall said. He was privy to a de-peering several years ago, he said, in which his network’s transit provider de-peered from the transit provider serving a satellite campus across the country. “One of our campuses was doing research with an entity over there and they couldn’t patch their traffic,” he said. “It took two or three days to resolve, and even email stopped working."

Regulatory Backstop?

As peering disputes become more public, bringing with them the looming threat of an Internet slowdown, some are calling for the government to start taking an interest. The FCC and other relevant governmental authorities “should be collecting information about peering,” said Sascha Meinrath, director of the New America Foundation’s Open Technology Institute. “Given that this is a vital infrastructure, it’s absurd how little we know about what’s actually happening behind the scenes."

Paid peering arrangements exist under a shroud of non-disclosure agreements. Because of that, no one has hard numbers on exactly how many peering agreements are paid, researchers said. “There needs to be transparency around what’s actually happening, since peering impacts nearly every facet of our society, yet isn’t understood,” Meinrath said.

"If possible, you want competition, rather than regulation,” said computer-science Professor Dah Ming Chiu of the Chinese University of Hong Kong. “Even in China, where government has a lot of power, they don’t dictate how ISPs peer.” The nation does “make sure ISPs peer,” he said. If there’s a monopoly on last-mile access, there might be a role for government oversight, he said. “But if there are alternatives, sooner or later, customers can choose to move to other ISPs."

True competition “doesn’t exist” in the last mile, said Prof. Misra. Customers rarely have more than two choices for broadband, he said. Even if there are ostensibly other options, customers are often locked into multiyear contracts, he said: “Eventually” the government “will have to step in.” Misra has proposed a “public option” for the Internet, where the government could “run its own ISP” and ensure access. He conceded that’s “unlikely to happen."

If the FCC wanted to study Internet peering, “I don’t think this would necessarily be the end of the world,” a Tier 1 ISP executive said. When it does so, the agency will find a market that’s working “remarkably well,” better than anything that could be achieved with regulation, he said.

"Internet peering has never been regulated, nor should it be,” said Robert McDowell, an FCC member for seven years until May and now affiliated with the Hudson Institute. “To do so would be to cross a dangerous regulatory Rubicon. History has proven that having peering arrangements worked out through private contractual negotiations is a phenomenally successful model for dispute regulation.” Bringing regulators into the space would be “harmfully disruptive on many levels,” he said, calling it a “giant step backward” that would “impair the dynamism of the global Internet market,” as well as “fuel international efforts to regulate the economics, operations and content” of the Internet.