Why BitTorrent causes so much jitter (high ping) and how

to fix it
June 1st, 2008 George Ou Leave a comment Go to comments

Anyone VoIP or online gamer who has a roommate or a family member who uses BitTorrent (or
any P2P application) knows what a nightmare it is when BitTorrent is in use. The ping (round
trip latency/jitter) goes through the roof and it stays there making VoIP packets drop out and
game play impossible. I’ve personally experienced this and many of my friends have
experienced it. When I had a roommate in 1999 and I had my first broadband connection, my
roommate started using the hot P2P (peer-to-peer) application of its day called Napster. As soon
as he started sharing, my online game would become slow as molasses and I’d experience a huge
500ms second spike in round-trip latency and any gamer knows that pings over 100 are
problematic.

When it’s just myself using BitTorrent, I can pause it while I make VoIP phone calls or when I
want to game. But when it’s time to ask your family member or roommate to stop using their
application, it becomes awkward because why should they stop using their application just so
you can use your application? My roommate paid for half the broadband fees so I couldn’t
always tell him to stop and he couldn’t always ruin my latency making it a contentious situation
for both of us. What makes it even more frustrating is that there’s plenty of bandwidth capacity
to theoretically satisfy both our needs, but something about these P2P applications make them
them very bad housemates.

I decided to do some research on this and ran a series of tests which resulted in some very
interesting data. I fired up various applications at varying rates of bandwidth utilization in the
upstream and downstream direction and I measured ping to my first router hop (ISP router)
beyond my home router. I ran continuous pings to the ISP router to check the round-trip latency
above the normal baseline latency of 12 ms and plotted out the results. This test methodology
measures network jitter (the variation of delay) on the ―last mile‖ which is what I’m primarily
concerned about because that’s where most of the damage is done by local usage of P2P
applications.

BitTorrent downloads cause huge amounts of latency

The first set of tests I did were download tests. First I downloaded a file using HTTP which ran
at 260 KB/sec (2.08 Mbps) which is roughly 87% of my peak download performance. I then set
BitTorrent to also download at 260 KB/sec to compare the effect on my ping times. To my
surprise, there was a significant difference in the amount of ping increase between HTTP
downloading and BitTorrent downloading despite the fact that both applications were
downloading at the same average rate.
When you look at the two graphs immediately above, you see that BitTorrent causes an average
increase in ping of more than 117 ms (milliseconds). Note that those 6 ping spikes in the graph
were actually bad enough that they caused the ping to time out which means the ping was higher
than one second. HTTP on the other hand didn’t increase the ping nearly as much but it still
caused an average of 46.7 ms increase in ping and it only peaked at 76 ms. So how is this
possible when both applications are using the same amount of bandwidth? I did some thinking
and this is the hypothesis I came up with.

Burst traffic fills up transmit queues

The difference in ping time is caused by the fact that HTTP is a single source of data whereas
BitTorrent is about 20 sources of data. Incoming data from from multiple sources via the fast
core of the Internet can sometimes clump closely together when multiple sources happen to
transmit data around the same time. So in the period of 20 ms (the interval between typical VoIP
packets), up to 200 max-size 1472 BYTE packets can occasionally build up in the DSLAM
transmit queue (blue square in illustration above labeled as ―download queue‖) causing my ping
requests to time out above the 1 second mark. But on average, we get around 23 of these packets
causing sitting in the DSLAM transmit queue causing an average increase in ping of 117 ms.
When there is a single transmitter, it might burst and clump packets close together but it won’t be
at the level of 20 transmitters.

With HTTP causing 76 ms of downstream delay, that means 15 of these 1472-BYTE packets are
sitting in the DSLAM transmit queue causing a less extreme increase in ping. This is still
problematic for VoIP communications and it can certainly ruin online gaming. So despite the
fact that there is plenty of remaining bandwidth for my VoIP or online gaming traffic, it’s the
non-uniformity of the incoming Internet traffic that causes my VoIP phone calls and games to
perform badly. Unfortunately since this is the downstream we’re talking about, the consumer
can’t do much about it on their own end of the pipe because the delay is at the DSLAM which
belongs to the ISP.

The only way to fix this problem is for the ISP to implement intelligent packet scheduling and
application prioritization at the DSLAM to re-order those VoIP or gaming packets to the front of
the transmit queue. With packet prioritization (generally referred to as QoS), your family
member, your roommate, or your own video downloads won’t need to be stopped and they won’t
interfere with your VoIP or gaming applications which makes everyone happy. Unfortunately,
these types of QoS services may never see the light of day if poorly conceived Net Neutrality
legislation gets passed that ban the sale of packet prioritization.

BitTorrent uploads cause excessive jitter

BitTorrent or P2P uploads also cause a lot of upstream jitter. I compared various types of upload
traffic patterns to see what kind of increase in the upstream ping times would result. First I tried
running BitTorrent with a 47 KB/sec (376 Kbps) bandwidth cap which was about 90% of my
upload capacity, then I ran BitTorrent with a 28 KB/sec (224 Kbps) bandwidth cap at 54% of my
upload capacity, and then I ran BitTorrent with a 10 KB/sec cap at 19% of my upload capacity.
In either the 28 KB/sec test or 10 KB/sec test, I’m not being greedy by hogging all the available
upstream bandwidth and I’m leaving more than the 11 KB/sec of bandwidth needed for VoIP
and online gaming. Yet I found that the additional ping caused by BitTorrent uploads (seeding)
was unacceptable for gaming and problematic for VoIP applications. Even when I severely
limited upload throughput to 10 KB/sec, it didn’t reduce the ping time spikes although it reduced
the frequency of those spikes. However, even fewer spikes in ping time can pose the same
problems for VoIP applications because they have to adjust their buffers to account for the
worst-case network conditions. This would seem to indicate that BitTorrent is bursting packets
rather than releasing the packets in a uniform and evenly spaced stream.

Next I tried using FTP at full throttle and I managed to get an FTP session going at 47 KB/sec
(90% of my peak load) yet the jitter caused by FTP at this extreme rate of throughput was less
than the jitter caused by operating BitTorrent at an average of 10 KB/sec. This would seem to
indicate that FTP is outputting data in a more consistent manner that BitTorrent.

Lastly, I ran some ping tests during some VoIP phone calls using the Lingo service (competitor
of Vonage). I had set Lingo to use the uncompressed G.711 which uses 11 KB/sec for both
upload and download which makes it very comparable to BitTorrent uploading at an average of
10 KB/sec. But as soon as I ran the ping tests, I was shocked to see virtually no increase in ping
times. I realized that this is because the VoIP ATA (Analog Telephony Adapter) device pulses
small packets at exact intervals of 20 milliseconds at 50 times a second.

Smoothing out the packet spacing to reduce jitter

After running these tests, I am beginning to conclude that it isn’t so much the amount of data that
causes excessive jitter; it’s the uniformity of data transmission. If the transmissions are spaced
evenly, other packets from other applications can slip in between the packets rather than getting
stuck behind multiple packets in the transmit queue. So would it be possible to engineer
BitTorrent to transmit data uniformly and what would be the effect? I came up with the
following chart to illustrate what this could mean for peaceful coexistence between
VoIP/Gaming and BitTorrent/P2P.

I calculated that the max-size 1472 BYTE packet takes 28.3 milliseconds to transmit over my
52,000 BYTE/sec broadband uplink. If BitTorrent bursts 3 packets upstream over a 100 Mbps
FastEthernet LAN connection, they will all sit in the upload queue of my DSL modem for 85 ms.
Meanwhile, my VoIP or gaming packets get stuck behind those three packets for an additional 57
ms in the queue. This is shown in the first example in the illustration below with large 1472-
BYTE red packets representing BitTorrent and small 223-BYTE green packets representing
VoIP.

In the second example, I show what happens if BitTorrent simply spaced their transmissions
evenly. It’s still less than ideal but at least it significantly reduces the jitter for VoIP or gaming
packets.

In the third example, I show the ideal scenario where BitTorrent would reduce its packet size to
815 BYTES or less and pulse them out in 20 ms intervals at 50 packets per second. BitTorrent
could essentially create a ―VoIP friendly mode‖ that allows VoIP packets to fit cleanly between
the BitTorrent packets and the increase in jitter would be no greater than 15.7 ms and would
typically average around 8 ms increase. BitTorrent could also have a ―Game friendly mode‖ that
uses 679-BYTE packets at 60 packets per second.

Now it is possible to solve this problem on the network level by prioritizing VoIP and gaming
packets in the home DSL modem upload queue. Unfortunately, I don’t have administrative
access to the modem and implementing VoIP or gaming prioritization on my home router
seemed to have no effect. Packets in the home router get forwarded as soon as they arrive with
100 Mbps Ethernet on both ends so there is nothing to reorder in the queue. More advanced
business-class routers like those from Cisco will allow you to configure the speed of the
FastEthernet connection to match your DSL throughput so that the queue will migrate from the
DSL modem to the router but this isn’t very practical for most people. So it would make sense
for application writers to try and make their application work as well as possible on the majority
of home networks and broadband networks without QoS.

While modifying BitTorrent or the P2P application may not significantly fix the downstream
problem, it would definitely fix the upstream jitter problem which means that people will be
more willing to seed and contribute to the health of BitTorrent. The download jitter may improve
if the dozens of peers sending information to me did it in a more uniform manner, but it is still
possible that those peers will transmit at the same time which still causes packet clumping and
bursting.

So why would BitTorrent or any P2P application vendor want to do this? Why couldn’t they just
say ―it’s not my problem‖? Because it would fix their reputation as a bad housemate and more
people would be willing to seed if it didn’t interfere with their other applications like VoIP or
gaming. Corporate IT departments may also ease their ban on the technology if it doesn’t trash
their Internet connection for the entire office. I am certainly a huge proponent of network-based
solutions for dealing with jitter problems, but improvements in the application can act to further
reduce network jitter even in the presence of existing network-based solutions. Furthermore, the
vast majority of consumers do not have access to network-based solutions and it only makes
sense for a P2P application vendor to cater to this market.

I will try to reach out to BitTorrent corporation to see if there is any interest in this and perhaps
someone would be interested in modifying open source Azureus. It would be a very good feature
to have or it would be a very interesting academic experiment at the very least.

0diggsdigg

Categories: Networking, P2P, Policy Tags:

Comments (147) Trackbacks (9) Leave a comment Trackback

1.

George Ou

June 3rd, 2008 at 07:11 | #1

Reply | Quote

I did use DD-WRT with a Buffalo router and the QoS latency gains were questionable. I
prioritized the ports for Team Fortress 2 (same as Counter Strike) and for ICMP (for ping
testing) and it didn’t do anything for the ping tests and the game play still sucked.
2.

website design

June 3rd, 2008 at 08:52 | #2

Reply | Quote

I think the best program is miu torrent

regards: http://ooyes.net

3.

The 8472

June 3rd, 2008 at 11:10 | #3

Reply | Quote

I must say that your analysis is flawed at best, politically motivated at worst.

First of all: TCP is stream-based from the application perspective, while you can control
packet scheduling and sizing to a certain extent by disabling nagle’s algorithm that’s an
ugly hack. TCP is a stream and it’s the responsibility of the whole network stack and the
routing equipment to take care of fair scheduling. Technically speaking it’s a no-brainer
to give priority to outgoing, small-sized packets, either at the OS or at the router level.
You shouldn’t blame bittorrent for it just because OS or router manufacturers don’t
implement it.

Decreasing packet sizes for a bulk transfer is also horrible from a network engineering
perspective, you get the maximum efficiency out of a latency-insensitive bulk transfer if
the packet size is maximized. That’s the one point of TCP … it auto-tunes itself to the
MTU.

And your take on network neutrality is also wrong, most aims would not prevent QoS
(which we’re talking about here) but only discriminating equal services based on origin,
destination or protocol being used. E.g. a comcast video store vs. a bittorrent video store.
Or vonage VOIP vs. Skype.
QoS to prioritize vonage AND skype over the bittorrent/comcast video store would still
be allowed.

4.
George Ou

June 3rd, 2008 at 14:02 | #4

Reply | Quote

8472 says: "First of all: TCP is stream-based from the application perspective, while you
can control packet scheduling and sizing to a certain extent by disabling nagle’s
algorithm that’s an ugly hack."

So you admit that it is possible for the application to trickle data to the TCP stack so you
are basically admitting that I am technically correct.

I’m not so much "blaming BitTorrent"; I’m saying there’s an opportunity to tweak the
application in such a way that it could address the vast majority of consumers who don’t
have a network-based solution in place and who aren’t network engineers. For the longest
time in the VoIP world, VoIP vendors expected the network to be re-engineered to
support Voice. Skype came along and made the application support the network and
made it "just work" and it took off like wildfire. So perhaps there is a lesson to be learned
here.

The "too many packets" argument is overblown. Most people have the wrong MTU set
for their DSL PPPoE broadband service; it’s the default 1500 BYTE instead of the
optimized 1492. That means the packets are being fragmented anyways so the 815 BYTE
packets isn’t so bad. Furthermore, the number 815-BYTE is only needed for slower
uplink connections and you’ll be able to use optimized 1464-BYTE packets at 50 PPS if
your uplink is faster.

You’re so busy criticizing me for my "political motivations" that you don’t even bother to
read the legislation. I’ve cut-pasted the exact language of the regulation in a comment
above and I’ll paste it here again for you. It absolutely does ban the SALE of QoS
services. So if you offer it to one customer you have to offer it to every customer which is
the forced bundling of services. That means those who don’t need QoS would be forced
to subsidize those who do. Either that or the ISP simply decides that not enough people
want it and they give it to no one.

Markey proposal from 2006:

SECTION 201. NETWORK NEUTRALITY.
(b) IN GENERAL.-Each broadband network provider has the duty-
(3) if the provider prioritizes or offers enhanced quality of service to data of a particular
type, to prioritize or offer enhanced quality of service to all data of that type (regardless
of the origin of such data) without imposing a surcharge or other consideration for such
prioritization or enhanced quality of service;

Snowe-Dorgan proposal from 2006:

(5) only prioritize content, applications, or services accessed by a user that is made
 available via the Internet within the network of such broadband service provider based on
the type of content, applications, or services and the level of service purchased by the
user, without charge for such prioritization;

Conyers-Lofgren proposal from 2008:

"If a broadband network provider prioritizes or offers enhanced quality of service to data
of a particular type, it must prioritize or offer enhanced quality of service to all data of
that type (regardless of the origin or ownership of such data) without imposing a
surcharge or other consideration for such prioritization or enhanced quality of service."

So what you have shown is that I am technically correct and that at best, your own
politics and ignorance blinds you to the truth about the legislation and at worst, you’re
trying to mislead the public on what Net Neutrality legislation is.

George Ou

5.

The 8472

June 3rd, 2008 at 15:41 | #5

Reply | Quote

> So you admit that it is possible for the application to trickle data to the TCP stack so
you are basically admitting that I am technically correct.

yes, absolutely. To quote RFC 1925:

(3) With sufficient thrust, pigs fly just fine. However, this is
not necessarily a good idea. It is hard to be sure where they
are going to land, and it could be dangerous sitting under them
as they fly overhead.

> "If a broadband network provider prioritizes or offers enhanced quality of service to
data of a particular type, it must prioritize or offer enhanced quality of service to all data
of that type (regardless of the origin or ownership of such data) without imposing a
surcharge or other consideration for such prioritization or enhanced quality of service."

exactly my point. The ISP could give "enhance quality of service" to ALL voip traffic,
"regardless of origin or ownership". Which is exactly what everyone wants. Schedule
small voip packets first, then let the bulk transfers through (such as VOD or bittorrent).
This is exactly the kind of network neutrality that does not interfere with QoS.

6.
 George Ou

June 3rd, 2008 at 16:17 | #6

Reply | Quote

I’m not sure what your objection is, we have cheap $50 multi-core processors today that
would hardly notice the extra load of trickling data periodically to the TCP stack. If an
application can be made to be latency-friendly in the absence of other network-level or
TCP stack-level fixes, which is the vast majority of customers, it only makes sense to
cater to that market. Your thinking is too ―in-the-box‖ and this kind of narrow minded
thinking that prevented PC telephony from taking off until Skype came along and made
their software just work. What consumers want is a P2P client that ―just works‖ and
doesn’t require them to buy QoS capable hardware or require them to be network
engineers. I use to build large enterprise networks for a living and I can configure QoS all
day long and even I don’t want to mess with this stuff if I don’t have to.

If these bans on QoS charges are turned in to law, then customers who never use VoIP
services because they prefer traditional land line services will be forced to subsidize QoS
services for those who do use VoIP. Then what if the ISP decides that it isn’t worth doing
if they can’t charge for it? Then you don’t get QoS even if you’re willing to pay for it.

I would love to get free massages too, but the last thing I want is the Government to
mandate free massages because that’s a sure way to effectively ban massages. What
you’re saying is that because you want free QoS, you want the Government to mandate
"free" QoS which is effectively bundled QoS or no QoS. What I’m saying is that this
kind of thinking is short sighted, selfish, and ultimately foolish.

Then what about IPTV service? You realize that IPTV service has to dynamically carve
out a portion of an FTTN broadband pipe with low latency properties right? Are you
suggesting that if a DSL provider like AT&T prioritizes their own U-Verse video, then
they must prioritize all other video streams because they happen to use the same
application? Do you not understand the unintended consequences of this? I’ll give you a
hint. You go passing these short sighted laws and you’ll see DSL providers switch to
HARD allocations of bandwidth for their IPTV service. So even if you don’t want IPTV
service from your DSL provider, you’re not getting that bandwidth back. Now what do
you do smart guy?

7.

The 8472

June 3rd, 2008 at 19:34 | #7

Reply | Quote
 This has nothing to do with subsidizing or forcing the use of QoS. ISPs just take the
wrong approach right now. Instead of fairly throttling all bulk protocols to ease
congestion (and thus make VoiP usable for other customers) they just throttle protocols
that do not represent their interests. The point is that all this is anti-competitive as you
can see with bell canada which rolls out its own video store but shapes bittorrent inc.’s
and vuze’s video store.

So, assuming that the overselling business model won’t go away one form of shaping or
another is necessary anyway and mandatory (as the user can’t choose). Net neutrality is
only about ensuring fairness of this shaping. To make it a proper QoS instead of Quality
of Business.

They alraedy have QoS devices… what do you think are ellacoya applicances doing? So
it’s not a matter of investment but a matter of policy/configuration.

Regarding IPTV. IPTV does not require low latencies, it only requires bandwidth
guarantees. And under net neutrality laws ISPs might be required to wholesale IPTV-
allocations to other IPTV providers if those other providers wish to offer similar services
over the internet. But unbundling is never a bad thing as you can see here in europe.

Anyway, we’re digressing… this is about subpar packet scheduling. I have initially stated
that it’s "possible to a certain extent" and "a hack". You make that into a "possible" and
turn that one into a "should be done".
No, it should not be done, because TCP applications are bandwidth-latency-product- and
jitter-agnostic. Doing any "tweaks" on the application level would be detrimental to
performance _at least_ under some edge cases, most likely under many more conditions.

If you have an issue then ask your OS manufacturer or router manufacturer why their
network stacks don’t do what a network stack should do.

The optimal and feasible solution lies in properly applied QoS-implementations. You
demand an automagic solution from p2p-developers… why should this be any easier for
them to implement – despite having less knowledge what’s currently going on on the
network than the OS stack or the router – than the OS/router manufacturer?

I suggest you aim your demands at them instead… i.e. demand a default-on QoS
implementation that gives priority to latency-sensitive services… and hey, windows even
claims to have a QoS-driver installed by default, maybe the VoiP application would just
have to request higher-priority scheduling and doesn’t… so it might even be the fault of
the VoiP application.

Point is that you’re looking for solution in the wrong place.

8.

Erick Art.
 June 3rd, 2008 at 19:34 | #8

Reply | Quote

I did use DD-WRT with a Buffalo router and the QoS latency gains were questionable. I
prioritized the ports for Team Fortress 2 (same as Counter Strike) and for ICMP (for ping
testing) and it didn’t do anything for the ping tests and the game play still sucked.

The one thing that is CRITIAL to do is enable QoS for both LAN and WAN and
prioritize them to the "premium" option. Also make sure your DD-WRT version is up to
date some of the older one had bugs Try using this firmware ver. this is what I am
running: DD-WRT v23 SP2 (09/15/06) std
Also best thing to test is http traffic latency. If bittorrent slows down your http traffic
enable QoS for http and test. it dramatically changed my experience with websurfing w/
bittorrent running. (another thing i should mention is that I am using rtorrent as my
bittorrent client)

9.

Craysh

June 3rd, 2008 at 19:35 | #9

Reply | Quote

@host
There’s a difference between QoS and Tiered internet. If you’re non-discriminitive, QoS
will NOT be banned.
However, what the ISP’s are doing now IS discriminitive. When they have competition
like Skype or vonage, and a phone com

10.

AaronCompNetSys

June 3rd, 2008 at 20:35 | #10

Reply | Quote

Internal network QoS: Tomato

http://www.polarcloud.com/tomato

Works amazingly well for my sad DSL. I primarily use Xbox Live while Bittorrent-ing.
 Its not magic, but it works as advertised, and once a particular connection is established is
prioritized, no worries.

11.

George Ou

June 3rd, 2008 at 20:39 | #11

Reply | Quote

You’re dodging the question on why you would prohibit the sale of QoS. Now answer the
question. Why do you want the Government to mandate free QoS or no QoS for
Broadband Internet?

You’re changing the subject to an irrelevant topic. Vuze isn’t a "service" that delivers
packets; Vuze offloads their bandwidth costs to the ISP using P2P and has absolutely no
comparison to a CDN Video on Demand model. Throttling P2P application is strictly a
fairness issue because P2P uses the multi-stream cheat to get a disproportionate amount
of bandwidth. Jacobson’s algorithm is being thoroughly exploited by P2P.
http://blogs.zdnet.com/Ou/?p=1078.

It’s also quite ignorant of you to believe that IPTV doesn’t require low latency. You’re
confusing it with Video on Demand which is buffered enough to not need low latency
and only flow rate assurance. IPTV is a live streaming service with very low buffering
primarily because consumers want to channel-surf. You cannot have a half-second delay
any time you flip the channel because it wouldn’t work the way people expect it to when
they compare it to Cable or Satellite TV service. You also need the low latencies if you
want the audio to sync up perfectly with the video.

So you must absolutely prioritize IPTV service with minimal bandwidth guarantees and
low latency guarantees if you want a functional TV service competitive with Cable and
Satellite. This is being prioritized at the expense of all other Internet traffic but you have
to do this or else the system doesn’t work. If you give other Internet-based video
streaming services (which are typically VoD services that don’t need low latency) equal
priority as the Net Neutrality bills require, you kill the IPTV service. Because the DSL
provider can’t afford to lose the IPTV business, they will simply be forced to use a fixed
slice of bandwidth from their FTTN service and never make that available to you or any
other content provider and this would be a horrible unintended consequence of these
rules.

8472 says: "I suggest you aim your demands at them instead… i.e. demand a default-on
QoS implementation that gives priority to latency-sensitive services… and hey, windows
even claims to have a QoS-driver installed by default, maybe the VoiP application would
just have to request higher-priority scheduling and doesn’t… so it might even be the fault
of the VoiP application."
 Again, you want the Government to mandate "free" QoS which makes about as much
sense as mandating "free" car wash service with gas. Sure you can demand it, but you’re
just mandating bundled service which ironically the advocates claim they’re against.

As for the QoS-driver in Windows, I would suggest you do a little more research about
that before you embarrass yourself further.

George Ou

12.

Craysh

June 3rd, 2008 at 21:45 | #12

Reply | Quote

@George Ou
What are you talking about? They’re not "offloading" their bandwidth costs, they pay
(exorbitantly) for their bandwidth to their ISP.
These Internet companies pay much more than the average web operator because the

13.

George Ou

June 3rd, 2008 at 21:52 | #13

Reply | Quote

Craysh says: "What are you talking about? They’re not "offloading" their bandwidth
costs, they pay (exorbitantly) for their bandwidth to their ISP."

Google (YouTube), Apple, Microsoft, Netflix etc pay a ton of money on bandwidth and
CDN costs. They pay their own end of the pipe and the internconnect fees are already
covered by the peering arrangements between the carriers. These companies for example
are not offloading their costs and they really can’t use P2P for their distribution model
because they need in-order delivery so that the user can watch the movie as they are
downloading it.

Vuze uses the P2P distribution model. They pay just enough to seed the content but the
vast majority of the distribution costs are pushed out to the ISPs and their customers.

Craysh says: "If the ISP’s offers "Unlimited" internet at X speed they need to deliver
that."
 ISPs don’t offer unlimited. Comcast for example does not advertise unlimited. No ISP
offers unlimited or guaranteed throughput. All broadband connections are shared.

14.

The 8472

June 3rd, 2008 at 23:38 | #14

Reply | Quote

> Because the DSL provider can’t afford to lose the IPTV business

err… are we talking about a DSL provider or an IPTV provider? Because if you’re saying
that the DSL provider and IPTV provider are not only the same entity but also are
inseparable you obviously get a conflict of interest and an anti-competitive market as all
IPTV competitors would have a disadvantage compared to the DSL/IPTV conglomerate.

So i suggest you make a clear distinction between access providers and content providers,
otherwise it would like you’re advocating lock-in schemes and mono-/duopolies.

> ISPs don’t offer unlimited.

maybe not in their fineprint, but in their adverts. And most people expect what they’ve
been promised and not what some weasels sneak into their contract…. most people don’t
even read it. So it’s a marketing-failure among other things. People have been fed with
big promises, now that they actually use what they have been promised the ISPs reap the
fallout. Deceptive advertising is hardly something one has to defend.

> You’re dodging the question on why you would prohibit the sale of QoS

i never said i would prohibit the sale of QoS. QoS can be sold to customers, as long as it
is applied equally to all services of the same type. E.g. you could sell them a "speed up
VoIP and games"-package. Which would apply to all VoIP services (as far as technically
possible) and not just prefer a specific vendor.

We have something similar in germany (although not technically related). DSL can
operated in interleaved or fastpath mode. The latter has (theoretically) higher error rates
but better RTTs, hence it’s sold as an additional option and frequently used by gamers.
This option applies to all services equally, and yet it can be sold.

> You also need the low latencies if you want the audio to sync up perfectly with the
video.
not really, as long as audio and video frames that belong together are in the same buffer
window it does not matter which one arrives earlier, it’s only important that you have
both of them.

> You cannot have a half-second delay any time you flip the channel because it wouldn’t
work the way people expect it to when they compare it to Cable or Satellite TV service.

people actually prefer VoD over TV, so i’d say they can live with "slow" channel flipping
if you replace it with a low-res preview feature for example. Not to mention that near-
guarantees for latency of 250ms is feasible with good QoS solutions.

> So you must absolutely prioritize IPTV service [...] if you want a functional TV service
competitive with Cable and Satellite.

oh really? is that the only mode of competition? With IPTV you have the advantage that
only that what is currently watched must be delivered. With cable and satellite you have
to deliver all at once through a limited resource (frequencies). Thus you can offer a much
broader choise with IPTV, i’d say this is a quite neat mode of competition.

> Vuze offloads their bandwidth costs to the ISP using P2P and has absolutely no
comparison to a CDN Video on Demand model.

The users serve as your CDN, that’s the only difference. Regarding your claim that p2p
applications cheat… this one is actually simple to solve with QoS and Diffserv. Simply
stuff P2P traffic into a traffic category below bulk and let them compete for the
remaining bandwidth, i.e. that’s what left after allocating low-latency, best-effort and
reliability traffic. The TOS/Diffserv value for that is 0×20 iirc.

And your claim that what ISPs doing right now is about fairness is bogus. If it would be
about fairness ISPs could still allocate all remaining bandwidth (see above, just via DPI
instead Diffserv) to P2P applications. But they don’t they simply hard-cap the traffic to
rediciously low amounts during certain hours (bell), completely sever TCP connections
24/7 instead of shaping them (comcast). See
http://arstechnica.com/news.ars/post/20080602-new-filings-reveal-extent-damage-of-
bell-canada-throttling.html
Does this look like adaptive shaping to ensure fairness to you? It certainly doesn’t to me.
It’s just a "let’s reduce the overall traffic, it’ll save us coasts"-approach, regardless of
necessity.

Another point is: Over here in mainland europe most ISPs do NOT shape p2p traffic.
Even in sparely populated (=> more expensive infrastructure) countries like sweden
where they have internet connections comparable to those in japan (100Mbit symmetric
to the home) there is near to none traffic-shaping. Especially nothing comparable to
Comcast. And that’s despite the fact that sweden is one of the world leaders in
filesharing.
The fact that such ISPs do exist alone proves that it IS possible to build the necessary
 infrastructure and stay competitive… thus it’s a failure of ISPs to build out infrastructure
or a lack of a larger scale infrastructure like open IXPs (the 3 largest IXPs are all in
europe: AMS-IX, DE-CIX, LINX). You can hardly blame the endusers for market
failure.

> Again, you want the Government to mandate "free" QoS

i said

—-
[...] than the OS/router manufacturer?

I suggest you aim your demands at them instead…

—-

where did i mention the govt in that paragraph? Making QoS-hardware/software

available to the enduser who has "issues" with his gaming+p2ping does not involve
politics. I only said it should be part of most network stacks, not that it must be.

The govt should only enforce fairness (no lock-in, no market distortion, non-anti-
competetive etc. etc.) on QoS on the ISP-side, that’s what net neutrality is about at its
core, subpar law proposals do not define net neutrality, they are only attempts at its
implementation.

15.

Craysh

June 3rd, 2008 at 23:38 | #15

Reply | Quote

So what happens when Comcast decides that Netflix new download services are eating
into their on-demand income? Or if Hulu starts getting more popular and people drop
Comcast cable and only keep their internet? With no limits on what the ISP can do, the
connection can be degraded to encourage (read:force) people to keep the ISP’s offering.
In an ideal free market situation, there would be competition so this wouldn’t be an issue,
since you could simply go with another ISP. Unfortunately, most cities only have one or
two choices at the most.

And you’re right, Vuze doesn’t have as much bandwidth costs as some of the major sites,
but they pay for that which they use. Any additional bandwidth that is used across the
network distributing those files are covered by the paying customers on either end of that
connection. If you believe that they should pay additional fees because other people are
using the service, then what’s to stop the ISP’s from charging other companies because
use of their programs consume large amounts of bandwidth too? Let’s take FTP for
 instance: To use FTP you only have to download a client such as Filezilla. Should
Filezilla be required to pay a surcharge every time someone uses their client to transfer
large files via FTP? No, they shouldn’t because the bandwidth that is being used is being
paid for by the person who’s using the client and the person paying the FTP server’s
bandwidth.
Or an even more common comparison: perhaps Mozilla should be charged because the
price in bandwidth they pay for (5.7MB for each copy of FireFox) pales in comparison to
the amount of data transferred by using it (viewing web pages daily etc).
It’s the same situation with Vuze. Vuze pays for the initial distribution bandwidth costs
(several times for each item I’m sure) because that’s all they use. Should Vuze pay extra
because the users of their product are swapping data between each other simply because
they use Vuze’s client? No, because the people on either side of that P2P network are
paying for the bandwidth they are using. So you see, every end of bandwidth usage is
paid for by the people who use it

As for ISP’s not being unlimited, they do not list limits on their site, contracts,
commercials, fliers, or anything else (unless you count Time Warner testing it’s usage
plan in Texas). If no bandwidth limits are disclosed, it’s "un-limited."
You’d be pretty pissed if you bought a car and for some reason it has an undisclosed "30
miles a day" limit imposed on it.

16.

Craysh

June 3rd, 2008 at 23:40 | #16

Reply | Quote

So what happens when Comcast decides that Netflix new download services are eating
into their on-demand income? Or if Hulu starts getting more popular and people drop
Comcast cable and only keep their internet? With no limits on what the ISP can do, the
connection can be degraded to encourage (read:force) people to keep the ISP’s offering.
In an ideal free market situation, there would be competition so this wouldn’t be an issue,
since you could simply go with another ISP. Unfortunately, most cities only have one or
two choices at the most.

And you’re right, Vuze doesn’t have as much bandwidth costs as some of the major sites,
but they pay for that which they use. Any additional bandwidth that is used across the
network distributing those files are covered by the paying customers on either end of that
connection. If you believe that they should pay additional fees because other people are
using the service, then what’s to stop the ISP’s from charging other companies because
use of their programs consume large amounts of bandwidth too? Let’s take FTP for
instance: To use FTP you only have to download a client such as Filezilla. Should
Filezilla be required to pay a surcharge every time someone uses their client to transfer
large files via FTP? No, they shouldn’t because the bandwidth that is being used is being
 paid for by the person who’s using the client and the person paying the FTP server’s
bandwidth.
Or an even more common comparison: perhaps Mozilla should be charged because the
price in bandwidth they pay for (5.7MB for each copy of FireFox) pales in comparison to
the amount of data transferred by using it (viewing web pages daily etc).
It’s the same situation with Vuze. Vuze pays for the initial distribution bandwidth costs
(several times for each item I’m sure) because that’s all they use. Should Vuze pay extra
because the users of their product are swapping data between each other simply because
they use Vuze’s client? No, because the people on either side of that P2P network are
paying for the bandwidth they are using. So you see, every end of bandwidth usage is
paid for by the people who use it

As for ISP’s not being unlimited, they do not list limits on their site, contracts,
commercials, fliers, or anything else (unless you count Time Warner testing it’s usage
plan in Texas). If no bandwidth limits are disclosed, it’s "un-limited."
You’d be pretty pissed if you bought a car and for some reason it has an undisclosed "30
miles a day" limit imposed on it.

17.

George Ou

June 4th, 2008 at 00:37 | #17

Reply | Quote

8472, you started off questioning my knowledge and now you show that you don’t even
grasp the basics. You don’t answer the questions and you chang the subject.

Do you not understand that DSL providers are the IPTV providers? The DSL provider’s
key incentive to building out FTTN infrastructure (AT&T U-Verse) is to be able to sell
IPTV services, it’s also important to the FTTH providers like Verizon FiOS. You remove
those incentives, they don’t build those bigger networks. These are private businesses that
have to pay back their investors. They’re not going to do it for charity just like Google
won’t spend a dime on building Broadband infrastructure or municipal Wi-Fi. When they
build out that infrastructure, they get to carve out a piece of that infrastructure for their
TV business. Verizon uses a separate optical wavelength separate from the broadband
service but AT&T and other FTTN providers dynamically allocate resources from the
broadband service and carve out a virtual circuit that assures sufficient throughput and
low latency. It is understood that FTTN and FTTH providers get to have this IPTV
service to justify their investment and it is understood that this creates better competition
in the TV space.

As for "unlimited", Comcast for example does NOT advertise unlimited speeds.
However, I have always said that I am for more transparency both from a public policy
point of view and a good business stand point. In fact I believe that Government
 oversight on transparency is good for the free market. However, you are being
unreasonable in your ignorant expectation that advertised peak speeds are supposed to be
for minimum speeds.

"i never said i would prohibit the sale of QoS. QoS can be sold to customers, as long as it
is applied equally to all services of the same type."

Yes but you’re supporting Net Neutrality legislation which does prohibit the sale of QoS
so you’re a hypocrite. I’m saying by definition, the FTTN provider should be able to
protect their own IPTV service and by definition they’re not allowing other Internet
content companies access to that bandwidth when the customer is using their IPTV.
However, the customer is in effect saying they want their IPTV service and they have the
choice to not purchase that IPTV service. This is what the consumer wants and the
consumer should be able to demand this type of QoS that does NOT apply to all content
providers.

"people actually prefer VoD over TV, so i’d say they can live with "slow" channel
flipping if you replace it with a low-res preview feature for example. Not to mention that
near-guarantees for latency of 250ms is feasible with good QoS solutions."

Again, don’t confuse the context of "VoD" here. We’re talking TV services for
NORMAL people and what NORMAL people expect out of their TV service. If YOU
don’t like it, YOU DON’T BUY THE IPTV service and you get that entire pipe for the
Internet. What is your problem?

"[...] than the OS/router manufacturer? I suggest you aim your demands at them
instead… "

Again, what does your OS and your routers have to do with download latency? Hello? Is
someone home up there? We’re talking DOWNLOAD. We’re talking about the
consumer’s right to tell their ISP what services and content providers they want
prioritized going over their own broadband link. We’re not talking about the ISP deciding
what to de-prioritize and prioritize. Net Neutrality legislation, which you say you support,
would ban consumer’s choice.

George Ou

18.

George Ou

June 4th, 2008 at 01:23 | #18

Reply | Quote
 I set BitTorrent (official client) to use 10, 28, and 47 KB/sec upstream out of a peak
sustainable upstream of 52 KB/sec.

Then I did a tracert to a common web site like yahoo.com just to figure out the first hop
beyond my home router and determined it was 12 ms minimum latency under clear
conditions.

Then I ran a ping to that first hop beyond my home router continuously as I downloaded
or uploaded data. If you’re in Windows, be sure to set the CMD console buffer to a
minimum of 3000 lines which is something I do anyways and I keep this as a permanent
setting and I set the window to display double the number or vertical lines. I then do a
select all and pasted it in to Excel 2007 (any version will do). Then I did a global replace
to erase everything except the ping number. Then I created a new column that detected 12
ms from that first column of data. This new column is the increase in ping over time that
I get to plot out in Excel.

19.

Jesse

June 4th, 2008 at 01:47 | #19

Reply | Quote

Great article George. I was wondering what tool you used for generating the graphs. I
want to see if I get similar result in my home network.

20.

Craysh

June 4th, 2008 at 02:04 | #20

Reply | Quote

They never say that it would be illegal to buy/use QoS software. But if they use that as an
excuse to throttle competitors, then yes it should be regulated.

21.

George Ou

June 4th, 2008 at 02:15 | #21

 Reply | Quote

"They never say that it would be illegal to buy/use QoS software. But if they use that as
an excuse to throttle competitors, then yes it should be regulated."

Learn to read Craysh. I’ve written it many times and the proposed Net Neutrality laws are
clearly preventing the ISP from offering QoS services. I’ve patiently explained why you
need to permit this kind of service and you’re not even taking the time to read or be
genuine about your arguments. You’re wasting my time here.

22.

George Ou

June 4th, 2008 at 04:16 | #22

Reply | Quote

Craysh says: "So what happens when Comcast decides that Netflix new download
services are eating into their on-demand income? Or if Hulu starts getting more popular
and people drop Comcast cable and only keep their internet? With no limits on what the
ISP can do, the connection can be degraded to encourage (read:force) people to keep the
ISP’s offering."

You’re changing the subject here to Comcast, but I’ll answer it. If Comcast or any ISP
degrades a specific service beyond reasonable network management, that should be a
anti-trust issue and it is illegal. But this is a totally separate issue from banning
consumers from purchasing QoS and harming the IPTV business.

Reasonable network management has multiple forms. Primitive forms try to make things
a little more fair by throttling the bandwidth hog applications but it’s not the ideal
solution since it isn’t completely accurate. This is the cheap and less-than-ideal way of
dealing with the problem but it shouldn’t be illegal. Comcast is going to a more advanced
form of network management that is independent of the protocol by the end of this year
and they’re already starting field trials. Comcast’s new system simply decides that if a
customer uses a disproportionate amount of bandwidth, say for example 2% uses 50% of
resources, then they throttle that 2% down towards 2% of the resources. It turns out that
the congestion problems are usually gone by the time they knock them down to 25% of
the resources and they don’t need to throttle any more. So the hogs still hog a huge
amount of bandwidth but nowhere near as much. When there is no congestion, they go
back to hogging 50% of the resources.

As to the rest of your post, I’ve already explained it before but I’ll keep it brief. At no
point should Vuze or Mozilla or Google or Microsoft or any other company be charged
for interconnect or off-ramp fees under the threat of blockage or unreasonable
degradation. The Internet is a best-effort service and there are no guarantees on
 throughput or latency. As long as there is no specific targeting of a service to harm that
service beyond reasonable network management as I defined in the previous paragraph,
then there should be no problems.

Craysh says: ―If no bandwidth limits are disclosed, it’s "un-limited."

Now you’re just being irrational and unreasonable and you’d be laughed out of court if
you tried to bring that complaint. I’m all for MORE transparency and I would be for
more Government oversight on this matter but you’re just being flat out unreasonable
with that kind of statement.

As for the ArsTechnica article on Bell Canada and the claims from CAIP, there are some
serious problems with CAIP’s allegations about VoIP blockage. They tried to shove 4.5
Mbps of traffic through a VoIP protocol which is clearly ludicrous since VoIP never
exceeds 83 Kbps in the worst case. It seems that they are deliberately trying to scare the
VoIP users when it’s clear that their test was designed to trigger the system to identify a
4.5 Mbps VoIP UDP stream when this would never happen in reality and the fact that no
Lingo or Vonage customers are complaining prove it. Because if you took a 32 Kbps or
82 Kbps VoIP stream and forced it down to 30 Kbps, you pretty much break the latter
completely and make the former very problematic with that much packet loss.

George Ou

23.

The 8472

June 4th, 2008 at 15:01 | #23

Reply | Quote

As i have stated before (which has been ignored): Looks like you (Ou) are confusing the
concept of net neutrality with (potentially flawed) law proposals implementing said
concept. Most QoS advocates are NOT against QoS, it’s useful to use various services
(p2p, voip, vod) simultanously after all. Net neutrality at its core is about preventing
discriminating, unreasonable and anti-competetive traffic-shaping.

You repeatedly cite law proposals. So yes, these proposals might need some work (or
maybe not, depending on how they are interpreted). But you mistake them for the
concepts advocated by network neutrality proponents themselves.

> Again, what does your OS and your routers have to do with download latency? Hello?
Is someone home up there? We’re talking DOWNLOAD.

Oh, and why the hell are you talking about "spacing" packets if this is all about
downloads? Spacing packets would only make (a small amount of) sense if we were
 talking about upload, at least there wouldn’t be any intermediate steps. If you talk about
download then this is a lost cause as packets get rescheduled en route.

But we are not talking about download, we ARE talking about upload, looking at your
own article would tell you as much:

> If BitTorrent bursts 3 packets upstream over a 100 Mbps FastEthernet LAN connection,
they will all sit in the upload queue of my DSL modem for 85 ms.

And what is comcast traffic-shaping? Right… bittorrent-uploads (seeding). So Net

neutrality in this context is also about upload. Although net neutrality _in general_ is
about fair QoS regarding download AND upload.

24.

The 8472

June 4th, 2008 at 15:01 | #24

Reply | Quote

i couldn’t say it any better: http://arstechnica.com/news.ars/post/20080604-arpanet-

architect-bring-fairness-to-traffic-management.html

25.

Hotkees

June 4th, 2008 at 15:01 | #25

Reply | Quote

I don’t have any problem with Bittorent. Last night while sleeping I downloaded 10
movies. Before I left for work I disabled Bittorent in the Network Properties. BT uses 2
ports and I simply uncheck them. Next time I want to download, I’ll go in and turn the
ports on.

26.

notgonnatellya

June 4th, 2008 at 18:58 | #26

Reply | Quote
8472 said
—–
You repeatedly cite law proposals. So yes, these proposals might need some work (or
maybe not, depending on how they are interpreted). But you mistake them for the
concepts advocated by network neutrality proponents themselves.
—–

If the laws are can be interepreted in such a way that it outlaws QOS, you should assume
that that interpretation will prevail at some point. Even if you can’t believe that, you can
be certain that there will be plenty of costly lawsuits making that claim.

8472 said
—–
err… are we talking about a DSL provider or an IPTV provider? Because if you’re saying
that the DSL provider and IPTV provider are not only the same entity but also are
inseparable you obviously get a conflict of interest and an anti-competitive market as all
IPTV competitors would have a disadvantage compared to the DSL/IPTV conglomerate.
—–

Using your logic, you should be fighting for legislation that forces Echostar to let
directTV use their satellites to transmit video. If the
cable company wants to set aside X MB/S for their video service, I see nothing wrong
with that. Would you also argue that I should be able to sell my gasoline using my local
Chevron’s Pumps?

Infrastructure isn’t free. I object to them giving preferential treatment to some external
video suplier over another. I’d also object if
they made using another IP more expensive than it currently is. But there’s a difference
between Skype or Vonage, which don’t require much
bandwidth, but do require QoS, and a Cable Network.

If you’re actually arguing that Verizon can’t reserve part of their network exclusively for
their video service, then you’ve lost me, and I’ve
been on the NN train for several years. If you don’t like that plan, you can go the route of
Lafayette Louisiana, and have your city build it’s own Fiber network…oh wait, they’re
providing their own video service as well.

If your goal is to get people to call their congressmen up and tell them to oppose net
neutrality, you’re doing a good job.

With that said, your link to the engineer that helped create arpanet was interesting. I’m
fairly certain that it’s an extremely inefficient use of resources and will likely mean
everyone
ends up with less up and down bandwidth, but it would force real numbers when selling
service.
 Personally, I think a better solution is that that figure is the minimum level of service, but
that someone can use more of it, (albeit at a lower priority than anything up to the min) if
its’ available.

27.

notgonnatellya

June 4th, 2008 at 19:06 | #27

Reply | Quote

There’s no order to them. 5/31 at the top, which goes to 6/1……6/1 at the bottom which
goes to 6/2 then 3 then 1 then 4

It makes it very hard to follow threads.

28.

George Ou

June 4th, 2008 at 19:07 | #28

Reply | Quote

I called tech support and they said it’s a bug in this blogging module. Maybe I should ask
them to reboot my instance or something.

29.

George Ou

June 4th, 2008 at 19:11 | #29

Reply | Quote

I was the one who introduced Nate Anderson to Anagran’s Flow Management technology
and I’ve stated in these threads that Anagran’s protocol agnostic solution is a much better
solution. However, I don’t believe that Government should outlaw inferior and less
expensive methods. The market place should shame ISPs in to using the better solutions
and this is what happened with Comcast.

30.
 The 8472

June 5th, 2008 at 14:10 | #30

Reply | Quote

@notgonnatellya

> Infrastructure isn’t free.

right. That’s why you pay subscriber fees for your broadband access. Just like telco
infrastructure isn’t free and you pay your telco fees. That doesn’t mean the telco g

31.

George Ou

June 5th, 2008 at 14:19 | #31

Reply | Quote

8472 says: "right. That’s why you pay subscriber fees for your broadband access. Just
like telco infrastructure isn’t free and you pay your telco fees. That doesn’t mean the
telco gets an exclusive/reserved part of that line to shove other services towards you. At
least over here telcos are forced to unbundle and resale services and it works quite well as
it allows smaller competitors to achieve greater coverage by doing wholesale in some
areas and run their own services in other areas."

In Germany, my friend gets IPTV service from his VDSL FTTN provider BUNDLED.
When you choose that service, it MUST have QoS in the form of bandwidth guarantees
AND latency guarantees.

8472 says: "My point is that reserving bandwidth is ok, as long as the user has the choice
what to reserve the bandwidth to."

You do have a choice, you don’t have to order the IPTV service if you don’t want that
bandwidth reserved. NOBODY buys IPTV service with the expectation that BitTorrent
will mess up your TV service and you’re being silly to suggest otherwise. It’s not shoved
down your throat at all, you have a choice.

The consumer also should have a choice to tell their ISP what services they want to
prioritize. You could have a situation where a household uses two Video on Demand
services but they want one to get more priority than the other. So why shouldn’t the
consumer tell their ISP to prioritize downstream iTunes more than XBOX Market Place
or vice versa? The point is that for downstream prioritization, the user has to rely on the
ISP’s gear because it is the transmitting device can exert the most control. We’re not
 talking about some sinister plot where Apple pays an ISP to lock out a competitor like
Microsoft or vice versa and certainly I would be open to making that form of exclusive
access illegal. But we’re talking about consumers choosing to prioritize a certain web site
above other web sites and the consumer should be able to block or prioritize any service
they want.

32.

Clifton

June 6th, 2008 at 01:17 | #32

Reply | Quote

Recent interesting post on how some colleges are dealing with bit torrent:
http://netequalizer.wordpress.com/2008/05/17/curbing-riaa-requests-on-your-student-
network/

33.

Jacob

June 6th, 2008 at 03:57 | #33

Reply | Quote

There are some great solutions here, but I have an ultimate solution that I use. Route your
bittorent traffic over a VPN service like Relakks (there’s many others out there). These
low cost services allow you to route all your traffic through one connection. Solves the
latency hands down, you’ll still need to throttle in your app as not to choke off your pipe,
but the effect of a VPN is that all those connections behave as one. It also adds privacy so
you have less to worry about from the RIAA. Relakks will result in a slower top
download speed, but I’ve found others which will run at full speed.

34.

Saka

June 18th, 2008 at 17:50 | #34

Reply | Quote
 I had the same problem with BT lagging everyone at home as well, I found that if we
limit the upload rate on our BT applications, it doesn’t cause as much of a problem for
others…

35.

Slade Wilson

June 18th, 2008 at 17:51 | #35

Reply | Quote

The 8472 said:

"@notgonnatellya

> Infrastructure isn’t free.

right. That’s why you pay subscriber fees for your broadband access. Just like telco
infrastructure isn’t free and you pay your telco fees. That doesn’t mean the telco gets an
exclusive/reserved part of that line to shove other services towards you. At least over here
telcos are forced to unbundle and resale services and it works quite well as it allows
smaller competitors to achieve greater coverage by doing wholesale in some areas and
run their own services in other areas."

If you say so, but it sure doesn’t sound like a free market to me. Should the cable
companies be forced to unbundle? What about satellite providers? Gas stations?

Why do you single out telcos? If AT&T or Verizon spend billions to create upgraded
fiber-based networks, why should they be forced to allow other providers to use those
networks when said providers didn’t invest a dime?

When there was only one network (the telephone network), an argument for heavy
regulation, etc. made some sense. But now there is competition: if I don’t want TV from
Comcast, I can get it from AT&T or DirecTV. If I don’t want Internet from AT&T, I can
get it from Comcast or Hughesnet or a fixed wireless provider or even a cellular
company. And so on.

If you force such unbundling, what you are really doing is discouraging such
infrastructure build-outs. Why should AT&T spend all that money to build out a new
fiber network? They’ll never make it back on $25/mo or $35/mo DSL services.

36.

Peter Gerassimoff
June 27th, 2008 at 04:17 | #36

Reply | Quote

The problem isn’t with the last mile link, but in the ISP itself. The latency seen is due to
bandwidth constraints somewhere along the path of the VOIP traffic. Low latency is an
artifact of a IP network with spare capacity. The spare capacity means that no packet is
lost and all packets have a low consistent latency. VOIP doesn’t really need low
latencies, but stable latencies. A call to the exact opposite point in the world will need
68ms just for the trip of 12.5K miles alone at the speed of light. And that is best case.
Most people won’t notice even a 100ms delay from what you say to be heard on the other
end. Even a satellite based phone call has a minimum 300ms delay with one hop (you to
POTS to sat to POTS to other). What gets VOIP screwed up is latency changes. Some at
200ms and some at 5ms will cause many low grade VOIP systems to mess up. But a
VOIP system that buffers 250ms at least would never have any problems.

But that variation is not just due to your end, but any blockages down the road. If an ISP
is shaping bittorrent traffic at his forwarding connection, the packets will pile up in your
router regardless of what BT does. Even 10KB/s will cause them to pile up. Yes slower
and less often, but they will pile up even with one per connection. The BT doesn’t see
that TCP has to retransmit that 1.5KB packet any number of times to get it through UDP.
It just sees 1.5KB of upload. And it will send about 6.6 such packets per second using
some moving average plus some overhead ones as well. But your router just sees the
average retransmissions required times 1.5KB IP packets times 6.6 or 10′s of KB/s usage
in the upload side. Those latency spikes you see are the IP retransmit packets piling up.
And that retransmission can be required through any given part of any BT peer
connection going through a site that is capacity constrained whether its deliberate or not.

The VOIP can similarly be messed by somewhere a network provider dropping packets at
random to shoehorn traffic through a bottleneck. Strict network neutral carriers of that
traffic actually would help streaming traffic by randomly dropping the packets. Since the
packets are dropped randomly, the latency would stay within the typical gaussian
distribution and slowly change allowing the applications to adjust. Since shaping requires
intelligence, dumb networks have much higher capacity, 2-10 times as much as smart
shaping ones. That extra capacity usually works to drastically reduce dropped packet
rates and thus make latency both low and stable. Which makes them look like the ideal
non blocking network. That is why most non QOS networks look like ideal ones. Also a
faster last mile takes care of more problems. That is why a 100/100mbps FTTH
connection doesn’t really have these issues. The bottlenecks are all somewhere else. Even
a 1.5KB packet is only 0.12ms on such a connection. 8000 packets per second can make
any problem with 100-200 peer connections seem effortless.

Think about a HDTV signal. Its 19.2Mbps. Even composed of 1.5KB packets, thats 1600
packets per second. 250ms delay means 400 packets are buffered. One or two packets
either way won’t cause any problems. And I had U-Verse. The HD channel switch times
 were more like 2-5 seconds, not 1/2 second. So latency hiccups shouldn’t affect VOIP
traffic in the last mile.

37.

kelly

March 29th, 2009 at 22:48 | #37

Reply | Quote

for me…using bittorrent at all causes online game play to be erratic

downloading at 3kb..downloading at 300 kb

uploading nothing or uploading at the max speed

it all causes games to feel screwy like there is some kind of packet loss or something

it doesnt have anything to do with limiting the upload…

38.

Switeck

June 10th, 2009 at 06:30 | #38

Reply | Quote

We do not know from the article alone:

1.What BitTorrent client is used in testing.
…although in reading the 100+ replies, about the 100th reply in:
"I set BitTorrent (official client) to use 10, 28, and 47 KB/sec upstream out of a peak
sustainable upstream of 52 KB/sec."
2.What settings are used in the BitTorrent client, BESIDES upload and download speed
limits!
3.What torrent/s are the tests run on, specifically how many peers and seeds are
there…and if the torrents are new, old, or even POISONED!
4.Firewalled conditions in BitTorrent?
5."Clean Line" tests…we don’t know if the tests were done on an internet connection that
is not also being hammered by spyware/viruses both incoming and outgoing…or old
BitTorrent traffic (DHT and peers/seeds retrying torrents you long-ago stopped).
6.If the tests’ computer/s is also running networking software and hardware known to
interfere with BitTorrent traffic or the internet connection in general. (Zone Alarm, I
choose you!)
7.What ISP service is being used? (some are real lemons who cripple the WHOLE line
 when BitTorrent traffic is detected!)
8.Even how testing was done between upload speed changes can have a huge effect on
the outcome. A short "cooldown" time between each upload speed reduction might make
the latency temporarily worse, as both the line and the BitTorrent client hasn’t adapted to
the new conditions.

In conclusion, the test is not scientific and the conclusion should not be treated as fact.

This old article is noteworthy in explaining much of the problem:

http://www.stuartcheshire.org/rants/Latency.html

Lastly, a BitTorrent/uTorrent troubleshooting guide:

http://forum.utorrent.com/viewtopic.php?id=15992

39.

George Ou

July 19th, 2009 at 07:56 | #39

Reply | Quote

@yacc143
Here’s another hint smart guy. Read the actual article. It tells you that the speed was
limited to 10% of the link speed yet it caused huge amounts of jitter, albeit lower
frequency when the percent saturation is lower.

40.

leuboo

December 11th, 2009 at 19:08 | #40

Reply | Quote

I now use widevpn to unblock so mcuh high ping

41.

Phil

February 2nd, 2010 at 20:35 | #41

Reply | Quote
 I was disappointed to see this thread devolve into a lot of heated yelling. The technology
exists right now to allow your Bit Torrent transfers to peaceably coexist with VoIP and
any other latency-sensitive applications on the same Internet connection, and the politics
can be limited to those parties sharing the same link (e.g., you and your roommates). No
need to involve the ISPs.

The answer is to implement a traffic-shaping router that prioritizes and controls your
upstream traffic, which you can do directly. Nearly every residential Internet link has a
much slower upstream than downstream, so addressing that bottleneck will make a huge
difference even if nothing at all is done on the downlink by the ISP.

I use a small single-board computer running Linux with the traffic shaping features
turned on. Its has two primary functions. First, it rate-limits the traffic it sends into my
broadband modem to ensure that a queue never builds within the modem – that’s what
causes the long latencies when Bit Torrent gets going. Second, it examines the IP Type of
Service (TOS) bits for the 6-bit DSCP (Differentiated Services Code Point) field. This is
the packet’s priority, as locally defined by me within my own network – it doesn’t
necessarily have to have any meaning once a packet gets to my ISP.

Some Bit Torrent clients such as Azureus/Vuze already provide a configuration page to
specify the DSCP value that should go into every packet it generates. (Unfortunately I am
running Vuze on OSX which still has a bug preventing this feature from working, so I am
temporarily having to use the iptables facility in Linux on the router to recognize my Bit
Torrent traffic by other means, specifically IP address and port number).

By throttling the flow into my modem, Linux pulls the queues back into itself where they
can be managed — where a new high priority packet can be sent ahead of an earlier, low
priority packet. Without this throttling, there would never be a queue in Linux because its
Ethernet link to my modem is far faster than the link beyond.

The third feature – and one that’s surprisingly useful all by itself – is to enable Stochastic
Fair Queueing within each priority level. So even when several streams all compete at the
same priority level, they are served in a round robin fashion; one bandwidth hog TCP
connection cannot force everyone to wait for an entire window to drain each time before
sending a packet.

Google for something called the ―wonder shaper‖ – this is the name given to these scripts
by their original author. They really do work wonders.

42.

Broken_bazooka

February 28th, 2010 at 18:23 | #42

Reply | Quote
 George Ou, you should have listened to what The 8472 had to say instead of just
reiterating yourself and calling him names.
From reading both of your posts, it is quite obvious that you two are in way different
leagues when it comes to skills in networking.

In networking, there’s the business standpoint, and there’s the technical standpoint. When
looking at history, the business standpoint doesn’t mean much, while the technical
standpoint is what stands the test of time.
Keep that in mind.

43.

Ron Barone

March 12th, 2010 at 12:46 | #43

Reply | Quote

You can use rate-limiting to proactivly advertise a congested network.

Cisco IOS supports this (aka traffic policing) but I have never seen this feature in home
routers.

Here’s a rundown of this… as I didn’t read the above articles since they were more
motivated at ISP’s control over data and not the problem you posted. Slowing down
bittorrent is a pill to fix the symptom under ideal circumstances. This would not be the
correct way to handle this since other transfers (FTP downloads etc) would cause your
queue to fill up. It isnt bit-torrent it’s simply the faster connection fire-hosing your
connection with packets (i will continue to explain)

Let me start by saying we’re talking TCP not UDP here. As you may already know the
only way to communicate to the sender to SLOW DOWN because you’re queues are
filling up is to drop the packets. When packets are dropped the upstream router sees a
congested network that’s unable to accept the packets ―that fast‖ and starts slowing down
the streaming of the packets. It’s much like a funnel (queue) where each connection is
pouring water (sending packets) into the same funnel (queue) and when you start spilling
(packet drop) the person that caused the spill (packet drop) slows down and the cycle of
slow-pour/fast-pour (fair packet queue) continues.

Now… What you can do is make your receive queue on YOUR equipment smaller and
make your committed bandwidth rate less (~1-5%) than your total available bandwidth.
This will cause the packets to be dropped keeping your queues mostly empty (except
possibly when starting a transfer). Doing this has each connection slow down before they
actually start building up packets (funnel spill over lol). This means any packet (ICMP,
UDP, TCP) is placed into a mostly empty queue making data reception faster because
everyone was told to slow down a small bit.
 This, of course, is exactly what happens now, so dropping those packets before it
becomes a problem is not a bad thing. It just causes it to happen sooner smoothing out
that nasty queue problem. It’s much like allowing your funnel to fill up but hitting the
person in the knees if they cause that water level to rise hogging the exit of other people’s
fluid. Ok so maybe that’s a bad analogy but I hope you get the point.

Also any kind of QoS or traffic shaping is complicated on shared networks (cable) due to
the up to millisecond calculations on everyone’s committed access rate and the total
available bandwidth available. The QoS doenst solve the problem it just makes the
problem go away for select items. This doesnt exactly help when you are playing games
on dynamic ports, but it does when it’s VoIP and the ports are predictable. Remember
though, VoIP QoS needs to be done BEFORE it gets to your router or you need to start
dropping some TCP packets to slow those other bastards down.

Some of my info may be inaccurate… It’s been awhile since I’ve had to manage
networks… so this information is about 10 years out of the back of my head. Search rate-
limiting or cisco traffic policing for more accurate information. In a perfect world the
congestion bit would be used better to proactively notify senders they were sending too
fast… In the mean time, dropping packets is the most effective way to handle this from
an end receiver.

44.

Sen

March 24th, 2010 at 18:50 | #44

Reply | Quote

…. so how would you fix it? am i the only one not getting it?

45.

Greg

March 29th, 2010 at 10:53 | #45

Reply | Quote

I use Vuze and this has always been a terrible problem for me especially as i’m sharing
the internet connection with 5 other guys.
In your article You point out that this seems to be a problem of too much buffering.
So i looked around the Vuze settings, and found entries called ―Socket SO_SNDBUF‖
and ―Socket SO_RCVBUF‖ (Located in Connection->Extended Settings). I set them both
to 3000 (2x the MTU), and now it seems to run much more smoothly. Haven’t done any
 measurements yet, but i feel quite satisfied now, as my pings have improved
substantially.
So, thanks a lot for the insight!

46.

Naresh

June 26th, 2010 at 23:43 | #46

Reply | Quote

Hi George Ou,
I have a little question in my mind. As i download any torrent through BITtorrent, i
noticed that there is so much amount of uploaded data consumed. Can u give some idea
about this. ..

Thanks..

47.

Brendan

July 31st, 2010 at 20:56 | #47

Reply | Quote

George,

Bittorrent can easily cause the network to get bogged down (latency-wise) by what I
would call an ACK storm. It sounds like this is what is happening in your case. Tomato
(not sure about DD-WRT) has an option to minimize this impact in the QOS settings.
When I did a similar test, torrent downloads did not have as large an impact as they had
before.

Regards,

–Brendan