You Won’t Believe Who is Behind Public Bittorrent Trackers Going Down So Often

Bittorrent Trackers go down a lot. If you’ve ever used torrents, you’ll know the frustration of one that won’t start, because the trackers are down. The reason for it will amaze you!

Ok, I thought I’d give a go at a buzzfeed-style headline, but the question is one that’s probably run through your mind at one time, and you probably will be a bit shocked/surprised at the answer. The answer simply put is…

You!

Well, not just you, but you and a bunch of people who have gotten into lazy habits. Still, it’s something easily preventable, and something You can do something about.

You see, one of the leading causes of tracker instabilities is just one thing – Traffic. Take TPB for instance. Right now, on the front page it says

51.305.526 peers (39.251.829 seeders + 12.053.697 leechers) in 5.953.886 torrents.

The actual value may differ when you read this, but let’s just take that as a starting point, and round it down to 50 million.

The average bittorrent client updates to a tracker roughly every 30 minutes. It’s an approximation that helps keep the maths a little simpler, but also takes into account starting, stopping, etc. So don’t take these figures as the worlds most accurate figure, just some ballpark ones to work from.

So, 50M peers, updating every 30 mins. How many per second is that? Well, 1800 seconds in that 30 minute period, so it’s

\frac{50,000,000}{1800}=27,777.77 per second

Sounds like a lot doesn’t it? And that’s just the AVERAGE. Something very popular comes out, and that figure will rise, but we’ll round it to a whole number, and again keep it simple.

Each TCP tracker request is (again to make things simple) about 100 bytes long. So there are 27,778 requests of 100 bytes hitting the tracker every second. That’s 2,777,800 bytes every second, or close to 2.8 megabytes (or 16.8 Mbit), each and every second, and that soon adds up. 2.8megabytes/second average works out to about 7.25 terabytes of data over 30 days. That’s a lot of data.

Then we get to the responses, which is when the tracker sends data back to the client. We’ll assume 25 peers are being sent back, because there are always small swarms, which won’t have enough peers for the default 50. So we’re looking at maybe another 260 bytes per response, or 7.2Mb/sec, taking the 30-day total to 18.6TB sent out, and ~25.8TB total.

That is, of course, for TCP-based trackers. UDP ones are slightly different, since it uses a send-receive-send-receive pattern, but overall it works out to 114 bytes sent, and 186 bytes returned which at our loads means 3.1Mb/sec received and 5.1Mb/sec sent, giving 30-day totals of 8.2Tb received, and 13.4Tb sent (21.2Tb total)

And of course, none of this takes into account overheads, which adds about 3% onto TCP traffic, and half as much for UDP traffic (a better example is here)

It sounds a lot, and it is; and what’s more, most of it is preventable.

But I don’t understand. Example?

In the case of the Pirate Bay, when a torrent is uploaded, it is actually modified. Not in a way that affects the hash, but though the addition of trackers. Used to be 3 now it’s 5, which is odd since I only had one tracker – the Pirate Party of Canada’s CaPT tracker – on the torrent of No Safe Harbor when I uploaded it. But now there are five if you get it via TPB, and they are:

  1. udp://tracker.openbittorrent.com:80
  2. udp://tracker.publicbt.com:80
  3. udp://tracker.istole.it:6969
  4. udp://tracker.ccc.de:80
  5. udp://open.demonii.com:1337
    And also
  6. DHT
  7. PEX

So what’s going on then?

Well, let’s look at things as a progression. As you can see they’re numbered. So let’s start with tracker number 1.

Tracker 1

Every client that runs this torrent will announce to this tracker. We all know that and understand that, right? It’s pretty obvious. So let’s take that as read, and move on.

Tracker 2

Now, the second tracker is a bit more nuanced. In theory, every client should announce to this tracker as well, but the reality is a little different. See, not all clients support multiple trackers If we assume that it’s about 1% of all clients, that means that on tracker 2, you will find listed 99% of the peers that are on tracker 1. What you won’t find, however, is anyone on tracker 2 that isn’t on tracker 1.

Tracker 3-5+

It’s the same story as tracker 2 above. In fact, the same is true for any number of additional trackers.

“Tracker 6” (Distributed Hash Table)

This was developed in 2005 to deal with the problem of unreliable trackers. At that time, the concept of multiple trackers was still not widespread, and not many clients handled them. DHT, based loosely on Kademlia, was the more robust solution. No matter how many trackers you put onto a torrent, they can all go down. DHT can’t, because it is not ‘one node’, but an amorphous ‘cloud’ of peers.

“Tracker” 7 (Peer EXchange, or PEX)

This is completely independent of the trackers. When you’ve connected to some peers, and have it active, it will swap peer info with connected peers. Basically, If you’re connected to peers W, X, Y, and Z, and Peer W is connected to A, B, and C, you’ll tell W about X, Y, and Z, and W will tell you about A, B, and C. It augments trackers, and means that even if there were two disparate swarms on a torrent (as can happen with DHT-only torrents, thanks to the incompatable Vuze and mainline systems), as soon as a single link is made (by a client that can handle both, for instance) the swarms start to intermingle.

So what we’ve got is a tracker that has all the peers (1) a bunch of other trackers that have most of the same peers and no new/different ones (2-5) a an alternative way to get peers (DHT, aka 6) and a way to keep all of them in one big group. So guess which one you can do away with without causing any issues?

Yes, it’s the duplicate trackers.

But what if a tracker goes down? Well, if the first tracker goes down, then yes, second tracker saves the day! (Hurrah!) And 3-5 are still completely redundant and a waste (Bugger!). And if it’s tracker 2-5 that goes down, who cares? It’s not made any different to the primary tracker. NOT ONE BIT. Even more fun comes when you realise that any client that supports multiple trackers, tends to also support DHT, and so DOESN’T NEED multiple trackers anyway.

Going back to the example, people running the original torrent, with the CaPT tracker on it, and those getting the torrent from TPB are going to be two different swarms, aren’t they? Well, yes, and no. If that was all we knew, then yes. If, however, someone on both groups had DHT running, then they’re going to become one swarm, thanks to PEX. DHT saves the day again!

But the question has to be asked, just WHY do people keep adding more trackers to a torrent? The answer is simple, and comes from a misunderstanding of how a tracker works, and what’s being done.

Why are people doing this?

In the main, people add trackers for one of two reasons, either a) to find more peers, or b) to try and add some resiliency. So let’s address these.

More Peers

More trackers = Worse for everyone

More peers make it faster; so if you want more peers, you add more trackers, because you can’t get peers without trackers, right? No. Once you’ve got the appropriate number of peers for your connection, trying to add more will just make things slower. Or, you don’t think that’s all the peers there are, and there must be some that no-one knows of right?

Both of these are due to errors understanding bittorrent. The first is a common fallacy. More peers does not mean more speeds. The fastest setup is the one that matches your connection, and more peers will make you slower (thanks to the increased overhead). As far as adding more peers through adding trackers, remember this. Those same people, with the same torrent, already have the same trackers you do, so they’d have to delete the ones already in the torrent, and add the same one you have, for them to be new peers.

Not very likely is it? No.

Add resiliency
This was the reason the ability to add multiple trackers to a torrent was created in the first place. So it clearly makes sense, right? No. Not really. In fact, the widespread use of it has caused more tracker downtimes than anything it’s prevented. As one tracker goes down, load on other, similar trackers (especially the 5 listed on TPB/KAT) stays pretty much the same, and may even increase slightly (as people hit ‘update tracker’ to see if they can get a down one to work). These then add up into more traffic, which can then take the working trackers down. It’s actually a vicious cycle where the misguided attempts to mitigate problems are the root cause of the problems.

What’s more, in 2005 a better solution was released (Vuze first, Bittorrent Inc  a few weeks later) for resilience. It’s DHT, and its means of resilience actually mirrors bittorrent for its swarm efficiency.

With a traditional download, the more people trying to download from a server, the slower it goes, until it’s eventually unresponsive, or at least very slow. Likewise a heavily loaded tracker starts to become less reliable the more popular it gets. Bittorrent shares the bandwidth of the download between many peers, and thus spreads the job between many. Take the initial seeder off after a short while and the downloa

facepalm

ds can still continue. DHT does the same with the job of tracker, sharing it between the thousands, meaning one or two or twenty nodes going offline won’t even be noticed.

Despite the fact it’s been around for more than 8 years at this point, people still often don’t understand DHT and what it is. You get advice for self-professed bittorrent ‘experts’ telling you that it has to be turned off for certain sites (usually their own activity-logging site where DHT undermines their revenue stream), etc. It’s so bad, that there’s a whole page on TorrentFreak about various DHT myths being debunked from 2009 and they STILL get repeated.

Other reasons
There are sometimes other reasons trotted out as well, and they’re equally fatuous. I once asked a certain release group’s representative why they had multiple trackers, and was told it was “so the torrent would start fast for people”. Thing is, it starts no faster with 5 or 50 trackers as it would with 1, and while DHT can be a bit slow with a few peers, DHT and a single tracker works quickly enough most people can’t tell if the sourcing is DHT or trackers.

Another reason given was as “wanting to show the most seeds and peers on sites”. Again, yet another ridiculous idea. Sites don’t scrape (that’s when they pull stats from trackers) multiple trackers, and then add them up; they pull from one, and use that because otherwise you could turn 50 into 500 just by using 10 trackers. One tracker works just fine for this.

So what can be done?

The key thing is, stop adding trackers. Sites and groups like TPB need to start reducing tracker numbers as well – and TPB is a big problem, adding 5 trackers to every torrent. What makes it more ironic is that 18 months ago, TPB founder Fredrik Neij submitted proposals to reduce tracker loads, and save trackers money, and yet the TPB site is adding massively more to the load (and cost) of running a tracker than the problems he was dealing with.

The second thing is, Turn on DHT if you haven’t already.

The whole thing about DHT is that it can find peers without them being locked on a specific tracker. If they are running the torrent, it can find them and link you to them. It also doesn’t go down. Every reason people have for running multiple trackers is quite literally something DHT does far better.

Sure, it seems like ‘everyone knows’ that adding more trackers is better, but everyone knew the Earth was at the center of the universe, everyone knew radium was good for what ails you, and everyone knew Alabama-Auburn was going to overtime.

It turns out, what everyone knows, is usually crap.

And now you know better.

  • jasperwillem

    What would happen if you have to force your Torrent Client to use more trackers as the first path by default in the settings? Would that help? This way, most downloaders would not connect to the extras (2 and above paths). And guides could tell that for a non-functioning torrent you can enable it (torrent by torrent), this way the overconnecting should be weeded out pretty soon.

    • ktetch

      No, it wouldn’t help. Just delete the trackers. And make sure DHT is enabled. That’s all you have to do.

  • jasperwillem

    What would happen if you have to force your Torrent Client to use more trackers as the first path by default in the settings? Would that help? This way, most downloaders would not connect to the extras (2 and above paths). And guides could tell that for a non-functioning torrent you can enable it (torrent by torrent), this way the overconnecting should be weeded out pretty soon.

  • dfrnz

    You’re right, the title for this excellent article was pretty crappy. Very informative, thank you for the post.

  • dfrnz

    You’re right, the title for this excellent article was pretty crappy. Very informative, thank you for the post.

  • Freonpsandoz

    I can’t use DHT. I have tried many times and it always causes my router to fail or reboot after a short time, killing my VPN connection, and causing the VPN monitor to terminate uTorrent to prevent exposure. The article you reference on DHT myths admits that this one is NOT a myth: many routers for the home market cannot handle the very large number of connections DHT makes. Most of us can’t afford to buy high-end routers when the one we have works for EVERYTHING EXCEPT DHT! So, why don’t you get the torrent client developers to do what I have been telling them for years without success: FIX THE DAMN PROBLEM WITH DHT KILLING HOME ROUTERS! Allow us to configure the resources used by DHT as we can for other uTorrent system resources.

    • ktetch

      the myths article was one I co-wrote in 09, almost 5 years ago. Even at that point hte number of routers that fell over due to DHT was low. I doubt there has been a single consumer one released in the last 5 years that falls over. There may still be some ISP-provided ones that fall over, but that’s not anything we can do anything about.

      The problem is not something that can be fixed by the client, it’s a 100% router issue. It happens because the ram gets filled with the UDP requests. Either because the ram is too small or poorly managed. giving people the chance to play aroudn with those settings will not make things better, because it doesn’t address the underlying issue. And it’s also just another area for people who don’t know what they’re doing, to mess about and screw things up with settings they think will ‘game’ the system.

      • “…that’s not anything we can do anything about.” Who is the “we” here? Is it completely impossible for torrent clients to provide any configuration parameters that would limit the total number of UDP requests to the router? As a retired software engineer, I strongly suspect that just isn’t true.

  • ckm8kng2

    Wow,bravo on the article,really spot on,especially people messing around with settings that will just end up slowing things down even more. My one question would be…..I feel everything you said applies for public trackers. On the other hand,if your d/l ing from a private tracker should you still turn off DHT? I was told by private tracker Im a member of to turn it off. 9 times out of 10 Im d/l ing from private but the times I am on public tracker I just leave DHT off and I dont notice any difference in speed at all. Do your rules apply to private trackers and if not,should I turn it on for times Im on public tracker like TPB?

    • ktetch

      Yes, turn it on. people who tell you to “turn it off” are morons, who don’t know what they’re talking about. And don’t call them ‘private trackers’, call them by their proper name – “activity logging trackers

      When DHT was introduced, the ‘private flag’ (from where the ‘private tracker’ gets its name) was part of the spec, and it disables DHT on that specific torrent when it’s turned on.
      There was one version of one client that didn’t do this, in 2005, it was quickly ostracized (and even then massively overblown)

  • nevermind (aka Ralph Wiggum)

    your a moron.
    you have no idea how trackers really work.
    if you did then you would understand the internet.
    sorry asshole but you require a minimum IQ 0f 130 to grasp the basic concept.
    chances are you wont even post this reply

    just proves my point that assholes like you think you r in control of the world.

    if you cant play … you would take your ball and go home…
    well… just leave the ball and just go…

    • OH I understand EXACTLY how trackers work, what they do, and don’t do. But please, do point out EXACTLY where I’m wrong. Unlike some people, I don’t put a word limit on comments (in fact, the longer and more detail-filled, the better!) Of course, many others disagree with your claims – funnily enough, one of the biggest traffic sources for this piece, are the official utorrent and bittorrent forums, where it’s often referenced by the support staff. Do the support staff at Bittorrent Inc. not know how trackers really work too?

      However, please elucidate.

      • IQ = 170 (aka Ralph Wiggum)

        actually “nevermind” poster is right. you have no idea what your talking about.

        trackers are meant to increase the connections and in turn increase the maximum possible speed so that the {torrent} file can be copied as fast as possible. the faster the speed = the less time the bandwidth required to u/d. if a tracker goes down or is being blocked, the other trackers connected sustain to keep the maximum configured torrent connections solidly connected on to the same server – (eg – udp://tracker.istole.it:80 & udp://tracker1.istole.it:80) the more non-related trackers (other servers) –the more likely the torrent will complete in a few minutes in hours if no tracker.
        [Hint – DHT keeps a very minimal peer connect without trackers.*(obviously you don’t know this – now how could you!)]
        of course – there is such a thing as too MANY trackers as this will slow down the downloader computer rather than the world.
        and hey – aren’t you forgetting the client internal settings to help to sustain all those peer/connecting connections?

        people like you rely on petty AIO clients like uTorrent that come pre-configured with very minimal user defined settings.
        like buying a car. it is all ready to go. pre-configured with a factory stereo and power windows. without these pre-confiurations, you would just do as you are with this . . bitch and complain and hope someone will listen.

        in this day in age the bandwidth speeds are ever increasing (I have access to 150MB Fiber) and it is the trackers that are gradually being outdated because of it the awesome advancements to accelerate the human race to a new level of computer processing.
        why do you think the trackers go offline and return stronger than ever …? UPGRADES ! – that’s why.
        not everyone has an XP like you.

        so you see – your big words and mater-of-fact attitude has diluted your perception of how the world really is… it doesn’t spin because of you.
        you need to stop trying to play teacher and just go with the flow like everyone else.
        you know – learn on your own, like the rest of us, and come to terms that you are not in control of how the world spins.
        if you can’t learn this simple mental exercise on your own – then may we suggest the local yellow pages and search under ‘Psychiatrist’.
        and hey guess what – TYP online – who woulda thunk eh??!!

        but be warned… the internet is about to take an unexpected turn for the better.
        (IPv4 > IPv6 > ?) and (analog cell > smart-devices > ?)
        will trackers still be viable in this new virtual world (?) – who knows
        the whole tracker concept may have to be re-thunk to accommodate the ever-changing and advancing computer accessed internet world

        at least people like you will not be with the rest of us.
        we go with the flow.

        all in all, you have wasted approx 15 min of each persons life reading this crap.
        so why don’t you take your immature – below averaged IQ and go play an on-line game. don’t forget to wear your gaming headset and complain to those who will listen to your nonchalance ramblings…MORON

        Here’s your ball – we don’t need it.
        Now just GO ALREADY !!!

        • Replying to your own comment isn’t really saying much.

          You were doing SO well, right up until “trackers are meant to increase the connections and in turn increase the maximum possible speed so that the {torrent} file can be copied as fast as possible.
          Oh wait, it was the second sentence, and as your first one was just you agreeing with yourself, it’s not saying much either.

          I suggest you go and read the specs, because they don’t “increase the connections” at all. it’s funny, but I’d almost forgotten that ‘explanation’ of trackers, haven’t seen it in must be 7-8 years, but it’s good to see the misinformation train still going on. Then again, a lot of sites rely on their ‘experts’ sticking to their guns, and swearing blind they know what they’re talking about. That’s true even when it’s a claim like yours, which is based on simplifying the role of a tracker about as much as you can, then focusing on the ‘how many’ part and forgetting the whole ‘what’ part that makes this seem so silly. Does, however, give me a prod that I need to get that visual demonstration of the multi-tracker problem (you should be able to understand where you went wrong [I hope], as it’ll be designed to be understood by an 85 year old judge that hates “the internet”)

          you need to stop trying to play teacher and just go with the flow like everyone else.
          Ah, you mean I need to stop shattering your illusions of knowledge, and ‘go with the flow’, because it makes you look good? Sorry, I have a passionate hatred of bullshit, and take great pleasure in calling out people on it; I’ve done it for many years. The whole Net Neutrality fight in the US is because I wouldn’t ‘just go with the flow’ but investigated Comcast’s use of Sandvine. This led to others verifying my research of Comcast doing it nationwide, which then led to the FCC investigating, which led to the rules Verizon got overturned last year and started the fight. Should have I stopped ‘playing at teacher’ and ‘go with the flow’ there?

          The world changes, improves, advances because people don’t “just go with the flow”. Every paradigm shift is based on that, and yes, it upsets people, people just like you who have been getting along very nicely peddling nonsense to stoke their sense of superiority. The smart ones got over it and learn from it, the honest ones admitted their mistakes – it would seem you are neither. For that I pity you, and hope that this will change before you enter your twenties – the last thing the world needs are more proud ultracrepadarians.

          all in all, you have wasted approx 15 min of each persons life reading this crap.
          Oh, it took me far less than 15 minutes to read. Maybe 2 minutes tops, and another 5 chuckling; but your comment isn’t crap, if only for the [I’m assuming unintentional] humour of it. I’m sure most of the other people who read it will have a similar opinion.

          Thank you, after a depressing few days, there’s nothing quite like a earnest pile of bovine excrement in the form of a poor attempt to flame to raise the spirits.

  • Pingback: Which TPB Clones Are Beached Whales? | Politics & P2P()

  • IQ = 170 (aka Ralph Wiggum)

    ha wrong again.

    and ha…wrong again.

    oh and of course…your wrong again on that ….(again)

    there are actually 13 of us IT professionals here.
    you just met three of us.

    you are not that smart.
    especially being from a low ended country.

    you don’t really have any friends do ya.

    your avatar shows you look like a faggot

    so just take your faggot ball and go away.

    besides – it has been over 7 months since anyone has ever commented on this page.
    that should be enough to kill yourself. (do the world a favour)

    your not that great of an internet guru.
    just another foreigner trying to be big in an even bigger world.

    if your so smart, then you would know where we are from.
    a country that has more than you will ever know.

    DUMMY

    you probably think i am in the states …don’t ya.
    see not smart.

    expect a DDOS
    ASSHOLE

    • Sorry, but when you said

      there are actually 13 of us IT professionals here.
      you just met three of us.

      all I could see was this
      Iq=170 is a Professional

      And I guess I’m supposed to follow it up with a ‘please Mr Skiddie, don’t DDOS me!“. Sorry, you’ll have to get in line, some other kid threatened me with that 3 weeks ago (in fact, right as I started the research for my latest piece), didn’t impress me much then either, but then, DDoS never has, because it’s completely useless unless you’re trying to give the impression of an online toddler tantrum.

      Anyway, Ralphie, have a GREAT day.

  • Experienced Trackerer

    This article is complete bs! I have 200+ trackers on every torrent I download and I get speeds of ~2mb+ (my max internet connection speed is about 2.4mb). When I use only 1-5 trackers then my speeds are constantly ~20kbps

    • Correlation does not equal causation.

      As I said to the other person, once you understand what a tracker actually does, you’ll see why your statement is complete crap. I mean, I did think that this piece (and the previous one) was easy enough to follow, but it seems not. So I’ll have to “web-ify” my ‘how to explain how bittorrent works to an 80yo technophobe’, specifically the bit about trackers and the role they play.

      I’ll give you a chance to redeem yourself, however. This is the bittorrent protocol specification on trackers (or if you prefer the ‘official version‘. Note that the UDP tracker spec is listed separately, as it came after). Go through it and tell me how the number of trackers affects your speed. It’ll be interesting to see.

  • JC

    I think a lot of the robust contrary opinions are held by posters that misunderstand the client / tracker interaction. Operation of a tracker is generally not in the experience of most downloaders and uploaders. So the disconnect and rumor mlll runs rampant since the voices of the knowledgeable is drowned out by your average internet loud mouth.

    • ktetch

      Pretty much

  • azroach

    So, we should remove all our trackers when torrenting? Whether uploading or downloading?

    What the hell is the point of a tracker then?

    • have one there is fine. More than one, doesn’t really improve things.

      The point of trackers was to coordinate swarms. They were part of the original 2001 spec. In 2005 DHT came along, and superseeded (pun intended) things. Now the only thing they have over DHT is that they can be used as an authentication server for a DRM based system. Otherwise, they’re less reliable, and can be used to play around with swarms and impose outside control. The point is, they were what we had when bittorrent started.

  • American GunNut

    Sometimes you can’t find a seed with only a few trackers. Sometimes the only way to find even a single seed is to dump a list of 40 trackers into the torrent.

  • Adrian Mazur

    May I ask how this articles apples to bittorrent clients with “always announce to all trackers” enabled?

    • ktetch

      Ok, I can understand your confusion somewhat.

      by ‘all trackers’, it means all listed trackers. It doesn’t mean ‘ever tracker that exists today’.

      Above I showed a listing that had 7 trackers listed, 1-5 were regular, plus DHT and PEX. What I didn’t include is that there might be a ‘3a’ tracker and a 3b tracker instead of just a single one there. Ordinarily, the client would announce to tracker 3a and not to 3b, unless 3a errors, when it would then go to 3b. It’s how the multiple-tracker protocol was designed to work. However, some [more popular] clients decided that instead of ‘fallback’ trackers, the default should be ‘alternative’ trackers, and that’s what the mess we have today is caused by.

      And again, at the same time as all this was starting out, then DHT appeared, and things started to use that as the fallback, and people forgot about the whole multitracker issue (because if the trackers go down due to bad behaviour, there’s STILL going to be DHT….

      tl;dr
      that option tells the client to treat fallback trackers as alternative trackers, is making a bad situation worse.

  • Fried

    The key thing is, stop adding trackers. Sites and groups like TPB need to start reducing tracker numbers as well – and TPB is a big problem, adding 5 trackers to every torrent. I really like your idea..

    Fried