Support BitTorrent v2 spec: BEP 52

[ewhal]

arvidn/libtorrent#2197
atomashpolskiy/bt#28

[anacrolix]

[christian-roggia]

Here you will find a good reddit TL;DR about the changes and improvements of The BitTorrent Protocol Specification v2:
https://www.reddit.com/r/programming/comments/6safyl/the_bittorrent_protocol_specification_v2/

For more detailed information read the bittorrent.org BEP:
http://bittorrent.org/beps/bep_0052.html

The most important improvements are:

• Introduction of SHA256 and globally unique hashes that will prevent hash collisions and therefore all known hash collision attacks
• Merkle tree for hashing pieces instead of chunk hashes

Not really sure this is currently really needed, but it is definitely something soon or later should be fully implemented (with the help of the community).

[anacrolix]

A useful link from a related issue: https://blog.libtorrent.org/2020/09/bittorrent-v2/.

[ghost]

@anacrolix when will you support this?

[anacrolix]

Not until I see some uptake in the wild. Currently it appears to be zero.

[KyleSanderson]

Any update on this one? This is absolutely "in the wild".

[anacrolix]

I am considering this, there's been an uptick in the community, I think usage might start to appear.

Is there any interest in funding this support?

[KyleSanderson]

I am considering this, there's been an uptick in the community, I think usage might start to appear.

Is there any interest in funding this support?

Specifically I have a PR into a project that already uses this library. autobrr/autobrr#491

This breaks a handful of sites by parsing the contents as the clients well support V2, but the library does not. As V2 has become widely accepted and mainstream over the last year, this is only going to become more prevalent as time progresses.

So, similar situation to you, as a user there's pain. I have an old parser for v1 I can dust off, but then the maintenance burden shifts.

[kovalensky]

When? Tell me honey when? I'm just to hungry for this, give me, give me that juicy BitTorrent v2 support, make it faster, deeper in implementation details, make it swim inside my web server, take all of my resources, make it bigger.

But seriously though, 6 years have passed, everyone had put their heads in a hole in the ground and saying that they need to see other ones to try it first, but the other ones think say same and as a result users are losing a great feature.

[fiatjaf]

How about not doing breaking changes in open protocols next? They should just delete BEP 52 and forget it has ever existed.

[KyleSanderson]

How about not doing breaking changes in open protocols next? They should just delete BEP 52 and forget it has ever existed.

That's not how the world works.

Transmission v4 just went GA with BT2 support, which means that every major client has full support for this now.

[fiatjaf]

Thank you for teaching me how the world works with your 6-year protocol update. Since you know how the world works, tell me for many years will the BT1 protocol still be supported alongside with BT2 because some people and apps won't switch.

[anacrolix]

I've not said I won't do it, but time is money and I have a lot to do. BEP 52 messes with the most fundamental part of BitTorrent so a transition will always be difficult. BEP 52 isn't really worthwhile for the most traditional use case for BitTorrent, downloading everything up front. The story is very different for indexers, search, and ephemeral use of BitTorrent for example in hosting websites, streaming etc. Again, I can accelerate this with some support, and/or can always take well considered PRs.

[kovalensky]

How about not doing breaking changes in open protocols next? They should just delete BEP 52 and forget it has ever existed.

That's not how the world works.

Transmission v4 just went GA with BT2 support, which means that every major client has full support for this now.

This is great news, but actually current main task is not the implementation of protocol itself, but more like implementation of its promised features as cross swarm seeding with clients having hash db in memory. So if clients typically searching dht could ask other clients if they have files with This root hash, and if requested clients did in their hash db, they would seed this file, reviving dead torrents, increasing connectivity and decentralization.
Other feature was deduplication of files like swarm merging by reducing their size on disk storing relativities in hash db.

It's just as most open-source projects there are more great features than man power to do this.

[balupton]

Could those with the ability to implement this propose a budget that they would need? Perhaps it can be crowd-funded.

[anacrolix]

It would take me about 2-3 weeks but could extend a bit more. It would likely spill over into anacrolix/dht a bit, require some special flags for bencoding, probably some refactoring of storage. There might also be some refactoring of peer connections, to handle the different hash/info requests. There would need to be a fair few tests. Maybe $3k?

[kovalensky]

Could those with the ability to implement this propose a budget that they would need? Perhaps it can be crowd-funded.

I don't mean to offend @anacrolix, but the current situation is that we need this to be implemented in something popular to get others' attention to do it.
I wrote to the libtorrent maintainer, his library is used in the second most popular open source qBittorrent, Deluge, and he has a lot of BEPs co-written, he said he would help PR, but currently his limiting factor is time, and he is rewriting caching mechanisms, so even bounty is not the case therefore, I am looking for a C++, Boost developer, to discuss a further task and start crowdfunding, I'll keep this topic updated in case.

[anacrolix]

Maybe I've missed something, but this GitHub issue tracks BitTorrent v2 support in anacrolix/torrent, which is written in Go. Completion of this issue will be when anacrolix/torrent implements BEP 52.

[izissise]

Hello, is there any news on this?

[anacrolix]

This is blocked on funding currently. The details are in an earlier comment.

[anacrolix]

I'm pretty keen to do this but I can't justify working on it.

[kovalensky]

I'm pretty keen to do this but I can't justify working on it.

[weebney]

Would be happy to help implementing this as I've been given a worthwhile amount of grant money to explore BEP 52 in 2024.

Edit: please contact me after the holidays. Happy 2024, everyone—hopefully we can get cracking on this soon.

[anacrolix]

@weebney what's your interest in supporting this development? I can't find any PM details, please use mine if there's a private aspect to your contribution.

I have a sponsor for this feature, I intend to start work on it soon. Additional sponsorship and help is welcome.

I've also updated the IssueHunt details, #138 (Webtorrent support) was previously funded in a similar way.

[weebney]

If you've got this covered, I'm likely to put my effort into other projects that need attention. Let me know if you need any help though—would be happy to contribute here on a personal basis.

[anacrolix]

Development on this has begun at https://github.com/anacrolix/torrent/tree/bittorrent-v2.

Some interesting things I've discovered since working on this:

Other major clients (Transmission 4, qBittorrent 4.4+) claim to have BitTorrent v2 support. It does not work in my testing. Both of those clients can consume hybrid torrents. They do not work with exclusively v2 torrents. Neither support creating hybrid or v2 torrents either (which is weird because qBittorrent claims to support it but I definitely don't see it). I'm using Transmission 4.0.5 and qBittorrent 4.6.3.

There's a lot of confusion about needing "special" support by trackers/DHT etc. My reading is that's all nonsense. You could add support in trackers to automatically combine swarms but it would not be trivial. If the client is aware of multiple swarms it can do this for both DHT and trackers. I have numerous downstream projects that rely on a single infohash and I believe there's a simple way to migrate this forward.

The file piece alignment sounds great, but it is a total nuisance trying to rejig 10+ years of structuring everything around pieces rather than files. It's likely still a net win. There's a lot of concern about it being inefficient for lots of small files. I read that as files smaller than the piece length, but I think actually it's only files that are smaller than the block size, 16 KiB, which is much more palatable.

[kovalensky]

which is weird because qBittorrent claims to support it but I definitely don't see it

BitTorrent v2 added to libtorrent in 2.* branch. When downloading qBittorrent from Fosshub, you have to choose "lt20, qt6" version.
They said they will drop lt12 in 5.0.0 as well as Windows 7 support.

There's a lot of confusion about needing "special" support by trackers/DHT etc.

One wave of confusion started when people didn't find a way to express themselves in term of trackers (public & private & public with rating). For reference, Rutracker doesn't allow hybrids, because of paddings in file list (fix for this is two lines of code btw) and double announces.

Neither it supports v2 only torrents because there should be specific code for this case. They are based on an old version of the TorrentPier engine which is the first engine supporting BitTorrent v2 (stats, file hashes display, etc..), but they will never update due to the custom modifications they made.

In common, discreditization of protocol started when private & public with rating trackers' developers due to lazyness for implementation (can relate though, it took me two weeks of debugging just to re-implement v2 compatible announcer with stats) started to come up with counter arguments and it accelerated with exxagerations from users completely unfamiliar with the protocol.

[anacrolix]

BitTorrent v2 added to libtorrent in 2.* branch. When downloading qBittorrent from Fosshub, you have to choose "lt20, qt6" version. They said they will drop lt12 in 5.0.0 as well as Windows 7 support.

Thanks! Very helpful. I see the lt20 version you mentioned, it's also available in homebrew.

One wave of confusion started when people didn't find a way to express themselves in term of trackers (public & private & public with rating). For reference, Rutracker doesn't allow hybrids, because of paddings in file list (fix for this is two lines of code btw) and double announces.

Yeah it's not been trivial to fix up the assumption that files are packed. It's also harder because v2 hashes unpadded pieces, where as v1 hashes including the padding files. So pieces essentially have different length depending on whether a torrent is v1 or v2.

In common, discreditization of protocol started when private & public with rating trackers' developers due to lazyness for implementation (can relate though, it took me two weeks of debugging just to re-implement v2 compatible announcer with stats) started to come up with counter arguments and it accelerated with exxagerations from users completely unfamiliar with the protocol.

That's handy to know. I have gotten it working, but there are lots of corner cases I expect to need to smooth over. It's complex enough that I'm starting to think Go is not sufficient for this, it would have been much easier to port a client in Rust.

The branch above can now download both hybrid and pure v2 torrents.

go install github.com/anacrolix/torrent/cmd/torrent@bittorrent-v2
wget https://libtorrent.org/bittorrent-v2-test.torrent
torrent download bittorrent-v2-test.torrent

Similarly you should be able to download the hybrid torrent at https://blog.libtorrent.org/2020/09/bittorrent-v2/, or hybrid torrents that are available elsewhere (they're much easier to find than pure v2 torrents).

It's very early days, and there will definitely be crashes and bugs. I also expect that pure v2 magnet links won't work yet.

[anacrolix]

Forgot to push the updated branch https://github.com/anacrolix/torrent/tree/bittorrent-v2. It's now pushed.

[anacrolix]

Support is now in master. Please try it out. It's not complete, but hybrid and pure v2 torrents should now be supported. There's missing support in some tooling. There's no hybrid or v2 torrent creator. Some of the storage backends may do the wrong thing. There's a few shortcuts taken in the protocol for now, they should improve over time.

[weebney]

Thanks @anacrolix
Let me know if you need any support; I'm back to "real" work soon and should still have some time to put towards a more complete implementation

[anacrolix]

If you have a downstream project, update to master and let me know if anything isn't working.

A few good hybrid and pure v2 torrents would be useful as test cases too. I have pinched a few from my DHT indexer which I've updated with v2 support, but there aren't many popular pure v2 torrents in the wild.

It would also be nice to have someone go over and/or test the dual-swarm support: Check that announcing are working on both v1 and v2 simultaneously for trackers and DHT.

I've done zero testing of BitTorrent v2 with WebRTC. There's very little extra needed to link it in if at all. I have no idea what the state of v2 support is in webtorrent.

Not yet supported:

• Replying to hash request
• Handling hash reject
• Asking for proof layers in outbound hash request
• Handling BEP 52's piece reject independently of BEP 6
• Creating hybrid and v2 torrents
• Handling infohash upgrades during handshake

Some cool things I'd like to have:

• Provide a torrent upgrader (take a storage as an input, produce a hybrid or v2 torrent that's as identical as possible)