Last post, we wrote a parser for bencoded torrent files. Now we need to connect to a tracker so we find out which peers can send us the file.
Before we jump into that though, let's review the general architecture of how to download a file through bit torrent.
If you already understand the general overview of BitTorrent or you don't care, skip right to the bit where we start to talk to trackers.
General Steps to Download via BitTorrent
To download a file through BitTorrent, you need to start with a .torrent file or a magnet link. For now we'll use a .torrent file directly. We'll add magnet link support as a separate feature later.
In any case, once you have the .torrent file, you decode the bencoding into something you can work with in your program (a dictionary in our case). The "announce" key lists the server that keeps track of the torrent stats, and the list of peers that are keeping the torrent alive. There is also an extension to the BitTorrent protocol which allows a torrent file to list backup trackers which can be used in case the primary tracker isn't available. Our code is going to assume the primary server is online for now. Retrying the backup servers will come later.
The meta spec talks about trackers as if they are an HTTP service, but you will notice that our parsed sintel.torrent lists the tracker as udp://tracker.leechers-paradise.org:6969. This is because there is an extension to the BitTorrent protocol to add support for UDP trackers. I'm not sure how similar the UDP trackers are to the HTTP ones, but for this post I'm going to focus on the UDP protocol because that's what's in the torrent file. If HTTP trackers is something that's important to you, let me know and I'll consider expanding this post. Otherwise this will be a later feature.
To join the peer-to-peer network, you need to perform a "hello handshake" with the tracker and then announce that you exist. The tracker will respond to the announce request with some info about the torrent and a list of peers that you can connect with to download the file you want. Your client should also report uploaded/downloaded/left stats. I think that this bandwidth reporting happens over time too, since it would make sense to me that the tracker prioritizes well behaved peers. I'm also not certain what verification methods are in place to tell if you're actually telling the truth about your upload/download.
When you have a list of peers that have the file you want, you connect to them on the port they advertised and request a "piece" of the file that you want. The torrent file contains a list of all the necessary pieces that are required to make up the full file. Once you acquire all the pieces, you can assemble into the final file. At this point you will have successfully downloaded a file from BitTorrent.
Before we get that far, we still need to make our client speak to trackers and peers. And talking to trackers in the point of this post, so let's get to it.
Talking to Trackers
As I mentioned earlier, we're going to be focusing on the UDP tracker protocol since that's what's listed in my torrent file. In order to implement this feature, I'll be consulting the UDP tracker spec extension.
To start talking to a tracker, we need to open up a UDP socket to the tracker address. To do this in node.js, we need to use the UDP/Datagram module, and the messages are going to be in node.js Buffer format, so we'll need to use the Buffer module as well.
const dgram = require('dgram')
// Buffer is available globally by default, so we don't need this line
// const Buffer = require('buffer').BufferThen we need to read the tracker out of our parsed torrent file and parse the host/port out of it using the url module from the node.js standard library.
const url = require('url')
let tracker = url.parse(torrent_data.announce)
let tracker_host = tracker.hostname
let tracker_port = tracker.portAnd then open a UDP socket to the tracker.
let socket = dgram.createSocket('udp4')Before we start announcing ourselves, we need to do a connection handshake with the tracker. This is necessary because we're using UDP instead of TCP. TCP has built-in verification that both IPs in the connection are correct. UDP doesn't have a similar verification system, so the UDP tracker protocol handles this at the application level.
To start the handshake, we first send a connection request packet to the tracker. The spec lists this as the correct format for a connect request:
Offset Size Name Value
0 64-bit integer protocol_id 0x41727101980 // tracker protocol ID
8 32-bit integer action 0 // connect
12 32-bit integer transaction_id
16Once we send a connect request, the server will send us a packet back with a connection id that we can use in our future announce and scrape requests. If we weren't lying about our IP then we'll receive the response.
Weird binary tricks
This section is info about how to pack binary data into packets in Javascript. It's not necessary that you understand the details here, so only read this section if you find it interesting.
In Javascript, you cannot store a 64 number in a single Number type variable if you want to use bitwise operators on it. The most you can safely store is 32 bits. You'll notice that the constant "protocol_id" at the beginning of the connection message is a 64 bit integer, so we're going to need to use 2 32-bit numbers to store the whole number correctly.
Disclaimer: This is the part where I had to check someone else's implementation to verify how to correctly set bits in our connect message (thanks @superafroman!). I ended up learning a lot about binary encoding here.
In order to store 0x41727101980 in 2 32-bit numbers, we can just chop off the last 32 bits (0x27101980, 4 bits per hex digit) and store that in one number, and take the remaining 0x417 in another number. Then we can write 0x417 to our connection message in the first 32 bits, and 0x27101980 into the next 32 bits. This results in the same value as if we had written a 64-bit number in one go.
This number chopping works because it's encoded in hexidecimal. 1 digit neatly contains a group of 4 bits, so adding or removing digits never changes the binary encoding of the surrounding digits. 0xf is 1111, 0xf0 is 11110000, and 0xf8 is 11111000. You'll notice that the 1111 at the beginning, which corresponds to the first f, never changes. The same is not true of base 10 decimal. You cannot take 123456 and break it apart into 123 and 456 and expect to get the same number if you concatenate their binary representations.
Constructing the connect message
So let's actually put this connection request together. First we need a message buffer that is 16 bytes.
let connect_message = Buffer.alloc(16)
Then we need to set the "protocol_id" on the connect message buffer. Accessing an array index on a nodejs buffer will allow you to read/write 1 byte (8 bits). So in order to write our first 32 bit number into the buffer, we need to write 4 8-bit chunks.
connect_message[0] = 0x417 >> 24
connect_message[1] = (0x417 >> 16) & 0xff
connect_message[2] = (0x417 >> 8) & 0xff
connect_message[3] = 0x417 & 0xff0x417 is short for the full 32 bits of 0x00000417 in this code. On the first line, we drop the right-most 24 bits of the number, which leaves us with just the 8 bits that will fit in this index of the buffer (the first 0x00 in this case).
Next we drop the rightmost 16 bits, which leaves us with 16 bits. In order to drop the higher bits that we already wrote into index 0, we use the bitwise AND operator to keep the rightmost 8 bits and discard the rest (so we're writing 0x00 here too).
Then we repeat the same trick, but only discarding the rightmost 8 bits (so we write 0x04 here).
And in the last write (index 3) we don't bit shift at all since it's the rightmost 8 bits that we want to write while discarding the 24 leftmost bits (0x17 here).
Once we've written 0x417 to the buffer, we do the same thing again for 0x27101980.
connect_message[4] = 0x27101980 >> 24
connect_message[5] = (0x27101980 >> 16) & 0xff
connect_message[6] = (0x27101980 >> 8) & 0xff
connect_message[7] = 0x27101980 & 0xffWe write 0x27 into index 4, 0x10 into 5, 0x19 into 6, and 0x80 into 7.
And now we have the 64 bit protocol ID written into the first 16 bytes of the connect message.
This code is a bit verbose, so let's simplify it. A function that writes a 32 bit int into a 4 byte buffer would simplify the code.
function int_buffer(int) {
let b = Buffer.alloc(4)
b[0] = int >> 24
b[1] = (int >> 16) & 0xff
b[2] = (int >> 8) & 0xff
b[3] = int & 0xff
return b
}Next, we need to write a 32-bit "action" field into the message. Since this is a 0, we can just leave it blank, which means that we won't touch bytes 8 - 11 in connect_message.
The last thing we need for the connect message is a transaction ID. This just needs to be a random 32-bit value which the server will send back to us with our connection_id.
let transaction_id = Math.round(Math.random()*0xffffffff)
// in this post this value is 0x2790757aSo at last we can construct our connect message.
let connect_message = Buffer.concat([
int_buffer(0x417), // first half of protocol id
int_buffer(0x27101980), // second half of protocol id
int_buffer(0), // action
int_buffer(transaction_id),
])Sending a message to the tracker
And now we can send our connect message through the socket we opened earlier.
function on_send(err) {
if (err) {
fatal(err)
}
console.log('Connect sent: '+connect_message.toString('hex'))
}
socket.send(connect_message, tracker_host, tracker_port, on_send)
function fatal(err) {
console.error(err)
process.exit(1)
}And we need to listen for a response from the server.
socket.on('message', function(response) {
console.log('Incoming: '+response.toString('hex'))
})When I ran this, I got the following:
Connect sent: 0000041727101980000000002790757a
Incoming: 000000002790757a49b2db7d0d692218
The spec states that a connect response will be in the following format:
Offset Size Name Value
0 32-bit integer action 0 // connect
4 32-bit integer transaction_id
8 64-bit integer connection_id
16Filled with the data I got in the response:
Offset Size Name Value
0 32-bit integer action 00 00 00 00 // connect
4 32-bit integer transaction_id 27 90 75 7a // same as we sent
8 64-bit integer connection_id 49 b2 db 7d // our new id
16 0d 69 22 18 // our new idSo let's parse our connect response into a dictionary.
let transaction_id = response.slice(4, 8)
let connection_id = response.slice(8, 16)And there we have it! Our connection id is 0x49b2db7d0d692218. We can now freely talk to the tracker.
Announce message
Next, we need to announce to the tracker that we want a file. The UDP spec lists this as the format for an announce request.
Offset Size Name Value
0 64-bit integer connection_id
8 32-bit integer action 1 // announce
12 32-bit integer transaction_id
16 20-byte string info_hash
36 20-byte string peer_id
56 64-bit integer downloaded
64 64-bit integer left
72 64-bit integer uploaded
80 32-bit integer event 0
84 32-bit integer IP address 0 // default
88 32-bit integer key
92 32-bit integer num_want -1 // default
96 16-bit integer port
98Some values that we need to fill here are connection_id, transaction_id (both of which we already have), info_hash, peer_id, downloaded, left (how much is left in the file we want), uploaded, event, IP address, key, num_want, and port.
The bittorrent spec says that the info_hash should be the 20 byte sha1 hash of the bencoded "info" field in the torrent file.
const crypto = require('crypto')
let info_hash = crypto.createHash('sha1')
info_hash.update(bencode(torrent_data.info))
info_hash = info_hash.digest()And the peer ID is a random string of 20 characters.
let peer_id = crypto.randomFillSync(Buffer.alloc(20))Downloaded, left, and uploaded should all be a 64 bit number. Node.js has BigInt available since version 10.4.0, so we'll use this for all three. Left is the sum of all the files in the torrent since we haven't downloaded anything yet.
let downloaded = BigInt(0)
let uploaded = BigInt(0)
let left = BigInt(0)
for (let file of torrent_data.info.files) {
left += BigInt(file.length)
}And we'll also need a new helper method to convert bigints into buffers. I'm not going to explain too much how this works, but it's basically the same as the int_buffer() version except in a for loop.
function bigint_buffer(value) {
let b = Buffer.alloc(64)
for (let i = 0; i < 8; i++) {
b[i] = value >> (BigInt(64 - (i*8+1))) & BigInt(0xff)
}
}Event is either 0 for none, 1 for completed, 2 for started, and 3 for stopped. Currently we haven't even connected to peers yet, so we'll leave it as 0.
let event = 0We're not going to set IP address since the server already knows who we are based on the initial handshake.
let ip_address = 0The UDP spec is a bit vague about the purpose of the key field. It does link to a similar spec page which clarifies slightly by stating that key is "A unique key that is randomized by the client.". I'm assuming that this key should be regenerated for each announce request, but I don't know that for sure.
let key = crypto.randomFillSync(Buffer.alloc(32))The original UDP spec is similarly quiet about "num_want", but the linked spec clarifies this as the max number of peers that we want in our reply. We'll set this to -1 to let the tracker decide.
let num_want = -1And finally the port field is the port that we're listening on so other peers can talk to us. Since we haven't implemented this functionality, we'll fib a little bit and say 6881. Other peers may try to connect to us, but since we're not listening they'll just move on and find someone who does. This behaviour will probably cause trackers to dislike you, and is very bad P2P citizenship, so do not run your program like this outside of testing.
let port = 6881Now that we have all the fields sorted out, we have enough info to create our announcement packet:
let announce_packet = Buffer.concat([
connect_response.connection_id,
int_buffer(0x1), // 'announce' action
connect_response.transaction_id,
info_hash,
peer_id,
bigint_buffer(downloaded),
bigint_buffer(left),
bigint_buffer(uploaded),
int_buffer(event), // event
int_buffer(ip_address), // IP address
key,
int_buffer(num_want),
int_buffer(port),
])And then we can send it after we get the connection response from the tracker. We'll only send the announce packet if the message we just got from the server was a "connect" response.
socket.on('message', function(message) {
console.log('Incoming: '+message.toString('hex'))
if (message.readInt32BE(0)==0x0) {
console.log('Processing connect message')
let transaction_id = message.slice(4, 8)
let connection_id = message.slice(8, 16)
let info_hash = crypto.createHash('sha1')
info_hash.update(bencoding.encode(torrent_data.info))
info_hash = info_hash.digest()
let peer_id = crypto.randomFillSync(Buffer.alloc(20))
let downloaded = BigInt(0)
let uploaded = BigInt(0)
let left = BigInt(0)
for (let file of torrent_data.info.files) {
left += BigInt(file.length)
}
let event = 0
let ip_address = 0
let key = crypto.randomFillSync(Buffer.alloc(32))
let num_want = -1
let port = 6881
let announce_packet = Buffer.concat([
connection_id,
int_buffer(0x1), // 'announce' action
transaction_id,
info_hash,
peer_id,
bigint_buffer(downloaded),
bigint_buffer(left),
bigint_buffer(uploaded),
int_buffer(event), // event
int_buffer(ip_address), // IP address
key,
int_buffer(num_want),
int_buffer(port),
])
socket.send(announce_packet, tracker.port, tracker.hostname, function(err) {
if (err) {
fatal(err)
}
console.log('Announce sent: '+announce_packet.toString('hex'))
})
} else {
console.log("I don't know what to do with this message, exiting")
process.exit(1)
}
})And now we see...
Announce sent: 832946ad4fef0b2d00000001b36ab31d79409b5674dc97a6257eb5fd8ca7a1afbf1df1139e40160c7134eb794241b7867a24ff8488ed21e2000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007b4ff770000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000572b39b6695d7133a7a1fb8ae07e0b7d368193c1edfd0ab58b89c7185cbfca5bffffffff00001ae1
Incoming: 00000001b36ab31d000006ee0000000000000001
I don't know what to do with this message, exiting
The code for this post is available for reference. If you have any questions or you find a problem in the post, let me know.
