Last post, we wrote a parser for bencoded torrent files. Now we need to connect to a tracker so we can arrange a connection to other peers.

Before we jump into that though, let's review the general architecture of how to download a file through bit torrent.

If you already understand the general overview of BitTorrent or you don't care, skip right to the bit where we start to talk to trackers.

General Steps to Download via BitTorrent

To download a file through BitTorrent, you need to start with a .torrent file or a magnet link. I'm assuming that a magnet link is just a URL that allows you to download the .torrent file on the fly, but I don't know this for sure. If it turns out to be more complicated than that, we'll find out when we get around to implementing as that feature.

In any case, once you have the .torrent file, you decode the bencoding into something you can work with in your program (a dictionary in our case). The "announce" key lists the server that keeps track of the torrent stats, and the list of peers that are keeping the torrent alive. There is also an extension to the BitTorrent protocol which allows a torrent file to list backup trackers which can be used in case the primary tracker isn't available. Our code is going to assume the primary server is online for now. Retrying the backup servers will come later.

The meta spec talks about trackers as if they are an HTTP service, but you will notice that our parsed sintel.torrent lists the tracker as udp://tracker.leechers-paradise.org:6969. This is because there is an extension to the BitTorrent protocol to add support for UDP trackers. I'm not sure how similar the UDP trackers are to the HTTP ones, but for this post I'm going to focus on the UDP protocol because that's what's in the torrent file. If HTTP trackers is something that's important to you, let me know and I'll consider expanding the post.

To join the peer-to-peer network, you need to perform a "hello handshake" with the tracker and then announce your existence. The tracker will respond to the annouce request with some info about the torrecnt and a list of peers that you can connect with to download the file you want. Your client should also report uploaded/downloaded/left stats. I think that this bandwidth reporting happens over time too, since it would make sense to me that the tracker prioritizes well behaved peers. I'm also not certain what verification methods are in place to tell if you're actually telling the truth about your upload/download.

When you have a list of peers that have the file you want, you connect to them on the port they advertised and request a "piece" of the file that you want. "info.pieces" is an entry in the torrent file, but it's currently encoded as a node.js Buffer because I don't yet know how to decode the data that it's holding (we'll get to this when we need to start connecting to peers).

Once you acquire all the pieces, you can assemble together into the file you wanted in the first place. At this point you will have successfully downloaded a file from BitTorrent. Now, we're not even close to there yet. We still need to make our client speak to trackers and peers. And talking to trackers in the point of this post, so let's get to it.

Talking to Trackers

As I mentioned earlier, we're going to be focusing on the UDP tracker protocol since that's what's listed in my torrent file. In order to implement this feature, I'll be consulting the UDP tracker spec extension.

To start talking to a tracker, we need to open up a UDP socket to the tracker address. To do this in node.js, we need to use the UDP/Datagram module, and the messages are going to be in node.js Buffer format, so we'll need to use the Buffer module as well.

const dgram = require('dgram')
// Buffer is available globally by default, so we don't need this line
// const Buffer = require('buffer').Buffer

Then we need to read the tracker out of our parsed torrent file and parse the host/port out of it using the url module from the node.js standard library.

const url = require('url')
let tracker = url.parse(torrent_data.announce)
let tracker_host = tracker.hostname
let tracker_port = tracker.port

And then open a UDP socket to the tracker.

let socket = dgram.createSocket('udp4')

Before we start announcing ourselves, we need to do a connection handshake with the tracker. If we were using TCP, the server would be able to trust the source IP address we send in our packets. All TCP connections start with a handshake to verify that packets can be delivered in both directions.

UDP has no such handshake by default, so we need to prove our IP to the tracker. To start the handshake, we first send a connection request packet to the tracker. The spec lists this as the correct format for a connect request:

Offset   Size            Name            Value
0       64-bit integer  protocol_id     0x41727101980 // magic constant
8       32-bit integer  action          0 // connect
12      32-bit integer  transaction_id
16

Once we send a connect request, the server will send us a packet back with a connection id that we can use in our future announce and scrape requests. If we weren't lying about our IP then we'll receive the response.

Weird binary tricks

This section is info about how to pack binary data into packets in Javascript. It's not strictly necessary that you understand the details here, so skip to the next section if you don't care.

In Javascript, you cannot store a 64 number in a single Number type variable if you want to use bitwise operators on it. The most you can safely store is 32 bits. You'll notice that the magic "protocol_id" at the beginning of the connection message is a 64 bit integer, so we're going to need to use 2  32-bit numbers to store the whole number correctly.

Disclaimer: This is the part where I had to check someone else's implementation to verify how to correctly set bits in our connect message (thanks @superafroman!). I ended up learning a lot about binary encoding here.

In order to store 0x41727101980 in 2 32-bit numbers, we can just chop off the last 32 bits (0x27101980, 4 bits per hex digit) and store that in one number, and take the remaining 0x417 in another number. Then we can write 0x417 to our connection message in the first 32 bits, and 0x27101980 into the next 32 bits. This results in the same value as if we had (somehow) written a 64-bit number in one go.

This number chopping works because it's encoded in hexidecimal. 1 digit neatly contains a group of 4 bits, so adding or removing digits never changes the binary encoding of the other digits. 0xf is 1111, 0xf0 is 11110000, and 0xf8 is 11111000. You'll notice that the 1111 at the beginning, which corresponds to the first f, never changes. The same is not true of base 10 decimal. You cannot take 123456 and break it apart into 123 and 456 and expect to get the same number if you concatenate their binary representations.

Constructing the connect message

So let's actually put this connection request together. First we need a message buffer that is 16 bytes (as shown in the message diagram above).

let connect_message = Buffer.alloc(16)

Then we need to set the "protocol_id" on the connect message buffer. Accessing an array index on a buffer will allow you to read/write 1 byte (8 bits). So in order to write our first 32 bit number into the buffer, we need to write 4 8-bit chunks.

connect_message[0] = 0x417 >> 24
connect_message[1] = (0x417 >> 16) & 0xff
connect_message[2] = (0x417 >> 8) & 0xff
connect_message[3] = 0x417 & 0xff

0x417 is short for the full 32 bits of 0x00000417 in this code. On the first line, we drop the right-most 24 bits of the number, which leaves us with just the 8 bits that will fit in this index of the buffer (the first 0x00 in this case).
Next we drop the rightmost 16 bits, which leaves us with 16 bits. In order to drop the higher bits that we already wrote into index 0, we use the bitwise AND operator to keep the rightmost 8 bits and discard the rest (so we're writing 0x00 here too).
Then we repeat the same trick, but only discarding the rightmost 8 bits (so we write 0x04 here).
And in the last write (index 3) we don't bit shift at all since it's the rightmost 8 bits that we want to write while discarding the 24 leftmost bits (0x17 here).

Once we've written 0x417 to the buffer, we do the same thing again for 0x27101980.

connect_message[4] = 0x27101980 >> 24
connect_message[5] = (0x27101980 >> 16) & 0xff
connect_message[6] = (0x27101980 >> 8) & 0xff
connect_message[7] = 0x27101980 & 0xff

We write 0x27 into index 4, 0x10 into 5, 0x19 into 6, and 0x80 into 7.

And now we have the 64 bit protocol ID written into the first 16 bytes of the connect message.

This code is a bit verbose, so let's simplify it. A function that writes a 32 bit int into a 4 byte buffer would simplify the code.

function int_buffer(int) {
	let b = Buffer.alloc(4)
	b[0] = int >> 24
	b[1] = (int >> 16) & 0xff
	b[2] = (int >> 8) & 0xff
	b[3] = int & 0xff
	return b
}

Next, we need to write a 32-bit "action" field into the message. Since this is a 0, we can just leave it blank, which means that we won't touch bytes 8 - 11 in connect_message.

The last thing we need for the connect message is a transaction ID. This just needs to be a random 32-bit value which the server will send back to us with our connection_id.

let transaction_id = Math.round(Math.random()*0xffffffff)

So at last we can construct our connect message.

let connect_message = Buffer.concat([
	int_buffer(0x417), // first half of protocol id
	int_buffer(0x27101980), // second half of protocol id
	int_buffer(0), // action
	int_buffer(transaction_id),
])

Sending a message to the tracker

And now we can send our connect message through the socket we opened earlier.

function on_send(err) {
	if (err) {
		console.error('Failed to send connect packet: '+err)
	} else {
		console.log('Sent: '+connect_message.toString('hex'))
	}
}
socket.send(connect_message, tracker_host, tracker_port, on_send)

And we need to listen for a response from the server.

socket.on('message', function(response) {
	console.log('Incoming: '+response.toString('hex'))
})

When I ran this, I got the following:

Sent: 0000041727101980000000002790757a
Incoming: 000000002790757a49b2db7d0d692218

The spec states that a connect response will be in the following format:

Offset   Size            Name            Value
0       32-bit integer  action          0 // connect
4       32-bit integer  transaction_id
8       64-bit integer  connection_id
16

Filled with the data I got in the response:

Offset   Size            Name            Value
0       32-bit integer  action          00 00 00 00 // connect
4       32-bit integer  transaction_id  27 90 75 7a // same as we sent
8       64-bit integer  connection_id   49 b2 db 7d // our new id
16                                      0d 69 22 18 // our new id

So let's parse our connect response into a dictionary.

let transaction_id = response.slice(4, 8)
let connection_id = response.slice(8, 16)

And there we have it! Our connection id is 0x49b2db7d0d692218. We can now freely talk to the tracker.

Announce message

Next, we need to announce to the tracker that we want a file. The UDP spec lists this as the format for an announce request.

Offset   Size    Name     Value
0       64-bit  integer  connection_id
8       32-bit  integer  action          1 // announce
12      32-bit  integer  transaction_id
16      20-byte string   info_hash
36      20-byte string   peer_id
56      64-bit  integer  downloaded
64      64-bit  integer  left
72      64-bit  integer  uploaded
80      32-bit  integer  event           0
84      32-bit  integer  IP address      0 // default
88      32-bit  integer  key
92      32-bit  integer  num_want        -1 // default
96      16-bit  integer  port
98

Some values that we need to fill here are connection_id, transaction_id (both of which we already have), info_hash, peer_id, downloaded, left (how much is left in the file we want), uploaded, event, IP address, key, num_want, and port.

The bittorrent spec says that the info_hash should be the 20 byte sha1 hash of the bencoded "info" field in the torrent file.

const crypto = require('crypto')
let info_hash = crypto.createHash('sha1')
info_hash.update(bencode(torrent_data.info))
info_hash = info_hash.digest()

And the peer ID is a random string of 20 characters.

let peer_id = crypot.randomFillSync(Buffer.alloc(20))

Downloaded, left, and uploaded should all be a 64 bit number. Node.js has BigInt available since version 10.4.0, so we'll use this for all three. Left is the sum of all the files in the torrent since we haven't downloaded anything yet.

let downloaded = BigInt(0)
let uploaded = BigInt(0)
let left = BigInt(0)
for (let file of torrent_data.info.files) {
	left += BigInt(file.length)
}

And we'll also need a new helper method to convert bigints into buffers. I'm not going to explain too much how this works, but it's basically the same as the int_buffer() version except in a for loop.

function bigint_buffer(value) {
	let b = Buffer.alloc(64)
	for (let i = 0; i < 8; i++) {
		b[i] = value >> (BigInt(64 - (i*8+1))) & BigInt(0xff)
	}
}

Event is either 0 for none, 1 for completed, 2 for started, and 3 for stopped. Currently we haven't even connected to peers yet, so we'll leave it as 0.

let event = 0

We're not going to set IP address since the server already knows who we are based on the initial handshake.

let ip_address = 0

The UDP spec is a bit vague about the purpose of the key field. It does link to a similar spec page which clarifies slightly by stating that key is "A unique key that is randomized by the client.". I'm assuming that this key should be regenerated for each announce request, but I don't know that for sure.

let key = crypto.randomFillSync(Buffer.alloc(32))

The original UDP spec is similarly quiet about "num_want", but the linked spec clarifies this as the max number of peers that we want in our reply. We'll set this to -1 to let the tracker decide.

let num_want = -1

And finally the port field is the port that we're listening on so other peers can talk to us. Since we haven't implemented this functionality, we'll fib a little bit and say 6881. Other peers may try to connect to us, but since we're not listening they'll just move on and find someone who does. This behaviour will probably cause trackers to dislike you, and is very bad P2P citizenship, so do not run your program like this outside of testing.

let port = 6881

Now that we have all the fields sorted out, we have enough info to create our announcement packet:

let announce_packet = Buffer.concat([
	connect_response.connection_id,
	int_buffer(0x1), // 'announce' action
	connect_response.transaction_id,
	info_hash,
	peer_id,
	bigint_buffer(downloaded),
	bigint_buffer(left),
	int_buffer(uploaded),
	int_buffer(event), // event
	int_buffer(ip_address), // IP address
	key,
	int_buffer(num_want),
	int_buffer(port),
])

And then we can send it after we get the connection response from the tracker. We'll only send the announce packet if the message we just got from the server was a "connect" response.

socket.on('message', function(response) {
	// if the response action is `connect` (type 0)
	if (response.readInt32BE(0)) {
		let transaction_id = response.slice(4, 8)
		let connection_id = response.slice(8, 16)
		let info_hash = crypto.createHash('sha1')
		info_hash.update(bencode(torrent_data.info))
		info_hash = info_hash.digest()
		let peer_id = crypot.randomFillSync(Buffer.alloc(20))
		let downloaded = BigInt(0)
		let uploaded = BigInt(0)
		let left = BigInt(0)
		for (let file of torrent_data.info.files) {
			left += BigInt(file.length)
		}
		let event = 0
		let ip_address = 0
		let key = crypto.randomFillSync(Buffer.alloc(32))
		let num_want = -1
		let port = 6881
		let announce_packet = Buffer.concat([
			connection_id,
			int_buffer(0x1), // 'announce' action
			transaction_id,
			info_hash,
			peer_id,
			bigint_buffer(downloaded),
			bigint_buffer(left),
			int_buffer(uploaded),
			int_buffer(event), // event
			int_buffer(ip_address), // IP address
			key,
			int_buffer(num_want),
			int_buffer(port),
		])
		socket.send(announce_packet, tracker_host, tracker_port, on_send)
	}
})

And now we see... nothing for now :) This post is a work in progress and will be finished at a later date.

The code for this post is available for reference. If you have any questions or you find a problem in the post, let me know.

Post index