In this post my aim is to connect to a tracker. (Originally I also wanted to request some peer info, but this post ended up progressing more slowly than I expected.)

Before we get to that though, I think we should first review the architecture of BitTorrent as far as I understand it. I realize that part 3 of a blog series might be a strange place to pause and cover the overall architecture, but I didn't think of it earlier, so here we are.

If you already understand the general overview of BitTorrent or you don't care, skip right to the bit where we start to talk to trackers.

General Steps to Download via BitTorrent

The path to downloading a file through BitTorrent starts with a .torrent file or a magnet link. I'm assuming that a magnet link is just a URL that allows you to download the .torrent file on the fly. However, I'm not certain about this. There might be something more tricky going on. We'll find out when we get around to implementing as that feature.

In any case, once you have the .torrent file, you parse it into a dictionary as we covered last time. The "announce" key lists the server that keeps track of the torrent stats, and the list of peers that are keeping the torrent alive. There is also an extension to the BitTorrent protocol which allows a torrent file to list backup trackers which can be used in case the primary tracker isn't available. Our code will support the new "announce-list" key as well as the old "announce" key.

The meta spec talks about trackers as if they are an HTTP service, but you will notice that our parsed sintel.torrent lists the tracker as udp://tracker.leechers-paradise.org:6969. This is because there is an extension to the BitTorrent protocol to add support for UDP trackers. I'm not sure how similar the UDP trackers are to the HTTP ones, but for this post I'm going to focus on the UDP protocol because that's what's in the torrent file. If HTTP trackers is something that's important to you, let me know and I'll consider expanding the post.

To join the peer-to-peer network, you need to announce your existence to the tracker, and the tracker will respond with a list of peers that you can connect with to download the file you want. Your client should also report uploaded/downloaded stats. I think that this bandwidth reporting happens over time too, since it would make sense to me that the tracker prioritizes well behaved peers. I'm also not certain what verification methods are in place to tell if you're actually telling the truth about your upload/download.

When you have a list of peers that have the file you want, you connect to them on the port they advertised and request a "piece" of the file that you want. "info.pieces" is an entry in the torrent file, but it's currently encoded as a node.js Buffer because I don't yet know how to decode the data that it's holding (we'll get to this when we need to start connecting to peers).

Once you acquire all the pieces, you can assemble together into the file you wanted in the first place. At this point you will have successfully downloaded a file from BitTorrent. Now, we're not even close to there yet. We still need to make our client speak to trackers and peers. And speaking to trackers in the point of this post, so let's get to it.

Speaking to Trackers

As I mentioned earlier, we're going to be focusing on the UDP tracker protocol since that's what's listed in my torrent file. In order to implement this feature, I'll be consulting the UDP tracker spec extension.

To start talking to a tracker, we need to open up a UDP socket to the tracker address. To do this in node.js, we need to use the UDP/Datagram module, and the messages are going to be in node.js Buffer format, so we'll need to use the Buffer module as well.

const dgram = require('dgram')
// Buffer is available globally by default, so we don't need this line
// const Buffer = require('buffer').Buffer

To connect correctly to a UDP tracker, we need to first review what UDP is lacking when compared against a normal HTTP tracker (which operates on TCP underneath).

First, in TCP each party confirms that the IP address of the other party is correct. This is because a TCP connection starts with a handshake: Computer A sends something called a SYN packet to computer B. Then computer B sends a SYN/ACK packet back to computer A to confirm that it can receive messages correctly. Once computer A has received the SYN/ACK, they send an ACK packet back to computer B to confirm that they also can receive messages correctly. Once both parties have received a packet and sent an acknowledgement, they both know that the other computer's IP is correct.

UDP has no such handshake by default, so the tracker would normally just have to trust that the IP we claim to be is actually who we are. But the tracker wants to confirm our IP, so the BitTorrent tracker protocol requires a handshake to be done at the beginning of the UDP connection in order to confirm that we are not lying about our IP. The way the handshake works is that we first send a connection request packet to the tracker. The spec lists this as the correct format for a connect request:

Offset   Size            Name            Value
0       64-bit integer  protocol_id     0x41727101980 // magic constant
8       32-bit integer  action          0 // connect
12      32-bit integer  transaction_id
16

Once we send a correct connect request, the server will send us a message back with a connection id that we can use in our future announce and scrape requests.

Weird binary tricks

This is the part where things get a bit tricky. It's not strictly necessary that you understand the details here, so skip to the next section if you don't care.

In Javascript, you cannot store a 64 number in a single variable if you want to use bitwise operators. In fact, the most you can safely store if you want to use them (and we will need to) is 32 bits. You'll notice that the magic "protocol_id" at the beginning of the connection message is a 64 bit integer, so we're going to need to use 2  32-bit numbers to store the whole number correctly.

Disclaimer: This is the part where I had to check someone else's implementation to verify how to correctly set bits in our connect message (thanks @superafroman!). I ended up learning a lot about binary encoding here.

In order to store 0x41727101980 in 2 32-bit numbers, we can just chop off the last 32 bits (0x27101980, 4 bits per hex digit) and store that in one number, and take the remaining 0x417 in another number. Then we can write 0x417 to our connection message in the first 32 bits, and 0x27101980 into the next 32 bits. This results in the same value as if we had (somehow) written a 64-bit number in one go.

This works because in hex, 1 digit neatly contains a group of 4 bits, so adding or removing digits never changes the binary encoding of the other digits. 0xf is 1111, 0xf0 is 11110000, and 0xf8 is 11111000. You'll notice that the 1111 at the beginning, which corresponds to the first f, never changes. The same is not true of base 10 decimal. You cannot take 123456 and break it apart into 123 and 456 and expect to get the same number if you concatenate their binary representations.

Actually constructing the connect message

So let's actually put this connection request together. First we need a message buffer that is 16 bytes (as shown in the message diagram above).

let connect_message = Buffer.alloc(16)

Then we need to set the "protocol_id" on the connect message buffer. Accessing an array index on a buffer will allow you to read/write 1 byte (8 bits). So in order to write our first 32 bit number into the buffer, we need to write 4 8-bit chunks.

connect_message[0] = 0x417 >> 24
connect_message[1] = (0x417 >> 16) & 0xff
connect_message[2] = (0x417 >> 8) & 0xff
connect_message[3] = 0x417 & 0xff

On the first line, we drop the right-most 24 bits of the number, which leaves us with just the 8 bits that will fit in this index of the buffer.
Next we drop the rightmost 16 bits, which leaves us with 16 bits. In order to drop the higher bits that we already wrote into index 0, we use the bitwise AND operator to keep the rightmost 8 bits and discard the rest.
Then we repeat the same trick, but only discarding the rightmost 8 bits.
And in the last write (index 3) we don't bit shift at all since it's the rightmost 8 bits that we want to write (while discarding the 24 leftmost bits).

Once we've written 0x417 to the buffer, we do the same thing again for 0x27101980.

connect_message[4] = 0x27101980 >> 24
connect_message[5] = (0x27101980 >> 16) & 0xff
connect_message[6] = (0x27101980 >> 8) & 0xff
connect_message[7] = 0x27101980 & 0xff

And now we have the 64 bit "protocol_id" written into the first 16 bytes of the connect message. This code is a bit verbose, so let's simplify it. A function that converts an int into an 32 bit (4 byte) buffer would simplify the code.

function int_buffer(int) {
	let b = Buffer.alloc(4)
	b[0] = int >> 24
	b[1] = (int >> 16) & 0xff
	b[2] = (int >> 8) & 0xff
	b[3] = int & 0xff
	return b
}

Next, we need to write a 32-bit "action" field into the message. Since this is a 0, we can just leave it blank, which means that we won't touch index 8 - 11 in connect_message.

The last thing we need for the connect message is a transaction ID. This just needs to be a random 32-bit value which the server will send back to us with our connection_id.

let transaction_id = Math.round(Math.random()*0xffffffff)

So at last we can construct our connect message, now we need to send it.

let connect_message = Buffer.concat([
	int_buffer(0x417), // first half of protocol id
	int_buffer(0x27101980), // second half of protocol id
	int_buffer(0), // action
	int_buffer(transaction_id),
])

Sending a message to the tracker

First we need to select a tracker to use. I'm going to hard-code a specific server address for now, since the point of this post is to talk to a tracker (selecting a tracker from the torrent file will be in a future post). In order to find a server that was online, I tested the list using telnet <host> <port> until I found a server that would connect with me.

We need to construct a socket to send and receive messages from.

let socket = dgram.createSocket('udp4')
socket.on('message', function(message) {
	console.log('Incoming: '+message)
})
function on_send(err) {
    if (err) {
    	console.error('Message failed to send: '+err)
    } else {
    	console.log('Started: '+transaction_id.toString(16))
    }
}
socket.send(connect_message, tracker_port, tracker_hostname, on_send)

When I ran this code, I got a response from the server that looked like this:

Started: bc2a5662
Incoming: <Buffer 00 00 00 00 bc 2a 56 62 6b 84 e8 5c 23 2c 1d 97>

The spec states that a connect response will be in the following format:

Offset   Size            Name            Value
0       32-bit integer  action          0 // connect
4       32-bit integer  transaction_id
8       64-bit integer  connection_id
16

And if we fill in the values that we got in our response:

Offset   Size            Name            Value
0       32-bit integer  action          00 00 00 00 // connect
4       32-bit integer  transaction_id  bc 2a 56 62 // same as we sent
8       64-bit integer  connection_id   6b 84 e8 5c 23 2c 1d 97 // our id
16

And there we have it! Our connection id is 0x6b84e85c232c1d97. We can now freely talk to the tracker. Next time we'll handle selecting a tracker automatically from our list, retrying sent messages that didn't get a response, and actually announcing ourselves.

The code for this post is available for reference. If you have any questions or you find a problem in the post, let me know.

Post index