The Bitcoin Protocol – 4 – Network Messages 1 – Version

Note: This is fourth in a multipart post explaining various aspects of Bitcoin protocol.

In the previous posts, we first looked at a Bitcoin block and studied a binary .dat file to see how various elements in a Block are laid out. After studying the block, we wrote a basic block parser in python.

But the question remains, how did these blocks get on your computer? Today we seek answers to these questions by looking at the network and messaging layer of Bitcoin network.

We will analyze network traffic generated by Bitcoin-qt and peek into messages exchanged with other nodes. Since Bitcoin is a peer to peer network, the protocol includes mechanisms to discover other nodes. Further, since all nodes may not be at same version of protocol, there are mechanisms for handling version mismatches.

Wireshark

The best way to analyze network traffic is with a network dissector tool called Wireshark. You can download Wireshark from this page. I recommend using the Development version as it includes Bitcoin parser that matches the latest protocol.

Once you download and install Wireshark, fire it up.

• If bitcoin-qt is running, quit it.
• In the filter toolbar of wireshark, type bitcoin and click apply (check out Wireshark documentation to learn more).
• Once the filter has been applied, begin capturing live network traffic.
• Now start bitcoin-qt

As bitcoin-qt connects to Bitcoin network, you will see packets appearing in the display window.

Messages

Bitcoin nodes connect to other nodes via TCP. The nodes typically listen for messages on port 8333 (although the protocol allows configuring nodes to listen on any port). Each message passed on the Bitcoin network has a well defined structure.

Message structure

Field Size	Description	Data type	Comments
4	magic	uint32_t	Magic value indicating message origin network, and used to seek to next message when stream state is unknown
12	command	char[12]	ASCII string identifying the packet content, NULL padded (non-NULL padding results in packet rejected)
4	length	uint32_t	Length of payload in number of bytes
4	checksum	uint32_t	First 4 bytes of sha256(sha256(payload))
?	payload	uchar[]	The actual data

The 4 byte magic number for the main bitcoin network is 0xD9B4BEF9 and when sent over the wire, it is converted into little endian as F9 BE B4 D9.

Following the magic number is a 12 byte field describing the command being sent in the message. This is usually an ascii text that is zero padded. For example, a version message would have the 12 byte command ‘v’ ‘e’ ‘r’ ‘s’ ‘i’ ‘o’ ‘n’ 0x0 0x0 0x0 0x0 0x0

Next we have 4 byte little-endian integer that tells us how long the payload in this packet is in bytes. In the example shown below its 0x00000064 = 100 bytes

The next 4 bytes are a checksum of the payload. This is created by creating sha256 hash of sha256 hash of payload and then discarding everything after the first 4 bytes of this 32byte hash. So, in  the example packet above, the bytes 0x35 0xd1 0x4f 0xef (after the hightlighted length) are the checksum bytes.

Note: In case the packet has no payload, then payload length is 0x0 and checksum bits are 0x5d 0xf6 0xe0 0xe2. The checksum in this case is generated as

•  sha256(sha256(<empty string>))
• = sha256(0x e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855)
• = 0x5df6e0e2761359d30a8275058e299fcc0381534545f55cf43e41983f5d4c9456

The remaining bytes in the packet are length bytes of payload. How each payload is decoded, depends upon the protocol version and the command.

Version Message

When a Bitcoin node wakes up, the first thing it does after it configures listener ports etc. is to send an outgoing version command to a known node on the Bitcoin network.  This is the first step in establishing a connection to remote node. No two nodes can communicate without first exchanging a version message.

Note: How the first bitcoin node to connect to is found depends upon the implementation of node. For bitcoin-qt there are well defined rules as described here.

The version message is a way of advertising the presence and capabilities of the node to other nodes on the network. The command used in the top level message structure is zero-padded string “version”.

This packet has a variable length payload and the structure of version message payload is of the form:

Field Size	Description	Data type	Comments
4	version	int32_t	Identifies protocol version being used by the node
8	services	uint64_t	bitfield of features to be enabled for this connection
8	timestamp	int64_t	standard UNIX timestamp in seconds
26	addr_recv	net_addr	The network address of the node receiving this message
version >= 106
26	addr_from	net_addr	The network address of the node emitting this message
8	nonce	uint64_t	Node random nonce, randomly generated every time a version packet is sent. This nonce is used to detect connections to self.
?	user_agent	var_str	User Agent (0x00 if string is 0 bytes long)
4	start_height	int32_t	The last block received by the emitting node
1	relay	bool	Whether the remote peer should announce relayed transactions or not, see BIP 0037, since version >= 70001

The payload for an example version message sent over the wire looks like this:

The payload starts with a 4 byte int32 Protocol Version. In the packet above, this would be 0x00011171 = 70001.

Following this is an 8byte bitfield specifying features to be enabled for this connection. The only available service at this point is NODE_NETWORK

Value	Name	Description
1	NODE_NETWORK	This node can be asked for full blocks instead of just headers.

Consequently, the value of these 8 byte field is 0x0000000000000001

After node services is another 8 byte field. This is a 64bit integer timestamp in terms of seconds since epoch. For the example packet above, this would be Feb 22, 2014 15:36:27.000000000 PST:

Next we have 26 byte address of the receiving node of this message. So this would be the remote node that you are instantiating connection to. The network addresses in Bitcoin messages follow a structure:

Field Size	Description	Data type	Comments
4	time	uint32	the Time (version >= 31402). Except in Version message.
8	services	uint64_t	same service(s) listed in version
16	IPv6/4	char[16]	IPv6 address. Network byte order. The original client only supports IPv4 and only reads the last 4 bytes to get the IPv4 address. However, the IPv4 address is written into the message as a 16 byte IPv4-mapped IPv6 address(12 bytes 00 00 00 00 00 00 00 00 00 00 FF FF, followed by the 4 bytes of the IPv4 address).
2	port	uint16_t	port number, network byte order

Note that for version message, the timestamp is not prefixed.

So for our payload, the address would have 8 byte services, 16 byte address and 2 byte port.

Another key point for network addresses is that Address and Port fields are not little-endian, but follow the network byte order, whereas services is little-endian. For the example payload shown above, the receiving node portion looks like this:

 We have services = 0x0000000000000001. The address is 12 bytes of 00 00 00 00 00 00 00 00 00 00 FF FF followed by 4 bytes of IPv4 address i.e. 0x5b.0x79.0x0e.0x2d or in decimal: 91.121.14.45. The last 2 bytes – 0x20 0x8d form a 4 byte big-endian integer 0x208d = 8333. So, in summary we are connecting to 91.121.14.45:8333.

Next we have our own address a.k.a emitting node address.

Again, our node services is 0x0000000000000001. The address is again an IPv4 address, so the first 12 bytes of the 16 byte field are 00000000000000000000ffff, followed by the address as seen by the outside world i.e. 0x0a.0x01.0x4c.0xe1 i.e. 10.1.76.225. Our listening port is 8333 as well, so the last 2 bytes are 0x20 0x8d, same as it was for the receiving node.

Next there is a 64bit random number which can be used by node’s implementation to detect duplicate payloads.

After the random nonce is a variable length string containing the name of the Node implementation a.k.a. User agent. A VarStr is defined as VarInt describing the length of the string followed by string itself. (We discussed VarInts in our first post).

Field Size	Description	Data type	Comments
?	length	var_int	Length of the string in bytes
?	string	char[]	The string itself (can be empty)

And VarInt:

Value	Storage length	Format
< 0xfd	1	uint8_t
<= 0xffff	3	0xfd followed by the length as uint16_t
<= 0xffffffff	5	0xfe followed by the length as uint32_t
–	9	0xff followed by the length as uint64_t

So in our example payload:

The length is 0x0f = 15 bytes. The actual string following the length is:

  Hex: 2f 53 61 74 6f 73 68 69 3a 30 2e 38 2e 36 2f
Ascii: /  S  a  t  o  s  h  i  :  0  .  8  .  6  /

Note: VarStr doesn’t need to be null terminated

The last element in the Version payload is Block start height. This is the height of the latest block in our database. As Bitcoin node adds more blocks to it’s database, this height grows as well. When the node finds another node with higher Block start address, it can request newer blocks from that node to update its database.

So, in our example payload, the height is 0x0003aa65 = 240229

Conclusion

This completes our look at the version message. We will look at some of the other network messages in the next few posts and then combine them together to see how each node communicating with other nodes with a few well defined messages builds such a resilient network.

If you are enjoying the posts, you can support me my donating to my tip address: 1GdiiJNGCE8jnytNQUC2FVhMankAJUyQrn

The Bitcoin Protocol – 3 – Block Parser continued

Note: This is third in a multi-part post and builds on work from previous post.

So far, we have been able to parse most of the block including the block header. The only thing that’s left to parse is the list of transactions. If we can successfully build a generic parser for a transaction, then we can just loop over and parse all the transactions in the block.

Transactions (in a Block)

Like the block header, transactions are a complex structure. So we will build a separate class to represent a transaction.

You may want to go back to our first post in this series and refresh up on how the transactions look in a block.

The transaction object itself contains two lists of complex structures, one for Transaction Inputs and another for Transaction Outputs. So we will also make special classes to represent a transaction input and another one to represent a transaction output.

So the list of new classes would be:

• class Transaction
• class Tx_Input
• class Tx_Output

In this post, we will not be validating the transactions, but only printing it’s contents.

Tx_Input

Let’s first create the Tx_Input object. An input contains following elements in order

1. 32byte hash or previous transaction
2. 4byte index in the list of outputs of the previous transaction
3. A VarInt containing length of scriptSig, and
4. scriptSig
5. 4byte integer sequence_no

So looks like parsing the Tx_Input should be straightforward using the utility functions we created. Just like our other classes, we will have a parse() method. The finished class looks like this:

      
class Tx_Input(object):
   def __init__(self):
      super(Tx_Input, self).__init__()

   def parse(self, stream):
      self.prevhash = read_hash32(stream)
      self.prevtx_out_idx = read_uint4(stream)
      self.txin_script_len = read_varint(stream)
      # TODO in later modules we will convert scriptSig to its own class
      self.scriptSig = stream.read(self.txin_script_len)
      self.sequence_no = read_uint4(stream)

   def __str__(self):
      return 'PrevHash: %s \nPrev Tx out index: %d \nTxin Script Len: %d \nscriptSig: %s \nSequence: %8x' % \
               (get_hexstring(self.prevhash),
                self.prevtx_out_idx,
                self.txin_script_len,
                get_hexstring(self.scriptSig),
                self.sequence_no)

   def __repr__(self):
      return __str__(self)

Tx_Output

In a similar vein, we can create our Tx_Output class by reading the following elements in order.

1. 8byte value containing amount of Satoshis in this output
2. A VarInt containing length of scriptPubKey
3. scriptPubkey

The finished class would look like this:

class Tx_Output(object):
   def __init__(self):
      super(Tx_Output, self).__init__()
      pass

   def parse(self, stream):
      self.value = read_uint8(stream)
      self.txout_script_len = read_varint(stream)
      self.scriptPubKey = stream.read(self.txout_script_len)

   def __str__(self):
      return 'Value (satoshis): %d (%f btc)\nTxout Script Len: %d\nscriptPubKey: %s' %\
               (self.value, (1.0*self.value)/100000000.00, 
                self.txout_script_len, 
                get_hexstring(self.scriptPubKey))

   def __repr__(self):
      return __str__(self)

Transaction Class

We can now use Tx_Input and Tx_Ouput classes to build our transaction class, which will contain the following elements in order.

1. 4byte version
2. VarInt count of number of Inputs.
3. List of inputs. Each input will be represented by a Tx_Input object.
4. VarInt count of number of Outputs.
5. List of outputs. Each output will be represented by a Tx_Output object.
6. 4byte lock time

The finished class would look something like this:

class Transaction(object):
   """Holds one Transaction as part of a block"""
   def __init__(self):
      super(Transaction, self).__init__()
      self.version = None
      self.in_cnt = None
      self.inputs = None
      self.out_cnt = None
      self.outputs = None
      self.lock_time = None

   def parse(self,stream):
      #TODO: error checking
      self.version = read_uint4(stream)
      self.in_cnt = read_varint(stream)
      self.inputs = []
      if self.in_cnt > 0:
         for i in range(0, self.in_cnt):
            input = Tx_Input()
            input.parse(stream)
            self.inputs.append(input)
      self.out_cnt = read_varint(stream)
      self.outputs = []
      if self.out_cnt > 0:
         for i in range(0, self.out_cnt):
            output = Tx_Output()
            output.parse(stream)
            self.outputs.append(output)
      self.lock_time = read_uint4(stream)

   def __str__(self):
      s = 'Inputs count: %d\n---Inputs---\n%s\nOutputs count: %d\n---Outputs---\n%s\nLock_time:%8x' % (self.in_cnt,
               '\n'.join(str(i) for i in self.inputs),
               self.out_cnt,
               '\n'.join(str(o) for o in self.outputs),
               self.lock_time)
      return s

Putting it all together

Finally, we are ready to loop over all the transactions in the Block. For this, we will update the parseBlockFile() function in our block class and append:

         self.transactions = []

         print 'List of transactions'
         for i in range(0, self.transaction_cnt):
            tx = Transaction()
            tx.parse(bf)
            self.transactions.append(tx)
            print '='*50
            print ' TX NUMBER: %d' % (i+1)
            print '='*50
            print tx
            print '\n'

The completed version of file with all the above additions can be found on github here.

When we run our usual command, we will see a large amount of data being output on the screen. To help us better look at the data, we can redirect to a file called parsed_output.txt.

coinlogic.info@proto $>python block.py blk00003.dat > parsed_output.txt

I have uploaded this parsed_output on github as well. Please open up this link and see how our parsed output looks!

This completes our basic parser for a block. Now we have a solid understanding of the structure of a block. In our next post, we will be look into how transactions are validated and Bitcoin’s scripting language.

If you are enjoying the posts, you can support me my donating to my tip address: 1GdiiJNGCE8jnytNQUC2FVhMankAJUyQrn

The Bitcoin Protocol – 2 – Block parser

Note: This is second part of a multi-part post and builds upon the work done in the first post.

In the last post, we took a block .dat file and analyzed it using hexdump. In the process, we learned the structure of a Bitcoin block. We also looked at the structure of transactions present in the block and analyzed the first transaction, which is also known as the coinbase transaction.

Today we put our learning into practice by building a basic block parser in python. The block parser would take the name of a binary file containing a block as input, build a Block object and print a pretty parsed  version of the block.

Let’s call our parser block.py. We will be working with the file blk00003.dat that we began using in the previous post. So, when invoking the parser our command would look something like this:

python block.py blk00003.dat

First, fire up your favorite code editor and save an empty python file in the directory (e.g.~/coinlogic/) where you copied blk00003.dat.

Once open, paste this as the contents of the file:

def parseBlockFile(blockfile):
   print 'Parsing block file: %s' % blockfile

if __name__ == "__main__":
   import sys
   usage = "Usage: python {0} " 
   if len(sys.argv) < 2:
      print usage.format(sys.argv[0])
   else:
      parseBlockFile(sys.argv[1])

I have also uploaded this version of file on github here. When you run this :

coinlogic.info@proto $>python block.py blk00003.dat 
Parsing block file: blk00003.dat

Great! Now that we have our file setup. Let’s start creating the block parser. We will create a class called Block to represent the block. We will create a method of this class called parseBlockFile() that takes the block file name as a parameter and parses and prints it.

Find this version of block.py on github as well

class Block(object):
   """A block to be parsed from file"""
   def __init__(self):
     self.magic_no = -1
     self.blocksize = 0
     self.blockheader = None
     transaction_cnt = 0
     transactions = None

     blockfile = None

   def parseBlockFile(self, blockfile):
      print 'Parsing block file: %s' % blockfile

def parseBlockFile(blockfile):
   block = Block()
   block.parseBlockFile(blockfile)

if __name__ == "__main__":
   import sys
   usage = "Usage: python {0} " 
   if len(sys.argv) < 2:
      print usage.format(sys.argv[0])
   else:
      parseBlockFile(sys.argv[1])

Running it, we get:

coinlogic.info@proto $>python block.py blk00003.dat 
Parsing block file: blk00003.dat

Magic number

Now, we are in a position to start parsing the block. We will open the file as binary and read it one byte at a time. The first item is a 4 byte magic number as little-endian integer. We can use python’s struct module to convert this 4 byte little-endian representation into a python integer such as this:

struct.unpack('I', f.read(4))[0]

Since, we will be reading integers of 1,2,4 and 8 bytes we can create some utility functions that will help us read these values. These functions can all accept a generic file or stream object that has a read function. Here is what these functions might look like:

def read_uint1(stream):
    return ord(stream.read(1))

def read_uint2(stream):
    return struct.unpack('H', stream.read(2))[0]

def read_uint4(stream):
    return struct.unpack('I', stream.read(4))[0]

def read_uint8(stream):
    return struct.unpack('Q', stream.read(8))[0]

Now that we have a function that can read a 4 byte integer, we are ready to read our magic number. Let’s open the file as binary by passing ‘rb’ as mode to open() and pass it to our utility function to read the integer.

The updated Block class looks like this:

class Block(object):
   """A block to be parsed from file"""
   def __init__(self):
     self.magic_no = -1
     self.blocksize = 0
     self.blockheader = None
     transaction_cnt = 0
     transactions = None

     blockfile = None

   def parseBlockFile(self, blockfile):
      print 'Parsing block file: %s\n' % blockfile
      with open(blockfile, 'rb') as bf:
         self.magic_no = read_uint4(bf)
         print 'magic_no:\t0x%8x' % self.magic_no

The full block.py file at this stage can be found at github here.

Let’s run this file and see if we get a print out of the magic number.

coinlogic.info@proto $>python block.py blk00003.dat 
Parsing block file: blk00003.dat

magic_no:	0xd9b4bef9

Awesome! So this matches the expected value of the block magic number. Things are looking good.

Blocksize

Next, we will read the block size in a similar fashion:

         self.blocksize = read_uint4(bf)
         print 'size:    \t%u bytes' % self.blocksize

The version of block.py at this stage is here. Running this version, we see that we have successfully parsed the size.

Parsing block file: blk00003.dat

magic_no:	0xd9b4bef9
size:    	30000 bytes

Block Header

Next we will parse the block header. This is a complex structure and deserves a class of it’s own. We will appropriately call this class BlockHeader. We also encounter some new data types when parsing the header, so we begin by creating some more utility functions to help us in parsing 32 bit hashes, VarInts and get pretty string representation of the 32 bit hashes.

def read_hash32(stream):
   return stream.read(32)[::-1] #reverse it since we are little endian

def read_merkle32(stream):
   return stream.read(32)[::-1] #reverse it

def read_time(stream):
   utctime = read_uint4(stream)
   #Todo: convert to datetime object
   return utctime

def read_varint(stream):
   ret = read_uint1(stream)

   if ret < 0xfd: #one byte int 
      return ret
   if ret == 0xfd: #unit16_t in next two bytes
      return read_uint2(stream)
   if ret == 0xfe: #uint32_t in next 4 bytes
      return read_uint4(stream)
   if ret == 0xff: #uint42_t in next 8 bytes
      return read_uint8(stream)
   return -1

def get_hexstring(bytebuffer):
   return ''.join(('%x'%ord(a)) for a in bytebuffer)

Now we can also create the BlockHeader class itself as well:

class BlockHeader(object):
   """BlockHeader represents the header of the block"""
   def __init__(self):
      super( BlockHeader, self).__init__()
      self.version = None
      self.prevhash = None
      self.merklehash = None
      self.time = None
      self.bits = None
      self.nonce = None

   def parse(self, stream):
      #TODO: error checking

      self.version = read_uint4(stream)
      self.prevhash = read_hash32(stream)
      self.merklehash = read_merkle32(stream)
      self.time = read_time(stream)
      self.bits = read_uint4(stream)
      self.nonce = read_uint4(stream)

   def __str__(self):
      return "\n\t\tVersion: %d \n\t\tPreviousHash: %s \n\t\tMerkle: %s \n\t\tTime: %s \n\t\tBits: %8x \n\t\tNonce: %8x" % (self.version, \
               get_hexstring(self.prevhash), \
               get_hexstring(self.merklehash), \
                str(self.time), \
                self.bits, \
                self.nonce)

   def __repr__(self):
      return __str__(self)

As you can see, we overrode the __str__ and __repr__ functions to print the contents of the class in a pretty format.

Finally, we update the parseBlockFile() method of Block class to instantiate a BlockHeader and parse it.

         self.blockheader = BlockHeader()
         self.blockheader.parse(bf)
         print 'Block header:\t%s' % self.blockheader

The version of block.py at this stage is available here. Running it, we can now see we have the Block Header successfully parsed as well!

coinlogic.info@proto $>python block.py blk00003.dat 
Parsing block file: blk00003.dat

magic_no:	0xd9b4bef9
size:    	30000 bytes
Block header:	
		Version: 1 
		PreviousHash: 0000000e5b44cb9b537e79227ba232b45cbd5af5112f186d3bca221 
		Merkle: 4955619e67e7856c365dd9e2e0f1d3f2e1227bacbf52d621b93476f1e5349 
		Time: 1311016847 
		Bits: 1a0abbcf 
		Nonce:  aa64562

Transaction Count

After the block header is a VarInt holding the count of transcations in this block. We can read it easily using our utility function as:

         self.transaction_cnt = read_varint(bf)
         print 'Transactions: \t%d' % self.transaction_cnt

Running this version would print transaction count after the block header:

coinlogic.info@proto $>python block.py blk00003.dat 
Parsing block file: blk00003.dat

magic_no:	0xd9b4bef9
size:    	30000 bytes
Block header:	
		Version: 1 
		PreviousHash: 0000000e5b44cb9b537e79227ba232b45cbd5af5112f186d3bca221 
		Merkle: 4955619e67e7856c365dd9e2e0f1d3f2e1227bacbf52d621b93476f1e5349 
		Time: 1311016847 
		Bits: 1a0abbcf 
		Nonce:  aa64562
Transactions: 	64

You can browse the version of complete file at this stage here.

Next

In the next post, we will create Transaction object and parse all the transactions in this block.

If you are enjoying the posts, you can support me my donating to my tip address: 1GdiiJNGCE8jnytNQUC2FVhMankAJUyQrn

The Bitcoin Protocol – 1 – Block dissected

Note: This is the first of a multi-part post

Today, we begin looking into the very core of the Bitcoin network – the protocol. A sound foundation into the protocol is necessary to gain a true understanding into how and why this system works.

For more details into the communication layer look for my posts on The Network. 

Since this blog is fairly focused on technical aspects, we do not go into philosophical discussions unless they aid in a technical understanding of the material presented. Therefore, we assume that the reader has gained a fairly basic understanding of Bitcoin concept and is now looking for more technical understanding of the system

I believe that the best way to understand theoretical concepts is to apply them practically. So during the process of this post, we will build a basic Bitcoin node in python one step at a time. Python is a simple language with very little learning curve, so even if you are not familiar with it, you can quickly grasp it’s syntax.