Network Debugging

With embedded systems, it is frequently necessary to look at the packets flowing on the network in order to see whether a problem is isolated, or caused by something else.

The problem is that there can be an enormous amount of data on the net and only one or two packets that you want to look at.  Especially when you have video streams flowing on the net, it becomes a search for a needle in a haystack.

The standard interactive network test tool is tcpdump.  It is also available on Windows as windump (http://www.winpcap.org/windump/install/).  Wireshark is also a nice front end for packet capture, but it works in a capture first, analyze later fashion.  With tcpdump, you can see what is going on in real time.

Packet Capture Examples

Tcpdump has a good man page, but it can be a bit overwhelming.  Once you saw a few real examples, though, deciphering the man page will be much easier.  The windump man page is exactly the same - they didn't even bother to change the name of the man page.

Display all packets on the wired port em1:
# tcpdump -nlX -i em1

Display all packets to/from a specific device:
# tcpdump -nlX -i em1 host 192.168.111.3

Display all packets to/from a specific host on a specific Berkeley port:
# tcpdump -nlX -i em1 host 192.168.111.3 and port 2000

Advanced Packet Capture Examples

Display all packets that has a specific byte in a specific place:
# tcpdump -nlX -i em1 ether[58] == 0x05

Display all packets that don't have two bytes in two specific places and are coming from a specific device:
# tcpdump -nlX -i em1 src host 192.168.111.3 and port 2000 and ether[58] != 0x05 and ether[59] != 0x7b

The trick is that you got to add the 14 byte ethernet frame header to the byte position (start counting your data from 0). The first byte (45H) that the below packet starts with is ether[14].

The result of the first search will look something like this:

14:52:56.860160 IP 192.168.111.12.55338 > 192.168.111.4.sieve-filter: UDP, length 66
    0x0000:  4500 005e 1513 4000 8011 861a c0a8 6f0c  E..^..@.......o.
    0x0010:  c0a8 6f04 d82a 07d0 004a cd76 3130 0000  ..o..*...J.v10..
    0x0020:  0000 0000 0000 0000 00d4 0000 057b 0000  .............{..
    0x0030:  0020 ffff ffff ffff ffff 41d4 c6bd 9c40  ..........A....@
    0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0050:  0000 0000 0000 0000 0866 0000 0daf       .........f....

I bolded the 057bH that I was looking for.  The second search will show packets where the 057bH is missing.  

This can settle an argument between two developers very quickly!

PCAP Files

You can also start a capture and let it save to a file, then analyze it later if you have to.  Tcpdump smart file handling can be configured to create multiple log files and rotate them, so that it will not overflow your available storage space.

It is easy to write data to a file:
# tcpdump -nlX -i em1 -w filename.pcap

Read from a file:
# tcpdump -nlX -r filename.pcap

It works exactly the same when analyzing a file offline as when you analyze data real-time.

La voila!