Reading and Parsing Data from Serial Ports

Read, Echo, Octal Dump, Head, Cat, Chat and Serial Ports

Anyone who tried to parse data from a serial port in a Bash script will have run into trouble really quickly.  This post explores a few different methods.

Some people prefer using minicom and expect, others just want to read a prompt from an embedded target such as an Arduino and send it a file without spending too much time on learning new tricks.  The best way to do it is the way that works for you!

Rather than fighting with an actual serial port (/dev/ttyUSB0), most of these examples use echo and pipes to send binary and ASCII data to a parsing utility to show what it does and how to use it.

In a nut shell, if you need to parse human readable ASCII data, use read.  If you have to parse unreadable binary data, use od or head.  If it has to be very fast, use cat.  Read has a built-in timeout that you can use to keep it from getting stuck waiting forever.  The others, you have to kill with a timer when something goes wrong.

When extensive error handling and timeouts are required also with ASCII data, use chat.  Chat is part of the pppd package and is usually installed by default.

Octal Dump and Head

This example uses echo to print out binary data to a pipe as a simulated serial port. Here shown with octal dump (od) to make the binary visible on screen:

$ echo -en "\x02\x05\x00\x01\x02\x0a\x0b\x0d\x0e" | od -tx1

0000000    02  05  00  01  02  0a  0b  0d  0e                            

0000011

Reading a number of data bytes with head works, but it doesn’t have a built-in timeout feature:

$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | head -c5 | od -tx1

0000000    40  41  00  42  01                                            

0000005

Reading a few data bytes directly with Octal Dump works, but it also has no built-in  timeout:

$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | od -N5 -tx1

0000000    40  41  00  42  01                                            

0000005

Read is best in a loop

Using read in a while loop on a mix of binary and ASCII data with od for debugging, shows the following funky behaviour:

#! /bin/bash

while read -t1 -n1 CHAR; do

echo $CHAR | od -tx1

done < <(echo -en "\x02\x05\x00\x41\x42\x02\x0d\x0a\x43")

0000000    02  0a                                                        

0000002

0000000    05  0a                                                        

0000002

0000000    0a                                                            

0000001

0000000    41  0a                                                        

0000002

0000000    42  0a                                                        

0000002

0000000    02  0a                                                        

0000002

0000000    0d  0a                                                        

0000002

0000000    0a                                                            

0000001

0000000    43  0a                                                        

0000002

So the 00H and 0AH gets absorbed as delimiters and a new 0AH added at the end of each token.

The 00H is especially bad to read and causes a reset from which it only recovers at the next 0AH, unless n=1.

Therefore, use od or dd or even head, to parse binary data and use read for human readable ASCII data.

It is important to put read in a loop, since it is very slow with opening the port, so you should not call read repeatedly - it will then likely drop characters.  In a while loop as above or below, it works better.

This works OK with read:

#! /bin/bash

while read -r -t1 ; do

echo $REPLY | od -tx1

echo $REPLY

done < <(echo -en "\x02\x05\x01NO CARRIER\x0d\x0aOK\x0d\x0a\x40”)

0000000    02  05  01  4e  4f  20  43  41  52  52  49  45  52  0d  0a    

0000017

NO CARRIER

0000000    4f  4b  0d  0a                                                

000000

OK

So read will parse ASCII tokens from garbage, provided that the garbage doesn’t contain 00H.

Sleep Timeouts

Here is an example to read binary data and put it in a file, with od and an error timeout:

#! /bin/bash

# Read 20 bytes with a 1 second error timeout

od -N20 -tx1 < <( echo -en "\x02\x05\x00\x01\x02\x03\x04\x0d\x0a" ) > /tmp/data.txt &

PID=$!

echo "PID=$PID"

sleep 1

kill $PID

cat /tmp/data.txt

0000000    02  05  00  01  02  03  04  0d  0a                            

0000011

In this case, od will either finish reading the data, or get killed  when the sleep times out, so your script will not hang if the device on the other side of the wire is dead.

Raw or Cooked Sushi

For working with actual serial ports, it is important to check whether to use use raw mode (don't wait for line terminators) or cooked mode (buffering on, wait for line end):

echo "Set serial port USB0 to 9600N81"
stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0 9600

Pussycat to the rescue

If the device under test is very fast and you experience dropped characters between a command and response, then you may need to use cooked mode with cat to read the port to a temporary file, then parse the file afterwards like this:

echo "Set serial port USB0 to 9600N81"
stty -F /dev/ttyUSB0 cooked
stty -F /dev/ttyUSB0 9600

FILE="/tmp/data.txt"
PORT="/dev/ttyUSB0"
 

echo -en "AT&V\r" > $PORT; cat < $PORT > $FILE &
PID=$!
sleep 1
kill $PID

if cat "$FILE" | grep "ERROR"; then
   let "ERRCNT+=1"
fi

The /tmp file system is a RAM disk, so it is much faster than writing to a hard disk.

If you need the received data in a Bash variable for further processing, do this:
DATA=$( cat $FILE )

Interactive Scripts with 'chat'

A typical exchange with a radio or other embedded device goes something like:

• Send a command
• Get a response
• If the response was good, then do an action
• If the response was bad, then quit

Automating that with echo, cat and if statements is difficult and most of the program may end up being error handlers.  The chat program is part of the pppd package and is probably installed by default.  You could use chat to do scripts more effectively than with standard bash commands.

http://www.tldp.org/HOWTO/PPP-HOWTO/x1219.htm

For example, receive ERROR, then abort, receive nothing then send at&v, receive OK then set register 9 to 2, or timeout after 3 seconds, in a one liner:

$ chat -v -s TIMEOUT 3 ABORT ERROR '' at&v OK ats9=2 </dev/ttyUSB0 >/dev/ttyUSB0 

With a few lines like the above in a script, you can do most anything with a simple radio modem device.

Serial Port Redirection with 'exec'

When playing with serial ports in scripts, you will find that each time you open and close the port, it takes time, so it can drop characters and the buffer contents become unknown also and may contain older junk making it hard to find what you are really looking for.   

Slow devices may be fine and dandy, but the same script with a faster device may never work right and leave you banging your head against the wall in frustration.

A simple trick to avoid opening and closing the port with each line of code, is to use the exec program to do permanent redirection of the stdio files.  After that, any program that reads and writes to stdio will talk to the serial port.  This is perhaps better explained with an example:

stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0 115200
DEV="/dev/ttyUSB0"
exec 4<&1 >"$DEV" <"$DEV"

File handle 0 is stdin, filehandle 1 is stdout and 2 is stderror, so 4<&1 means redirect stdout to a new handle 4 and the >"$DEV" means simultaneously also redirect it to the USB serial port and lastly, <"$DEV" means redirect the USB serial port to stdin.

From then on, your chat script doesn't have to add <"$DEV" >"$DEV" to the end of every line, the serial port will remain open for the duration of the whole script.  Having duplicate handles is useful to still be able to access the screen and keyboard for a user response:

DEV="/dev/ttyUSB0"
exec 3<&0 4<&1 >"$DEV" <"$DEV"
chat -v -s TIMEOUT 3 ABORT ERROR '' at&v OK ats9=2
echo "Press any key" >&4
read -u 3 -n 1 RESPONSE

You could also call chat inside an if statement for improved error handling:
if chat -v -s TIMEOUT 3 ABORT ERROR '' at&v OK ats9=2; then
   echo "Error S9" >&2
   exit 1
fi
echo "done" >&4

More explanations on exec redirection here: http://wiki.bash-hackers.org/howto/redirection_tutorial

Serial Port Tips

www.aeronetworks.ca/2015/01/serial-port-io.html
www.aeronetworks.ca/2014/10/serial-ports-revisited.html
www.aeronetworks.ca/2014/01/crcs-and-serial-ports.html
www.aeronetworks.ca/2013/10/serial-port-tricks.html
www.aeronetworks.ca/2013/05/usb-serial-device-with-unknown-ids.html
www.aeronetworks.ca/2015/10/reading-and-parsing-data-from-serial.html
www.aeronetworks.ca/2013/05/compile-moxa-serial-widget-device.html

La voila!

Herman