Saturday, October 24, 2015

Nothing to Fear

Armand Jean du Plessis, Cardinal-Duke of Richelieu and Fronsac, 17th century Prime Minister of France, reputedly said something to the effect of:

"Give me six lines written by an honest man and I will find something in it to hang him with".

There are probably many things he said, that he didn't say, but it is a nice quote.

Beware of a Man in a Dress

The French encrypted communications up to the early 19th century, using simple ciphers known as petits chiffres. These were short notes, based on 50 numbers. Later, they began to write letters using a combination of 150 numbers, known as the Army of Portugal Code.  By 1812, new cipher tables were sent from Paris based on 1400 numbers and derived from a mid-18th century diplomatic code.

 18th Century Paris Cipher

Many people think that 'If you have nothing to hide, then you have nothing to fear', or simply state 'I have nothing to hide', meaning that they don't care about ubiquitous government surveillance.  Well, if you are that boring, then no-one would want to talk to you!

In reality, even the worst, boring, dullards try to keep their bank account, tax return, passport and credit card private and even Germans usually at least wear budgie smugglers when they go for a swim...

To me, the biggest problem is organized criminals who can drain a bank account in seconds, ruin your credit rating, sell your house while you are on holiday and state security agents acting like the infamous Cardinal, who can plant incriminating evidence on your IT systems to hang you with.  Just watch what is happening to Hillary Clinton and the Aussie PM who ran private email servers, which were legal at the time, but which are now being systematically stuffed with 'new discoveries' of 'classified information', much like the tiny mountain cabin of Ted Kazinsky, that was searched multiple times and each time delivered a treasure trove of new 'evidence' - there must have been a rather large cave under O'l Ted's shack.

Encrypt everything!

Encrypt your phone, your tablet, your laptop PC, your desktop PC, your email.  Buy a Black Phone.  Use the Red Phone.  Install RetroShare.  Use Free Software.

Use Keepass to save the passwords and keep the master password in your wallet if you don't trust your memory.

Do not make it easy for online miscreants hiding in their mother's basement on the other side of the world, to destroy your life.

Upon the death of the most revered Cardinal, Francois Marie Arouet (Voltaire), who wrote rather more than six lines criticizing the dear Cardinal said:

'He was a kind and generous man, 
provided of course, that he is really dead.'


Voltaire is really dead too, unfortunately.

Ecrasez l'Infame.


Sunday, October 4, 2015

Reading and Parsing Data from Serial Ports

Read, Echo, Octal Dump, Head, Cat, Chat and Serial Ports

Anyone who tried to parse data from a serial port in a Bash script will have run into trouble really quickly.  This post explores a few different methods.

Some people prefer using minicom and expect, others just want to read a prompt from an embedded target such as an Arduino and send it a file without spending too much time on learning new tricks.  The best way to do it is the way that works for you!

Rather than fighting with an actual serial port (/dev/ttyUSB0), most of these examples use echo and pipes to send binary and ASCII data to a parsing utility to show what it does and how to use it.

In a nut shell, if you need to parse human readable ASCII data, use read.  If you have to parse unreadable binary data, use od or head.  If it has to be very fast, use cat.  Read has a built-in timeout that you can use to keep it from getting stuck waiting forever.  The others, you have to kill with a timer when something goes wrong.

When extensive error handling and timeouts are required also with ASCII data, use chatChat is part of the pppd package and is usually installed by default.

Octal Dump and Head

This example uses echo to print out binary data to a pipe as a simulated serial port. Here shown with octal dump (od) to make the binary visible on screen:
$ echo -en "\x02\x05\x00\x01\x02\x0a\x0b\x0d\x0e" | od -tx1

0000000    02  05  00  01  02  0a  0b  0d  0e                            


Reading a number of data bytes with head works, but it doesn’t have a built-in timeout feature:

$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | head -c5 | od -tx1

0000000    40  41  00  42  01                                            


Reading a few data bytes directly with Octal Dump works, but it also has no built-in  timeout:
$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | od -N5 -tx1

0000000    40  41  00  42  01                                            

Read is best in a loop

Using read in a while loop on a mix of binary and ASCII data with od for debugging, shows the following funky behaviour:
#! /bin/bash

while read -t1 -n1 CHAR; do

echo $CHAR | od -tx1

done < <(echo -en "\x02\x05\x00\x41\x42\x02\x0d\x0a\x43")

0000000    02  0a                                                        


0000000    05  0a                                                        


0000000    0a                                                            


0000000    41  0a                                                        


0000000    42  0a                                                        


0000000    02  0a                                                        


0000000    0d  0a                                                        


0000000    0a                                                            


0000000    43  0a                                                        


So the 00H and 0AH gets absorbed as delimiters and a new 0AH added at the end of each token.

The 00H is especially bad to read and causes a reset from which it only recovers at the next 0AH, unless n=1.

Therefore, use od or dd or even head, to parse binary data and use read for human readable ASCII data.

It is important to put read in a loop, since it is very slow with opening the port, so you should not call read repeatedly - it will then likely drop characters.  In a while loop as above or below, it works better.

This works OK with read:
#! /bin/bash

while read -r -t1 ; do

echo $REPLY | od -tx1

echo $REPLY

done < <(echo -en "\x02\x05\x01NO CARRIER\x0d\x0aOK\x0d\x0a\x40”)

0000000    02  05  01  4e  4f  20  43  41  52  52  49  45  52  0d  0a    



0000000    4f  4b  0d  0a                                                



So read will parse ASCII tokens from garbage, provided that the garbage doesn’t contain 00H.

Sleep Timeouts

Here is an example to read binary data and put it in a file, with od and an error timeout:
#! /bin/bash

# Read 20 bytes with a 1 second error timeout

od -N20 -tx1 < <( echo -en "\x02\x05\x00\x01\x02\x03\x04\x0d\x0a" ) > /tmp/data.txt &


echo "PID=$PID"

sleep 1

kill $PID

cat /tmp/data.txt

0000000    02  05  00  01  02  03  04  0d  0a                            


In this case, od will either finish reading the data, or get killed  when the sleep times out, so your script will not hang if the device on the other side of the wire is dead.

Raw or Cooked Sushi

For working with actual serial ports, it is important to check whether to use use raw mode (don't wait for line terminators) or cooked mode (buffering on, wait for line end):
echo "Set serial port USB0 to 9600N81"
stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0 9600

Pussycat to the rescue

If the device under test is very fast and you experience dropped characters between a command and response, then you may need to use cooked mode with cat to read the port to a temporary file, then parse the file afterwards like this:

echo "Set serial port USB0 to 9600N81"
stty -F /dev/ttyUSB0 cooked
stty -F /dev/ttyUSB0 9600

echo -en "AT&V\r" > $PORT; cat < $PORT > $FILE &
sleep 1
kill $PID

if cat "$FILE" | grep "ERROR"; then
   let "ERRCNT+=1"

The /tmp file system is a RAM disk, so it is much faster than writing to a hard disk.

If you need the received data in a Bash variable for further processing, do this:
DATA=$( cat $FILE )

Interactive Scripts with 'chat'

A typical exchange with a radio or other embedded device goes something like:
  • Send a command
  • Get a response
  • If the response was good, then do an action
  • If the response was bad, then quit
Automating that with echo, cat and if statements is difficult and most of the program may end up being error handlers.  The chat program is part of the pppd package and is probably installed by default.  You could use chat to do scripts more effectively than with standard bash commands.

For example, receive ERROR, then abort, receive nothing then send at&v, receive OK then set register 9 to 2, or timeout after 3 seconds, in a one liner:
$ chat -v -s TIMEOUT 3 ABORT ERROR '' at&v OK ats9=2 </dev/ttyUSB0 >/dev/ttyUSB0 

With a few lines like the above in a script, you can do most anything with a simple radio modem device.

Serial Port Redirection with 'exec'

When playing with serial ports in scripts, you will find that each time you open and close the port, it takes time, so it can drop characters and the buffer contents become unknown also and may contain older junk making it hard to find what you are really looking for.   

Slow devices may be fine and dandy, but the same script with a faster device may never work right and leave you banging your head against the wall in frustration.

A simple trick to avoid opening and closing the port with each line of code, is to use the exec program to do permanent redirection of the stdio files.  After that, any program that reads and writes to stdio will talk to the serial port.  This is perhaps better explained with an example:

stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0 115200
exec 4<&1 >"$DEV" <"$DEV"

File handle 0 is stdin, filehandle 1 is stdout and 2 is stderror, so 4<&1 means redirect stdout to a new handle 4 and the >"$DEV" means simultaneously also redirect it to the USB serial port and lastly, <"$DEV" means redirect the USB serial port to stdin.

From then on, your chat script doesn't have to add <"$DEV" >"$DEV" to the end of every line, the serial port will remain open for the duration of the whole script.  Having duplicate handles is useful to still be able to access the screen and keyboard for a user response:

exec 3<&0 4<&1 >"$DEV" <"$DEV"
chat -v -s TIMEOUT 3 ABORT ERROR '' at&v OK ats9=2
echo "Press any key" >&4
read -u 3 -n 1 RESPONSE

You could also call chat inside an if statement for improved error handling:
if chat -v -s TIMEOUT 3 ABORT ERROR '' at&v OK ats9=2; then
   echo "Error S9" >&2
   exit 1
echo "done" >&4

More explanations on exec redirection here:

Serial Port Tips

La voila!