Sunday, October 4, 2015

Reading and Parsing Data from Serial Ports

Read, Echo, Octal Dump and Serial Ports

Anyone who tried to parse data from a serial port in a Bash script will have run into trouble really quickly.  This post explores a few different methods.
Some people prefer using minicom and expect, others just want to read a prompt from an embedded target such as an Arduino and and send it a file without spending too much time on learning new tricks.  The best way to do it is the way that works for you!

Rather than fighting with an actual serial port (/dev/ttyUSB0), these examples use echo and pipes to send binary and ASCII data to a parsing utility to show what it does and how to use it.

In a nut shell, if you need to parse human readable ASCII data, use read.  If you have to parse unreadable binary data, use od or head.  Read has a built-in timeout that you can use to keep it from getting stuck waiting forever.  The others, you have to kill with a timer when something goes wrong.

This example uses echo to print out binary data to a pipe or a serial port. Here shown with octal dump (od) to make the binary visible on screen:
$ echo -en "\x02\x05\x00\x01\x02\x0a\x0b\x0d\x0e" | od -tx1

0000000    02  05  00  01  02  0a  0b  0d  0e                            

Reading a number of data bytes with head works, but it doesn’t have a built-in timeout feature:

$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | head -c5 | od -tx1
0000000    40  41  00  42  01                                            

Reading a few data bytes directly with Octal Dump works, but it also has no built-in  timeout:
$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | od -N5 -tx1

0000000    40  41  00  42  01                                            

Using read in a while loop on a mix of binary and ASCII data with od for debugging shows the following funky behaviour:
#! /bin/bash
while read -t1 -n1 CHAR; do
echo $CHAR | od -tx1
done < <(echo -en "\x02\x05\x00\x41\x42\x02\x0d\x0a\x43")

0000000    02  0a                                                        
0000000    05  0a                                                        
0000000    0a                                                            
0000000    41  0a                                                        
0000000    42  0a                                                        
0000000    02  0a                                                        
0000000    0d  0a                                                        
0000000    0a                                                            
0000000    43  0a                                                        
So the 00H and 0AH gets absorbed as delimiters and a new 0AH added at the end of each token.

The 00H is especially bad to read and causes a reset from which it only recovers at the next 0AH, unless n=1.

Therefore, use od or dd or even head, to parse binary data and use read for human readable ASCII data.

This works OK with read:
#! /bin/bash
while read -r -t1 ; do
echo $REPLY | od -tx1
echo $REPLY
done < <(echo -en "\x02\x05\x01NO CARRIER\x0d\x0aOK\x0d\x0a\x40”)
0000000    02  05  01  4e  4f  20  43  41  52  52  49  45  52  0d  0a    
0000000    4f  4b  0d  0a                                                

So read will parse ASCII tokens from garbage, provided that the garbage doesn’t contain 00H.

Here is an example to read binary data and put it in a file, with od and an error timeout:
#! /bin/bash
# Read 20 bytes with a 1 second error timeout
od -N20 -tx1 < <( echo -en "\x02\x05\x00\x01\x02\x03\x04\x0d\x0a" ) > /tmp/data.txt &
echo "PID=$PID"
sleep 1
kill $PID
cat /tmp/data.txt
0000000    02  05  00  01  02  03  04  0d  0a                            

In this case, od will either finish reading the data, or get killed  when the sleep times out, so your script will not hang if the other side is dead.
For working with actual serial ports, it is important to use raw mode (turn buffering off):
echo "Set serial port USB0 to 9600N81"
stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0 9600

La voila!


Monday, September 21, 2015

Zenity Progress Dialogue

Zenity is a nice and simple way to prettify Bash scripts, similar to kdialog, xdialog and a few others, but on the whole I think the best of the lot.  Though there is nothing preventing you from mixing them up if one has a feature that would be more useful for something.

I started using Zenity years ago, when there wasn't much documentation and consequently didn't use the progress dialogue the right way.  The result being that some scripts eventually broke.  For example, there was no information on how to make the progress bar progress - so I only used the whirr-whirr pulsate effect.

No Progress

In the past, I simply started a pulsater, pushed it into the background, saved the PID and later killed it when necessary, like this:

zenity --progress &
echo "Do something"
kill $PID

That used to worked fine, but recently on Fedora 22, it just sits there and does nothing.  It won't even blink.  Dead as a doornail.  Bah, humbug...

When All Else Fails...

Eventually, I found the documentation and RTFMed:

The problem is that the progress dialog has a new bug and it expects to receive something over stdin in order to get going.  If it never gets anything, it does nothing - in previous versions it worked fine in the background, but no-more.  When it is pushed into the background with &, it just sits there and sulks.

The correct way to make a progress pulsater that must do something and exit based on a condition, at which point one has to stop the pulsater, is like this:
while TRUE, do
  echo "This Will Do Nothing"
  echo "#Change The Text"
  sleep 5
  let "CNT+=1"

  if [ "$CNT" -eq 10 ]; then
    # Close the progress dialog with 100%
    echo "100"
    exit 0

  echo "Continue doing nothing"
) | zenity --progress \
--width=350 \
--title="The Title"
--text="The Default Text" \
--no-cancel \
--auto-close \

The --auto-close parameter will cause the progress dialogue to stop when you echo "100" percent.

An echo statement that starts with a # will change the text and one without, will be ignored.  The --no-cancel will suppress the Cancel button, since it doesn't make sense.

La voila!

Saturday, September 19, 2015

Version Control For Scripts

Running a full blown version control system such as subversion or CVS for Bash scripts is a bit of overkill.  Most people don't use any version control at all on scripts and revert back to the ancient way of saving multiple copies of the files.

The inevitable result is a mess of files called script1, script2, script.ok, script.bad... and then after a few days, you cannot remember which is which.

Fortunately, there is an ancient, light weight versioning system that is very easy to set up and which is perfect for the single user case: RCS.

Setup RCS

It is probably already installed (try whereis rcs), otherwise do:
# yum install rcs

In your script directory, make a directory to keep the archives:
$ mkdir RCS

Edit each script and add the $Id$ keyword to the top, right underneath the familiar #! /bin/bash command:
#! /bin/bash

That will substitute your name, date, time and version number of the file upon checkout, so you always know which version of the script you got.

Check In

As soon as you made an important change, check the file in, with ci:
$ ci scriptname

Type a proper description of the change, it is for your own benefit, when you have to look at a problem, possibly years later.

The file will disappear!

Check Out

Check it out again with co:
$ co scriptname 

By default you get the latest version.  You can also check out older versions using the -r parameter and switch between versions to see which one works best, without losing anything, since they are all in the archive file.

Important Tricks

If you are the only person working on the scripts, then remove strict locking.  This will make your life much easier:
$ rcs -U *

Now you can check files in with the -u option, so they don't disappear and gets checked out again immediately:
$ ci -u scriptname
You can also set an alias (add to the bottom of ~/.bashrc), to save you some more typing, so that you never have to use the co command again, just ci filename:
$ alias ci='ci -u'

That is pretty much all there is to it.

Do read the man pages of ci, co and rcs for details on what to do when the inevitable happens and you want to look at the logs and revert to an older version of a script.

Friday, September 11, 2015

If It Is Free, Take Two... Aspirin

Free Candy

There are many an old proverb about free stuff, but the latest action by Microsoft takes the cake.

Even people who declined the free Windows 10 upgrade, will now also have this bloated piggy forced down their throats:

It is one thing sending out free security patches and feature upgrades, but a DVD sized download will cause lots of trouble for thousands of users.  Many a hard disk doesn't have that much free space and many users are on bandwidth limited networks and may get hefty overage bills, for something they did not ask for.

On the other hand, what MS may be finally admitting to, is that Windows 8 is so bad, that the only solution is to replace the whole rotten thing - for free.

Free Insecurity

To add insult to injury, Windows 10 has very serious security issues, which I have investigated a bit with tcpdump and listed in another post:

At this point in time, I cannot recommend to anyone to use Win10.  It is not worth the security risk.

Also, there are complaints that some of the new spyware features are also being pushed out to Windows 7 and 8.

Eventually, there will be some trustworthy third party firewall programs that will make it easy to strangle it all, but until then, Windows 7, 8 and 10 should only be used behind a separate draconian firewall device.  Using a vanilla Win10 laptop PC in a coffee shop or airport would be a disaster waiting to happen, like WinXP SP1 was.

A major problem with 3rd party firewalls though, is that I have never seen a proper review done by someone who actually tested it with a packet sniffer to see if it really works, since most reviewers are ordinary journalists with little or no technical knowledge, who don't know the first thing about tcpdump or wireshark.

The Real Problem: Dirty Tricks

I am not really worried about marketing data sent to Microsoft.  I am worried about what else gets sent in and out and who else is listening while the packets travel half way around the world.  Any government security agency with half a clue can plant incriminating data on Windows computer systems.  If MS can download a whole DVD worth of junkware to half the desktop machines in the world, then how hard could it be for the NSA or FSB to plant child porn, terrorism related junk, or classified data on a machine and then hit the owner with a long jail sentence?

If you were wondering about dirty tricks, have a look at what is happening to Hillary Clinton.  The US authorities has retroactively classified data that wasn't classified at the time.  Now, they can charge her and the previous administrator of her mail server and put both of them in the clink.  The only thing saving Hillary, is the fact that her husband is an ex president, but if I were her mail server administrator, then I'd move to another country, far, far away.

From Soft In The Middle, To Soft In The Head

What on earth were MS thinking?



Sunday, September 6, 2015

UAE Visa Renewal Red Tape

Latest Visa Renewal Process in Al Ain

Go to SEHA Immigrant Screening Centre opposite Al Ain Hospital.
Hours: 7 am till 3 pm, Sun - Thu

You have three choices:
  • Go with the enormous crowd gathered under the palm trees...
  • Take the Fast Track - limited numbers - be there early
  • VIP service

VIP Medical Service

  • Go in the main front entrance and go upstairs. 
  • Walk through the Fast Track men's waiting room to the VIP waiting section.  
  • There is a tiny little VIP sign above the door.
  • There will be almost nobody, so the service is immediate.
  • You'll be done in about 20 minutes.
  • You need: Passport and passport copies (first 4 pages plus visa)
  • Cost: Dh500
  • In before 10h00, get results by 2 pm the same day
  • You need to get a blood test and a chest X-ray
  • They will send you a SMS when they are done around 1 pm - be there at 2 pm.

Typing Office - Emirates ID

  • There is a Typing Office (Registry shop) to the East of Jimmy Mall, also in the Municipal building.
  • Renew your Emirates ID for Dh370
  • You will get another SMS if it worked in a few minutes
  • Take the form to the Emirates ID building and have it stamped at reception - they don't even check it.

New Visa

  • Now your employer has 10 days to get you a new visa, after which you can collect you new ID at the Post Office.
  • You have to give your employer your medical report, stamped Emirates ID form, a copy of your medical insurance card, four photos and your passport, and hope you get it back...

La voila!


Friday, September 4, 2015

Audio Networking with Sox and Netcat

Sox + Netcat = VoIP

The idea is to hook the sound devices on two machines together so that one has a bidirectional intercom between them.  This way, one can have a radio hooked to a distant computer and a headset on a local computer and do VoIP with a remote radio transceiver over a LAN.

The main thing missing then is the PTT switch, which one can do with another netcat proxy to the serial control port of the transceiver and radio control software running locally to set the channel and key the radio.  This would make a good setup for a radio ham who wants to key a remote radio in his garden shack, from the comfort of his living room.

For streaming, one needs a headerless, self synchronizing CODEC protocol, to connect to a running stream.  Examples are ADPCM, CVSD and LPC10.

Sox gotchas

Sound Exchange has three programs: sox, rec and play.  The difference is that if you run sox, it will glom onto the sound device for both read and write, but it can only do one thing at a time, so it can either record or play, but not both at the same time.

The rec and play programs however, can run independently and concurrently, so those can be used to make a bidirectional intercom.

UNIX systems have many different sound systems: OSS, ALSA, Coreaudio, Pulseaudio, Jack...  Sox knows how to handle them, if you let it use the default settings.  So get the default to work first with a simple mixer application such as aumix, rec and play, before you try something complicated.

Sox also has internal buffering that is extremely large: 8 kilobytes.  The result is that sox by default has a huge delay.  When you use compression, then the more you compress, the longer the delay gets, because the slower stream takes longer to fill the buffers.  You can reduce that with the --buffer parameter.

In general, a simpler sound system such as ALSA or OSS will have smaller buffers and lower delays than the complex pulseaudio or coreaudio, so stick to the basics for best results.

The general syntax is: sox inputspec inputfile outputspec outputfile

A lone "-" instead of a file, tell sox to use stdio or a pipe, so read the examples carefully.  The data flow is from left to right in all the below examples.

Default setup

First get sox to work on the default sound device:

Make a noise:
$ cat /dev/urandom | sox -traw -r44100 -b16 -eunsigned-integer - -d

Play a tone:
$ play -n synth 10 sin 500

$ rec -r8000 -tcvsd file

$ play -r8000 -tcvsd file

Don't bother with the below if you cannot get the above to work.

Hook sox to netcat on two computers:

Assuming IP addresses and

With CVSD, be sure to set the sample rate and reduce the buffer size to reduce the delays.  The resulting network bandwidth is about 1 kilobyte per second and the end to end delay is about 1 second.  CVSD is a very simple CODEC, so it works well on an embedded processor.

On the first computer:
$ rec —buffer 32 -tcvsd -r8000 - | nc -u -l 5555 | play —buffer 32 -tcvsd -r8000 -

and on the second computer:
$ rec —buffer 32 -tcvsd -r8000 - | nc -u 5555 | play —buffer 32 -tcvsd -r8000 -

Just two one liners!

The above works and I had it running between a Macbook Pro with OSX (Mac sox is available from Homebrew) and a Fedora Linux laptop.  I also experimented on a BSD laptop and it behaves the same. 

The main problem is with delays due to buffering and the more one compresses the data stream, the longer it takes to fill the pipes, hence longer delays.  The LPC10 CODEC will reduce the network bandwidth further, but the delay will be ridiculous.

To make it robust, you may want to put a while loop with a sleep delay around the above, so that if sox or netcat would exit for whatever reason, it would start again, after waiting a little bit for the dust to settle:
$ while true; do rec —buffer 32 -tcvsd -r8000 - | nc -u 5555 | play —buffer 32 -tcvsd -r8000 -; sleep 1; done

...and it is still a one liner!  OK, technically, it is three.

It took me about 3 days to figure the above out, so when all else fails, RTFM and try again.

It is certainly much better than two paper cups and string...

La voila!


Saturday, June 27, 2015

Mac Backups With Rsync to Encrypted Flash Media

Backup Woes

The main advantage of using a Mac is that it defaults to idiot mode and the main annoyance with a Mac is that it defaults to idiot mode, but underneath it all there is BSD...

The Mac Time Machine backup utility is one such idiot application which is probably fine for a large number of people,  but which doesn't fit my usage scenario.  Coupled with the Disk Utility which makes it hard to format a Flash Memory Device properly, it is even worse.

I want to make backups to USB and SD memory devices, since they are small, solid state and convenient.  However, to make a proper backup, the device should be formatted with HFS+ and it must be Encrypted.  The Disk Utility defaults to the prehistoric FAT in order to support file exchange with legacy Windows machines and FAT doesn't support proper file ownership and permissions.

Also, it is rather self defeating if your HDD is encrypted, while your backups are in plain text on a FAT disk.

Format a Removable Disk with HFS+

  • Run Disk Utility
  • Select ‘1 Partition’ from the Volume Scheme menu.
  • Enter a name eg MACBACKUP1
  • Select Mac OS X Case Sensitive Extended (Journaled) from the Format menu.
  • Click the ‘Options’ button and select ‘GUID Partition Table’ 

 Enable File Ownership Data

  • Right click the flash drive and select ‘Get Info’.
  • Click the lock icon in the bottom right corner.
  • Remove the check mark from ‘Ignore ownership on this volume.

Encrypt the Flash Disk

  • Right click the disk icon and select Encrypt.
  • Generate and save your password in KeepassX.

Backup with Rsync

Now finally, the media is ready to be used for a secure backup.  Time Machine works if the media is formatted this way, but it defaults to backing up everything and a Flash Disk is usually too small to backup everything.  Also, there is no point in backing up the whole system, since one can download a new system from Apple.  I only need to backup my Documents folder - everything else is expendable.

Make a file called /Volumes/MACBACKUP1/sync
Make a directory called /Volumes/MACBACKUP1/bak

Edit the sync script with vi:

#! /bin/bash
rsync -avz --progress --delete --max-delete=10 --max-size=20M ~/Documents /Volumes/MACBACKUP1/bak

Make it executable with chmod +x sync

By selecting a max-size of 20 Megabytes, I exclude things like large archives and ISO images.  Deletions are limited to ten files, in order to prevent a possible disastrous delete of the whole backup set, following a local delete disaster.  The result is a 51 GB backup set, which fits on a 64 GB memory schtick - nice.

Now, you can insert the backup device and click the sync script to backup whatever should be backed up to that device and thereby backup Documents to a USB schtick and backup Pictures to a SD card for example, while having it all secure and encrypted, in case the flash thingy gets lost or stolen or copied by the NSA/CSIS/GCHQ/FSB...

La voila!