Ethernet Funnies

Sometimes it is very hard to connect to an embedded system, because the designers cut some corners to simplify the system and keep memory use down, or simply because the system is prehistoric and full of bugs.

One such device worked fine provided that there was a little ethernet switch between the laptop machine and the target, but a direct connection between the laptop machine and target device only worked about half the time.  Even the little switch sometimes could not connect.

MAC, PHY, MAG

An ethernet interface device consists of three main parts: The Media Access Controller (MAC), the Physical Interface (PHY) and a set of transformers - the Magnetics.  When you plug a cable in, the PHY sends out little pulses to figure out what is going on and then swaps the wires around internally and changes the speed and duplex settings to make the interface work.

The trouble was that the target only supports 100 Mbps, while the laptop machine wanted to run at 1 Gbps and the two just could not reach agreement.

Ethtool

The ethtool program can be used to configure the ethernet interface device manually:
# ifconfig em0 up
# ethtool em0
# ethtool -s em0 speed 100 duplex full autoneg off
# ethtool em0
# ifconfig em0 192.168.111.1 netmask 255.255.255.0

That forced the laptop machine to the correct speed and duplex settings, turned the broken auto negotiation off and then life was good.

Shortly after writing the above, I ran into a case where the embedded system works better with a 100 Mbps half duplex connection, but the auto-negotiation usually resulted in a full duplex connection.

# ethtool -s em0 speed 100 duplex half autoneg off

Problem fixed.

These weird issues are usually due to a bad board layout around the ethernet chip set.

Reference

More information here:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s1-ethtool.html


La voila!

Herman

Compile The Latest ffplay From Source

Compile latest version of ffplay from source

Note that this script doesn't overwrite the existing ffmpeg installation:
https://trac.ffmpeg.org/wiki/CompilationGuide/Centos

The gotcha in the above guide is the SDL-devel package without which ffplay will not build.

#! /bin/bash
yum install autoconf automake cmake freetype-devel gcc gcc-c++ git libtool make mercurial nasm pkgconfig zlib-devel SDL-devel


mkdir ~/ffmpeg_sources


cd ~/ffmpeg_sources
git clone --depth 1 git://github.com/yasm/yasm.git
cd yasm
autoreconf -fiv
./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin"
make
make install
make distclean


cd ~/ffmpeg_sources
git clone --depth 1 git://git.videolan.org/x264
cd x264
PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin" --enable-static
make
make install
make distclean


cd ~/ffmpeg_sources
hg clone https://bitbucket.org/multicoreware/x265
cd ~/ffmpeg_sources/x265/build/linux
cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="$HOME/ffmpeg_build" -DENABLE_SHARED:bool=off ../../source
make
make install


cd ~/ffmpeg_sources
git clone --depth 1 git://source.ffmpeg.org/ffmpeg
cd ffmpeg
PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/ffmpeg_build" --extra-cflags="-I$HOME/ffmpeg_build/include" --extra-ldflags="-L$HOME/ffmpeg_build/lib" --bindir="$HOME/bin" --pkg-config-flags="--static" --enable-gpl --enable-libx264 --enable-libx265 --enable-ffplay
make
make install


cd /usr/lib
ln -s /home/username/bin/ffplay fffplay

Play Video With Low Latency Using ffplay

This is the fastest I can make ffplay:

$ ffplay -threads 2 -flags low_delay -fflags nobuffer -rtbufsize 32768 -fast -probesize 800000 -analyzeduration 800000 -ss 1 -framerate 50 udp://224.0.1.6:2006

Play Video With ffmpeg

This also quite fast, just to show that you can play video directly with ffmpeg using SDL output:

$ ffmpeg -i udp://224.0.1.6:2006 -f sdl -

Play Video With gstreamer

Just because it also pertains to playing video, here are a couple tricks with gstreamer:

$ gst-launch -v playbin uri=udp://224.0.1.6:2006

First try a video test pattern though:

$ gst-launch-0.10 videotestsrc ! ffmpegcolorspace ! autovideosink

If the test pattern doesn't work, then there is no point in continuing - you got to fix it.

La Voila!

Herman

Windows Insanity

Unsolicited Advertisements

To add insult to injury, Microsoft Windows 10 now displays advertisements right in the File Explorer. 

I'll leave it to you to figure out why that is a very bad security problem.

 

Rusted Sieve

Windows 10 is about as secure as a rusted sieve, with a few deliberate holes poked in it for good measure.  Microsoft essentially tries to convert your Personal Computer into a cell phone, which is a purpose built blabbing and tracking device.  They don't seem to understand the word 'personal' in 'PC' though.

https://answers.microsoft.com/en-us/windows/forum/windows8_1-update/what-is-diagnostics-tracking-service-which-was/253fe2ec-fba6-4240-bfb8-2a3bdc801ed1?auth=1

Examples of data we collect include your name, email address, preferences and interests; browsing, search and file history; phone call and SMS data; device configuration and sensor data; and application usage.

Recently, the Diagnostics Tracking Service (DiagTrack) service was renamed to the Connected User Experiences and Telemetry service.  Sigh...

Microsoft, in their infinite wisdom, hooked up a plethora of tracking systems and tunnels, to capture your keystrokes, your voice, every address you visit on the web, your WiFi router passwords and your disk drive encryption keys.  I guess they figured that your camera is already captured by Skype, so they didn't need to add that to the list.  I can see many a giant law-suit lurking on the horizon because of this.

To further rub salt into the wounds, a long laundry list of CIA exploits have been published by WikiLeaks http://www.bbc.com/news/technology-39221421 - these are mostly for Windows.  The point being that Windows already has enough holes and it doesn't need Microsoft deliberately adding more.

Irresponsible Use

At this point, all I can say is that using MS Windows for anything except maybe playing games, is irresponsible.

You should not risk using MS Windows for shopping, banking, business or government.  It is not suitable for that.

Here is a list of utilities that can be used to attempt to clean Windows 10 and try to prevent it from blabbing to all and sundry about everything you do with your computer.

ShutUp10

http://www.oo-software.com/en/shutup10

Techne

http://techne.alaya.net/?p=12499

DisableWinTracking

https://github.com/10se1ucgo/DisableWinTracking

BlockWindows

https://github.com/WindowsLies/BlockWindows

GWX Control Panel

http://blog.ultimateoutsider.com/2015/08/using-gwx-stopper-to-permanently-remove.html

Spybot Anti-Beacon

https://www.safer-networking.org/spybot-anti-beacon/

Evil Addresses

A list of evil hard coded addresses I found, that should be blocked in a router:
134.170.30.202
137.116.81.24
204.79.197.200
23.218.212.69
65.39.117.230
65.55.108.23

I have no idea what these addresses are and there may be many more.  These are hard coded connections in Windows, that cannot be blocked with a domain name server or hosts file.

Domain Names

My hosts file that I made after looking at packets with tcpdump:
127.0.0.1 dns.msftncsi.com
127.0.0.1 ipv6.msftncsi.com
127.0.0.1 win10.ipv6.microsoft.com
127.0.0.1 ipv6.msftncsi.com.edgesuite.net
127.0.0.1 a978.i6g1.akamai.net
127.0.0.1 win10.ipv6.microsoft.com.nsatc.net
127.0.0.1 en-us.appex-rf.msn.com
127.0.0.1 v10.vortex-win.data.microsoft.com
127.0.0.1 client.wns.windows.com
127.0.0.1 wildcard.appex-rf.msn.com.edgesuite.net
127.0.0.1 v10.vortex-win.data.metron.life.com.nsatc.net
127.0.0.1 wns.notify.windows.com.akadns.net
127.0.0.1 americas2.notify.windows.com.akadns.net
127.0.0.1 travel.tile.appex.bing.com
127.0.0.1 www.bing.com
127.0.0.1 any.edge.bing.com
127.0.0.1 fe3.delivery.mp.microsoft.com
127.0.0.1 fe3.delivery.dsp.mp.microsoft.com.nsatc.net
127.0.0.1 ssw.live.com
127.0.0.1 ssw.live.com.nsatc.net
127.0.0.1 login.live.com
127.0.0.1 login.live.com.nsatc.net
127.0.0.1 directory.services.live.com
127.0.0.1 directory.services.live.com.akadns.net
127.0.0.1 bl3302.storage.live.com
127.0.0.1 skyapi.live.net
127.0.0.1 bl3302geo.storage.dkyprod.akadns.net
127.0.0.1 skyapi.skyprod.akadns.net
127.0.0.1 skydrive.wns.windows.com
127.0.0.1 register.mesh.com
127.0.0.1 BN1WNS2011508.wns.windows.com
127.0.0.1 settings-win.data.microsoft.com
127.0.0.1 settings.data.glbdns2.microsoft.com
127.0.0.1 OneSettings-bn2.metron.live.com.nsatc.net
127.0.0.1 watson.telemetry.microsoft.com
127.0.0.1 watson.telemetry.microsoft.com.nsatc.net
127.0.0.1 win8.ipv6.microsoft.com
127.0.0.1 go.microsoft.com
127.0.0.1 windows.policies.live.net

How Many More?

If the above lists haven't convinced you to shun this super quality spying system, then consider that there may be many more leaks that we haven't found yet and as soon as Microsoft figures out that most holes are plugged by the above tools, then they are sure to add new ones to keep the data flowing - as evidenced by the recent rename and rework of their networked sniffing service.  It is bound to keep happening to throw off the defenders.

Therefore I still think that the only somewhat secure way to use Windows 7, 8 and 10 is in a virtual machine with networking disabled.  The trouble is that you cannot analyze encrypted tunnels with packet inspection in a router, so you have to unplug the network cable.

The only real solution is to use UNIX - buy a Mac, install Red Hat Linux Fedora or BSD, since securing MS Windows is a futile game of Whack a Mole.

In addition, Windows also has a low level exploit in the UEFI BIOS, that allows a perpetrator to install WPBT code that will run before the system starts up.  This was supposed to be used for an anti-theft system, but MS botched it.

Amusing use of WGA Remover

The most amusing fix, is to install Windows 7 and then run WGA Remover instead of registering it.
http://www.majorgeeks.com/files/details/wga_remover.html

Microsoft will then think that you are running an illegal copy and out of sheer spitefulness will not offer you any unsolicited Windows 10 bugs, adware and downgrades, but you will get security updates.  I recently tried it and it seems to keep Win7 Pro from morphing into Win7.10 Shopping Channel Edition.

I like this approach, since it reflects their attack upon our sensibilities right back to them.

A Simple Fix for the Windows 7 Black Screen Annoyance

You don't normally need to download any special tools to fix Windows Annoyance problems, but you need to be aware that there is a user higher than Administrator, called Local User.  So you need to take ownership of a couple of files in the system32 directory, then zap them.

Click Start, All Programs, Accessories, Right Click on Command Prompt and select Run As Administrator, then:

You can use slmgr to input the key if the machine is online or if you have a phone and the right number and depending on where you are in the world, that may be a tall order.

Zap a couple files in the system32 directory:C:\> cd \windows\system32

Take ownership of the sppcomapi.dll file and rename it:
C:\> takeown /f sppcomapi.dll
C:\> ren sppcomapi.dll sppcomapi.dll.bad

(If the above doesn't work due to the file being in use - restart and try again and if that doesn't work either, get the Microsoft 'handle' utility:
https://technet.microsoft.com/en-us/sysinternals/bb896655
Run 'handle filename' to list all matching handles, then 'handle -c id -p pid' to release it.)

Make an empty file to replace it:
C:\> nop > sppcomapi.dll

Take ownership of the slui.exe file and rename it:
C:\> takeown /f slui.exe
C:\> ren slui.exe slui.exe.bad

Make an empty file to replace it:
C:\> nop > slui.exe

Now the Windows Annoyance system cannot run.

Reboot Windows
Log in

Right click on desktop, personalize:
Reset the desktop background to something pretty.


Once you turned all the junk off, Windows starts up and runs significantly faster.


-. --- / .-- .. -. -.. --- .-- ... --..-- / -. --- / -.-. .-. -.--

Sigh...

Herman

Dropbear - Embedded SSH Daemonology

A Bear of Very Little Brain

The name Dropbear is intriguing since it makes me think of grizzlies and gummy bears. Real Aussies know that a Drop Bear is a carnivorous marsupial with a particular taste for foreign hikers.  I love sugar - who doesn't - but I should not eat it anymore.  I found that cinnamon makes a good substitute in most things, but I digress, this is not supposed to be a treatise on sugary treats or scary marsupials.

The Dropbear SSH daemon can be compiled with various options, but when one is faced with an existing system that cannot easily be changed, then one has to make do with what one got.
I was trying to download log files and video off an ARM based system and it took forever.  So I experimented with the SSH encryption and compression options to speed it up.  Since a typical embedded system has a dinky little processor, selecting a simpler encryption algorithm can make a huge difference.

AES vs Blowfish

The standard copy command "scp user@target:~/data ." ran at all of 6 Mbps.   I could see the grass growing, which is really special, since I live in a desert.

The default algorithm is AES256, which has special instructions on a X86 type processor to speed it up, but not on an embedded ARM based target.  When I tried Bruce Schneier's Blowfish "scp -c blowfish-cbc user@target:~/data ." it immediately ran at more than double the speed, clocking 13.5 Mbps.  It felt like flying compared to the previous.

Arcfour would run even faster, but Dropbear doesn't have it by default and some people are concerned that Arcfour is not secure anymore, though that is actually a Windows implementation problem.  IMHO Arcfour is not much worse than Blowfish - sorry Bruce...

Header Compression

I then tried header compression (the -C option) and it halved the speed again.  So this poor ARM processor really doesn't like the Zip algorithm either.

Process Control

Finally, I checked to see what the target processor was doing with 'top' and found that one running process was consistently sapping 25% of the processor power, so I thought I would hit the jackpot if I simply suspend that process while downloading.

A bit of remote job control using pidof and kill:
$ PID=$(ssh user@target "pidof -s processname")
$ ssh user@target "kill -SIGSTOP $PID"

Then I did my download test again and disappointingly found that the resulting speed-up was only 5% from 13.5 Mbps to 14 Mbps - where did the other 20 go?  Oh well, I'll take that little bit too thanks.

After the download one can resume the suspended task with:
$ ssh user@target "kill -SIGCONT $PID"

SSH Password Scripting with ssh-askpass

Another disappointment with this version of Dropbear was that it doesn't seem capable of public key authentication, only passwords and typing a password all the time gets boring really quickly, but OpenSSH is not particularly script friendly.

Fortunately there is a utility on Red Hat systems called ssh-askpass.  Users of Debian distributions will have to compile it from source, since it is not in the repositories, due to some misplaced concerns with protecting evil users of SSH against themselves.

Save your target password in a variable called SSHPASS, then use a command like this:
$ SSHPASS=password
$ ssh-askpass -e ssh user@target "remotecommand"

Zenity has a password entry dialogue that is useful for this type of problem.  Later in a script, I'd blank out the password so it doesn't hang around in memory too long, to salve my conscience.


La voila!

Herman

Nothing to Fear

Armand Jean du Plessis, Cardinal-Duke of Richelieu and Fronsac, 17th century Prime Minister of France, reputedly said something to the effect of:

"Give me six lines written by an honest man and I will find something in it to hang him with".

There are probably many things he said, that he didn't say, but it is a nice quote.

Beware of a Man in a Dress

The French encrypted communications up to the early 19th century, using simple ciphers known as petits chiffres. These were short notes, based on 50 numbers. Later, they began to write letters using a combination of 150 numbers, known as the Army of Portugal Code.  By 1812, new cipher tables were sent from Paris based on 1400 numbers and derived from a mid-18th century diplomatic code.

 18th Century Paris Cipher

Many people think that 'If you have nothing to hide, then you have nothing to fear', or simply state 'I have nothing to hide', meaning that they don't care about ubiquitous government surveillance.  Well, if you are that boring, then no-one would want to talk to you!

In reality, even the worst, boring, dullards try to keep their bank account, tax return, passport and credit card private and even Germans usually at least wear budgie smugglers when they go for a swim...

To me, the biggest problem is organized criminals who can drain a bank account in seconds, ruin your credit rating, sell your house while you are on holiday and state security agents acting like the infamous Cardinal, who can plant incriminating evidence on your IT systems to hang you with.  Just watch what is happening to Hillary Clinton and the Aussie PM who ran private email servers, which were legal at the time, but which are now being systematically stuffed with 'new discoveries' of 'classified information', much like the tiny mountain cabin of Ted Kazinsky, that was searched multiple times and each time delivered a treasure trove of new 'evidence' - there must have been a rather large cave under O'l Ted's shack.

Encrypt everything!

Encrypt your phone, your tablet, your laptop PC, your desktop PC, your email.  Buy a Black Phone.  Use the Red Phone.  Install RetroShare.  Use Free Software.

Use Keepass to save the passwords and keep the master password in your wallet if you don't trust your memory.

Do not make it easy for online miscreants hiding in their mother's basement on the other side of the world, to destroy your life.

Upon the death of the most revered Cardinal, Francois Marie Arouet (Voltaire), who wrote rather more than six lines criticizing the dear Cardinal said:

'He was a kind and generous man, 

provided of course, that he is really dead.'

Voltaire

Voltaire is really dead too, unfortunately.

Ecrasez l'Infame.

Herman

Reading and Parsing Data from Serial Ports

Read, Echo, Octal Dump, Head, Cat, Chat and Serial Ports

Anyone who tried to parse data from a serial port in a Bash script will have run into trouble really quickly.  This post explores a few different methods.

Some people prefer using minicom and expect, others just want to read a prompt from an embedded target such as an Arduino and send it a file without spending too much time on learning new tricks.  The best way to do it is the way that works for you!

Rather than fighting with an actual serial port (/dev/ttyUSB0), most of these examples use echo and pipes to send binary and ASCII data to a parsing utility to show what it does and how to use it.

In a nut shell, if you need to parse human readable ASCII data, use read.  If you have to parse unreadable binary data, use od or head.  If it has to be very fast, use cat.  Read has a built-in timeout that you can use to keep it from getting stuck waiting forever.  The others, you have to kill with a timer when something goes wrong.

When extensive error handling and timeouts are required also with ASCII data, use chat.  Chat is part of the pppd package and is usually installed by default.

Octal Dump and Head

This example uses echo to print out binary data to a pipe as a simulated serial port. Here shown with octal dump (od) to make the binary visible on screen:

$ echo -en "\x02\x05\x00\x01\x02\x0a\x0b\x0d\x0e" | od -tx1

0000000    02  05  00  01  02  0a  0b  0d  0e                            

0000011

Reading a number of data bytes with head works, but it doesn’t have a built-in timeout feature:

$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | head -c5 | od -tx1

0000000    40  41  00  42  01                                            

0000005

Reading a few data bytes directly with Octal Dump works, but it also has no built-in  timeout:

$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | od -N5 -tx1

0000000    40  41  00  42  01                                            

0000005

Read is best in a loop

Using read in a while loop on a mix of binary and ASCII data with od for debugging, shows the following funky behaviour:

#! /bin/bash

while read -t1 -n1 CHAR; do

echo $CHAR | od -tx1

done < <(echo -en "\x02\x05\x00\x41\x42\x02\x0d\x0a\x43")

0000000    02  0a                                                        

0000002

0000000    05  0a                                                        

0000002

0000000    0a                                                            

0000001

0000000    41  0a                                                        

0000002

0000000    42  0a                                                        

0000002

0000000    02  0a                                                        

0000002

0000000    0d  0a                                                        

0000002

0000000    0a                                                            

0000001

0000000    43  0a                                                        

0000002

So the 00H and 0AH gets absorbed as delimiters and a new 0AH added at the end of each token.

The 00H is especially bad to read and causes a reset from which it only recovers at the next 0AH, unless n=1.

Therefore, use od or dd or even head, to parse binary data and use read for human readable ASCII data.

It is important to put read in a loop, since it is very slow with opening the port, so you should not call read repeatedly - it will then likely drop characters.  In a while loop as above or below, it works better.

This works OK with read:

#! /bin/bash

while read -r -t1 ; do

echo $REPLY | od -tx1

echo $REPLY

done < <(echo -en "\x02\x05\x01NO CARRIER\x0d\x0aOK\x0d\x0a\x40”)

0000000    02  05  01  4e  4f  20  43  41  52  52  49  45  52  0d  0a    

0000017

NO CARRIER

0000000    4f  4b  0d  0a                                                

000000

OK

So read will parse ASCII tokens from garbage, provided that the garbage doesn’t contain 00H.

Sleep Timeouts

Here is an example to read binary data and put it in a file, with od and an error timeout:

#! /bin/bash

# Read 20 bytes with a 1 second error timeout

od -N20 -tx1 < <( echo -en "\x02\x05\x00\x01\x02\x03\x04\x0d\x0a" ) > /tmp/data.txt &

PID=$!

echo "PID=$PID"

sleep 1

kill $PID

cat /tmp/data.txt

0000000    02  05  00  01  02  03  04  0d  0a                            

0000011

In this case, od will either finish reading the data, or get killed  when the sleep times out, so your script will not hang if the device on the other side of the wire is dead.

Raw or Cooked Sushi

For working with actual serial ports, it is important to check whether to use use raw mode (don't wait for line terminators) or cooked mode (buffering on, wait for line end):

echo "Set serial port USB0 to 9600N81"
stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0 9600

Pussycat to the rescue

If the device under test is very fast and you experience dropped characters between a command and response, then you may need to use cooked mode with cat to read the port to a temporary file, then parse the file afterwards like this:

echo "Set serial port USB0 to 9600N81"
stty -F /dev/ttyUSB0 cooked
stty -F /dev/ttyUSB0 9600

FILE="/tmp/data.txt"
PORT="/dev/ttyUSB0"
 

echo -en "AT&V\r" > $PORT; cat < $PORT > $FILE &
PID=$!
sleep 1
kill $PID

if cat "$FILE" | grep "ERROR"; then
   let "ERRCNT+=1"
fi

The /tmp file system is a RAM disk, so it is much faster than writing to a hard disk.

If you need the received data in a Bash variable for further processing, do this:
DATA=$( cat $FILE )

Interactive Scripts with 'chat'

A typical exchange with a radio or other embedded device goes something like:

• Send a command
• Get a response
• If the response was good, then do an action
• If the response was bad, then quit

Automating that with echo, cat and if statements is difficult and most of the program may end up being error handlers.  The chat program is part of the pppd package and is probably installed by default.  You could use chat to do scripts more effectively than with standard bash commands.

http://www.tldp.org/HOWTO/PPP-HOWTO/x1219.htm

For example, receive ERROR, then abort, receive nothing then send at&v, receive OK then set register 9 to 2, or timeout after 3 seconds, in a one liner:

$ chat -v -s TIMEOUT 3 ABORT ERROR '' at&v OK ats9=2 </dev/ttyUSB0 >/dev/ttyUSB0 

With a few lines like the above in a script, you can do most anything with a simple radio modem device.

Serial Port Redirection with 'exec'

When playing with serial ports in scripts, you will find that each time you open and close the port, it takes time, so it can drop characters and the buffer contents become unknown also and may contain older junk making it hard to find what you are really looking for.   

Slow devices may be fine and dandy, but the same script with a faster device may never work right and leave you banging your head against the wall in frustration.

A simple trick to avoid opening and closing the port with each line of code, is to use the exec program to do permanent redirection of the stdio files.  After that, any program that reads and writes to stdio will talk to the serial port.  This is perhaps better explained with an example:

stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0 115200
DEV="/dev/ttyUSB0"
exec 4<&1 >"$DEV" <"$DEV"

File handle 0 is stdin, filehandle 1 is stdout and 2 is stderror, so 4<&1 means redirect stdout to a new handle 4 and the >"$DEV" means simultaneously also redirect it to the USB serial port and lastly, <"$DEV" means redirect the USB serial port to stdin.

From then on, your chat script doesn't have to add <"$DEV" >"$DEV" to the end of every line, the serial port will remain open for the duration of the whole script.  Having duplicate handles is useful to still be able to access the screen and keyboard for a user response:

DEV="/dev/ttyUSB0"
exec 3<&0 4<&1 >"$DEV" <"$DEV"
chat -v -s TIMEOUT 3 ABORT ERROR '' at&v OK ats9=2
echo "Press any key" >&4
read -u 3 -n 1 RESPONSE

You could also call chat inside an if statement for improved error handling:
if chat -v -s TIMEOUT 3 ABORT ERROR '' at&v OK ats9=2; then
   echo "Error S9" >&2
   exit 1
fi
echo "done" >&4

More explanations on exec redirection here: http://wiki.bash-hackers.org/howto/redirection_tutorial

Serial Port Tips

www.aeronetworks.ca/2015/01/serial-port-io.html
www.aeronetworks.ca/2014/10/serial-ports-revisited.html
www.aeronetworks.ca/2014/01/crcs-and-serial-ports.html
www.aeronetworks.ca/2013/10/serial-port-tricks.html
www.aeronetworks.ca/2013/05/usb-serial-device-with-unknown-ids.html
www.aeronetworks.ca/2015/10/reading-and-parsing-data-from-serial.html
www.aeronetworks.ca/2013/05/compile-moxa-serial-widget-device.html

La voila!

Herman

Zenity Progress Dialogue

Zenity is a nice and simple way to prettify Bash scripts, similar to kdialog, xdialog and a few others, but on the whole I think the best of the lot.  Though there is nothing preventing you from mixing them up if one has a feature that would be more useful for something.

I started using Zenity years ago, when there wasn't much documentation and consequently didn't use the progress dialogue the right way.  The result being that some scripts eventually broke.  For example, there was no information on how to make the progress bar progress - so I only used the whirr-whirr pulsate effect.

No Progress

In the past, I simply started a pulsater, pushed it into the background, saved the PID and later killed it when necessary, like this:

zenity --progress &
PID=$!
echo "Do something"
kill $PID

That used to worked fine, but recently on Fedora 22, it just sits there and does nothing.  It won't even blink.  Dead as a doornail.  Bah, humbug...

When All Else Fails...

Eventually, I found the documentation and RTFMed:
https://help.gnome.org/users/zenity/stable/progress.html.en

The problem is that the progress dialog has a new bug and it expects to receive something over stdin in order to get going.  If it never gets anything, it does nothing - in previous versions it worked fine in the background, but no-more.  When it is pushed into the background with &, it just sits there and sulks.

The correct way to make a progress pulsater that must do something and exit based on a condition, at which point one has to stop the pulsater, is like this:
 
CNT=0
(
while TRUE, do
  echo "This Will Do Nothing"
  echo "#Change The Text"
  sleep 5
  let "CNT+=1"

  if [ "$CNT" -eq 10 ]; then
    # Close the progress dialog with 100%
    echo "100"
    exit 0
  fi

  echo "Continue doing nothing"
done
) | zenity --progress \
--width=350 \
--title="The Title"
--text="The Default Text" \
--no-cancel \
--auto-close \
--pulsate

The --auto-close parameter will cause the progress dialogue to stop when you echo "100" percent.

An echo statement that starts with a # will change the text and one without, will be ignored.  The --no-cancel will suppress the Cancel button, since it doesn't make sense.

La voila!

Version Control For Scripts

Running a full blown version control system such as subversion or CVS for Bash scripts is a bit of overkill.  Most people don't use any version control at all on scripts and revert back to the ancient way of saving multiple copies of the files.

The inevitable result is a mess of files called script1, script2, script.ok, script.bad... and then after a few days, you cannot remember which is which.

Fortunately, there is an ancient, light weight versioning system that is very easy to set up and which is perfect for the single user case: RCS.

Setup RCS

It is probably already installed (try whereis rcs), otherwise do:
# yum install rcs

In your script directory, make a directory to keep the archives:
$ mkdir RCS

Edit each script and add the $Id$ keyword to the top, right underneath the familiar #! /bin/bash command:
#! /bin/bash
#$Id$

That will substitute your name, date, time and version number of the file upon checkout, so you always know which version of the script you got.

Check In

As soon as you made an important change, check the file in, with ci:
$ ci scriptname

Type a proper description of the change, it is for your own benefit, when you have to look at a problem, possibly years later.

The file will disappear!

Check Out

Check it out again with co:
$ co scriptname 

By default you get the latest version.  You can also check out older versions using the -r parameter and switch between versions to see which one works best, without losing anything, since they are all in the archive file.

Important Tricks

If you are the only person working on the scripts, then remove strict locking.  This will make your life much easier:
$ rcs -U *

Now you can check files in with the -u option, so they don't disappear and gets checked out again immediately:
$ ci -u scriptname
 
You can also set an alias (add to the bottom of ~/.bashrc), to save you some more typing, so that you never have to use the co command again, just ci filename:
$ alias ci='ci -u'

That is pretty much all there is to it.

Do read the man pages of ci, co and rcs for details on what to do when the inevitable happens and you want to look at the logs and revert to an older version of a script.

If It Is Free, Take Two... Aspirin

Updates - Untracking Tools

I was wondering when privacy enhancing utilities would be come available. Here is the first one:
http://www.oo-software.com/en/shutup10

the second one:
http://techne.alaya.net/?p=12499

and the third one:
https://github.com/10se1ucgo/DisableWinTracking

I still think that the only secure way to use Windows 7, 8 and 10 is in a virtual machine with networking disabled.  The trouble is that you cannot stop encrypted tunnels with deep packet inspection in a router, so you have to unplug the network cable so to speak.

The only real solution is UNIX - buy a Mac, install Linux Fedora or PC-BSD.  Anything else is futile.

Free Candy

There are many an old proverb about free stuff, but the latest action by Microsoft takes the cake.

Even people who declined the free Windows 10 upgrade, will now also have this bloated piggy forced down their throats: http://www.theregister.co.uk/2015/09/10/windows_10_forced_download/

It is one thing sending out free security patches and feature upgrades, but a DVD sized download will cause lots of trouble for thousands of users.  Many a hard disk doesn't have that much free space and many users are on bandwidth limited networks and may get hefty overage bills, for something they did not ask for.

On the other hand, what MS may be finally admitting to, is that Windows 8 is so bad, that the only solution is to replace the whole rotten thing - for free.

Free Insecurity

To add insult to injury, Windows 10 has very serious security issues, which I have investigated a bit with tcpdump and listed in another post: http://www.aeronetworks.ca/2015/02/windows-10-on-virtualbox.html

At this point in time, I cannot recommend to anyone to use Win10.  It is not worth the security risk.

Also, there are complaints that some of the new spyware features are also being pushed out to Windows 7 and 8.

Eventually, there will be some trustworthy third party firewall programs that will make it easy to strangle it all, but until then, Windows 7, 8 and 10 should only be used behind a separate draconian firewall device.  Using a vanilla Win10 laptop PC in a coffee shop or airport would be a disaster waiting to happen, like WinXP SP1 was.

A major problem with 3rd party firewalls though, is that I have never seen a proper review done by someone who actually tested it with a packet sniffer to see if it really works, since most reviewers are ordinary journalists with little or no technical knowledge, who don't know the first thing about tcpdump or wireshark.

The Real Problem: Dirty Tricks

I am not really worried about marketing data sent to Microsoft.  I am worried about what else gets sent in and out and who else is listening while the packets travel half way around the world.  Any government security agency with half a clue can plant incriminating data on Windows computer systems.  If MS can download a whole DVD worth of junkware to half the desktop machines in the world, then how hard could it be for the NSA or FSB to plant child porn, terrorism related junk, or classified data on a machine and then hit the owner with a long jail sentence?

If you were wondering about dirty tricks, have a look at what is happening to Hillary Clinton.  The US authorities has retroactively classified data that wasn't classified at the time.  Now, they can charge her and the previous administrator of her mail server and put both of them in the clink.  The only thing saving Hillary, is the fact that her husband is an ex president, but if I were her mail server administrator, then I'd move to another country, far, far away.

From Soft In The Middle, To Soft In The Head

What on earth were MS thinking?


-. --- / .-- .. -. -.. --- .-- ... --..-- / -. --- / -.-. .-. -.--

Sigh...

Herman

UAE Visa Renewal and Car Selling Red Tape

Latest Visa Renewal Process in Al Ain - 2020

Go to SEHA Immigrant Screening Centre opposite Al Ain Hospital.
Hours: 7 am till 3 pm, Sun - Thu

You have three choices:

• Go with the enormous crowd gathered under the palm trees...
• Take the Fast Track - limited numbers - be there early
• VIP service

Hmm...

VIP Medical Service

• Go in the main front entrance and go upstairs. 
• Walk through the Fast Track men's waiting room to the VIP waiting section.  
• There is a tiny little VIP sign above the door.
• There will be almost nobody, so the service is immediate.
• You'll be done in about 20 minutes.

• You need: Passport and passport copies (first 4 pages plus visa)
• Cost: Dh500 (that is why there is almost nobody, but it is worth it)
  
• In before 10h00, get results by 2 pm the same day (someone else may pick up with the receipt)
• You need to get a blood test (no fasting) and a chest X-ray
• They will send you a SMS when they are done around 1 pm - be there at 2 pm.

More info here: http://www.capitalhealth.ae/faqs/

Typing Office - Emirates ID

• There is a Typing Office (Registry shop) to the East of Jimmy Mall, also in the Municipal building.
• Renew your Emirates ID for Dh370
• You will get another SMS if it worked in a few minutes
• Take the form to the Emirates ID building and have it stamped at reception - they don't even check it.

New Visa

• Now your employer has 10 days to get you a new visa, after which you can collect you new ID at the Post Office.
• You have to give your employer your medical report, stamped Emirates ID form (Emiratis love stamps), a copy of your medical insurance card, four photos and your passport, and hope you get it back... 

 

Selling Your Old/Cheap Car - 2020

Most expats buy a cheap car, since they don't know how long they will stay in the UAE.  I ended up staying 11 years, but all good things eventually comes to an end and then you sit with one or two cars, that you don't quite know what to do with.  Some people abandon their car at the airport and fly off, but that will result in a 3000 Dirham fine that will hit you next time you fly through Dubai - so don't do that.

If you happen to live in Al Ain and you want to get your old/cheap car sold with the least amount of hassle, give Munaver Jaman a call at 055 499 4045.

The big difference is that he will come to you.

The only thing you need to do is go to the traffic police and tell them you want to sell your car and sign the transfer form.

After that, Munaver does everything between the seller, buyer and traffic police, online with his phone (some police confirmation codes are exchanged between everyone's smart phones to make it work) and he charges the buyer a small fee for the whole rigmarole.

So I can recommend this guy, he is quiet and sharp and knows what he is doing.

 

Closing Your Etisalat Accounts

Dealing with Etisalat (like any telecom company the world over!) is a total nightmare.  The only thing worse, is trying to call Revenue Canada from overseas.  Etisalat has numerous kiosks/shops in every major town, with a large crowd of people patiently waiting outside.  There is a ticket system, but you need to get in first, to get the ticket...

 

You can call them (101 for voice response, or 800 6665 for a real agent), but then you end up in an interminable queue with boring music on hold.  The queue can time out after an hour or two while you are on hold, so then you have to start all over again.  Whichever way you do it, it will be a day in your life that will never be refunded.

So, as with all telecom companies, the lesson is to never, never, never, set up automatic bill payment.  It is better to pay the bill manually every month, or better yet, pre-pay.  That way, when you want to leave, you can just leave and don't have to cancel anything - they will eventually figure out that the account is inactive - maybe.

Closing Your Bank Account

Your bank will have a minimum balance indicated in your terms of service.  For FAB, the minimum is 3000 Dirhams.  When the account reaches zero, the bank will automatically close the account.

So there are two ways to go about it:

• Use your smartphone and transfer everything you can to your home country, but then Dh3000 ($817) would likely end up as a gift to the bank - ouch...
• Go to the bank and wait in line to talk to an agent, transfer every last drop and close the account.

It will help if you don't have a credit card, or any automatic bill payments, otherwise you may lose another day of your life.

 

La voila!

Herman

Audio Networking with Sox and Netcat

Sox + Netcat = VoIP

The idea is to hook the sound devices on two machines together so that there is a bidirectional intercom between them.  This way, you can have an amateur SSB radio hooked to a distant computer in your radio shack and a headset on a local computer in your living room and do VoIP with the remote radio transceiver over your LAN.

I noticed that there is a steady stream of people reading this post.  Please note that Sox with CVSD and netcat will work fine and consume very little processing time on a tiny ARM embedded system, but gstreamer with raw audio over UDP may be a better choice if you have a half decent system on both ends of the link and want the best quality audio.  Therefore, do look into gstreamer also.

For a remote ham radio, the main thing missing, is the PTT switch, which one can do with another netcat proxy to the serial control port of the transceiver and radio control software running locally to set the channel and key the radio.

For streaming, one needs a headerless, self synchronizing CODEC protocol, to connect to a running stream.  Examples are ADPCM, CVSD and LPC10.

Sox gotchas

Sound Exchange has three programs: sox, rec and play.  The difference is that if you run sox, it will glom onto the sound device for both read and write, but it can only do one thing at a time, so it can either record or play, but not both at the same time.

The rec and play programs however, can run independently and concurrently, so those can be used to make a bidirectional intercom.

UNIX systems have many different sound systems: OSS, ALSA, Coreaudio, Pulseaudio, Jack...  Sox knows how to handle them, if you let it use the default settings.  So get the default to work first with a simple mixer application such as aumix, rec and play, before you try something complicated.

Sox also has internal buffering that is extremely large: 8 kilobytes.  The result is that sox by default has a huge delay.  When you use compression, then the more you compress, the longer the delay gets, because the slower stream takes longer to fill the buffers.  You can reduce that with the --buffer parameter.

In general, a simpler sound system such as ALSA or OSS will have smaller buffers and lower delays than the complex pulseaudio or coreaudio, so stick to the basics for best results.

The general syntax is: sox inputspec inputfile outputspec outputfile

A lone "-" instead of a file, tell sox to use stdio or a pipe, so read the examples carefully.  The data flow is from left to right in all the below examples.

Default setup

First get sox to work on the default sound device:

Make a noise:
$ cat /dev/urandom | sox -traw -r44100 -b16 -eunsigned-integer - -d

Play a tone:
$ play -n synth 10 sin 500

Record:
$ rec -r8000 -tcvsd file

Play:
$ play -r8000 -tcvsd file

Don't bother with the below if you cannot get the above to work.

Hook sox to netcat on two computers:

Assuming IP addresses 192.168.1.1 and 192.168.1.2.

With CVSD, be sure to set the sample rate and reduce the buffer size to reduce the delays.  The resulting network bandwidth is about 1 kilobyte per second and the end to end delay is about 1 second.  CVSD is a very simple CODEC, so it works well on an embedded processor.

On the first computer:
$ rec —buffer 32 -tcvsd -r8000 - | nc -u -l 5555 | play —buffer 32 -tcvsd -r8000 -

and on the second computer:
$ rec —buffer 32 -tcvsd -r8000 - | nc -u 192.168.1.1 5555 | play —buffer 32 -tcvsd -r8000 -

Just two one liners!

The above works and I had it running between a Macbook Pro with OSX (Mac sox is available from Homebrew) and a Fedora Linux laptop.  I also experimented on a BSD laptop and it behaves the same.

The main problem is with delays due to buffering and the more one compresses the data stream, the longer it takes to fill the pipes, hence longer delays.  The LPC10 CODEC will reduce the network bandwidth further, but the delay will be ridiculous.

To make it robust, you may want to put a while loop with a sleep delay around the above, so that if sox or netcat would exit for whatever reason, it would start again, after waiting a little bit for the dust to settle:

$ while true; do rec —buffer 32 -tcvsd -r8000 - | nc -u 192.168.1.1 5555 | play —buffer 32 -tcvsd -r8000 -; sleep 1; done

...and it is still a one liner!  OK, technically, it is three.

It took me about 3 days to figure the above out, so when all else fails, RTFM and try again.

It is certainly much better than two paper cups and string...

La voila!

Herman

Mac Backups With Rsync to Encrypted Flash Media

Backup Woes

The main advantage of using a Mac is that it defaults to idiot mode and the main annoyance with a Mac is that it defaults to idiot mode, but underneath it all there is BSD...

The Mac Time Machine backup utility is one such idiot application which is probably fine for a large number of people,  but which doesn't fit my usage scenario.  Coupled with the Disk Utility which makes it hard to format a Flash Memory Device properly, it is even worse.

I want to make backups to USB and SD memory devices, since they are small, solid state and convenient.  However, to make a proper backup, the device should be formatted with HFS+ and it must be Encrypted.  The Disk Utility defaults to the prehistoric FAT in order to support file exchange with legacy Windows machines and FAT doesn't support proper file ownership and permissions.

Also, it is rather self defeating if your HDD is encrypted, while your backups are in plain text on a FAT disk.

Format a Removable Disk with HFS+

• Run Disk Utility
• Select ‘1 Partition’ from the Volume Scheme menu.
• Enter a name eg MACBACKUP1
• Select Mac OS X Case Sensitive Extended (Journaled) from the Format menu.
• Click the ‘Options’ button and select ‘GUID Partition Table’ 

 Enable File Ownership Data

• Right click the flash drive and select ‘Get Info’.
• Click the lock icon in the bottom right corner.
• Remove the check mark from ‘Ignore ownership on this volume.’

Encrypt the Flash Disk

• Right click the disk icon and select Encrypt.
• Generate and save your password in KeepassX.

Backup with Rsync

Now finally, the media is ready to be used for a secure backup.  Time Machine works if the media is formatted this way, but it defaults to backing up everything and a Flash Disk is usually too small to backup everything.  Also, there is no point in backing up the whole system, since one can download a new system from Apple.  I only need to backup my Documents folder - everything else is expendable.

Make a file called /Volumes/MACBACKUP1/sync
Make a directory called /Volumes/MACBACKUP1/bak

Edit the sync script with vi:

#! /bin/bash
rsync -avz --progress --delete --max-delete=10 --max-size=20M ~/Documents /Volumes/MACBACKUP1/bak

Make it executable with chmod +x sync

By selecting a max-size of 20 Megabytes, I exclude things like large archives and ISO images.  Deletions are limited to ten files, in order to prevent a possible disastrous delete of the whole backup set, following a local delete disaster.  The result is a 51 GB backup set, which fits on a 64 GB memory schtick - nice.

Now, you can insert the backup device and click the sync script to backup whatever should be backed up to that device and thereby backup Documents to a USB schtick and backup Pictures to a SD card for example, while having it all secure and encrypted, in case the flash thingy gets lost or stolen or copied by the NSA/CSIS/GCHQ/FSB...

La voila!

Herman

An Angel Dancing On a Pin Head

Magic Eye Indicators

As a little child, I found the green angel wings indicator on my parent's Philips radiogram fascinating.  I sat and watched and wondered how it worked each time we played a record.

So, let's add a little dancing angel to my 20 Watt audio guitar amplifier:
http://www.aeronetworks.ca/2015/02/cool-amplifier.html

Why the blazes does it take three obscure keys (Cmd, Shift, 4) to do a screen grab on a Mac?  
Whoever thought that up needs to get three lashes with a wet noodle.

You can get your own dancing angel from Tubes Store in Chelyabinsk:
http://tubes-store.com/product_info.php?cPath=31_59&products_id=5

Basically, it is a miniature cathode ray tube with a curved zinc sulphide fluorescent screen.  The electron beam is deflected by an electrostatic target in front of the screen, powered by a built-in triode amplifier, so everything you need (minus the PSU) comes in one convenient glass package.  Some indicators are viewed end on making a cat's eye circle, others from the side, like this one.  If you already have a thermionic valve amplifier, then hooking a volume indicator up is easy.

As always with any electronic device, the biggest problem is finding the pin-out.  It has the usual triode valve elements: Heater, Cathode, Anode and Gate, but with an additional Screen, behind a Target (deflector).  So you can use the triode part as an amplifier to power the target, which controls the size of the shadow and width of the wings, thus making quivering angel wings on the screen.

The 6E1P / EM80 is super simple, since the Target and Anode are connected together internally, so you don't have to:

• Pin 1: Gate
• Pin 2: Cathode
• Pins 4, 5: Heater
• Pin 7: Anode
• Pin 9: Screen 

Viewed from the bottom pin side, the pins are numbered clock wise, starting at the gap on the right.

These devices used to be manufactured in Martin in Slovakia and in Moskow in Russia and were licensed copies of the Philips tubes from the Netherlands.  Contrary to Old Skool American propaganda about the 'iron curtain', there was a lot of co-operation and trade between Western Europe, Central Europe and the Soviets - same as today.  My father in law was a Slovakian truck driver and crossed the fortified border to Austria regularly.  There really wasn't much difference between the villages on either side of the border and everyone watched the same TV channels, but blue jeans were in short supply.  Any Slovakian teenager with a pair of jeans was 'waaay cool' and my wife can still remember how she showed off when she got her first pair.

Unfortunately, I could not see any clear indication of exactly where my angel tubes came from, but the Russian lettering shows it was one of these two factories, since Philips would not print their tubes in Cyrillic.

Here is an amusing old Rusky circuit:

As with all tubes, the voltages are not very critical.  A tube can run on anything from 100V for a miniature, to 250V DC for a big fella.   Somewhere in the middle at about 150V is perfectly fine.  Similarly, the heater is rated at 6.3V AC and anything between 6 and 7 V RMS is OK.  So don't worry much about the power supplies - if it works, it works...

The thing to bear in mid with tubes, is that the gate operates at a voltage negative with respect to the cathode.  The envelope detector diodes are oriented such as they are, to create a negative control voltage.  If you put them the other way around, then the circuit won't work. You can also put a trimmer and bypass capacitor on the cathode, so you can adjust the DC bias and get the wings to close.

To make you little angel dance, simply put an AM envelope detector on the Gate, heat the Cathode, Bias the Anode, keep your fingers away from the high voltage parts and Bob's your uncle!

BTW, it may be a good idea to put a rubber mat on the floor in your workshop.

Zzzappp! Owww!!! 

More information here: http://www.aeronetworks.ca/2016/08/audio-vu-meter.html

Have fun,

Herman