Wednesday, November 18, 2015

Compile The Latest ffplay From Source

Compile latest version of ffplay from source

Note that this script doesn't overwrite the existing ffmpeg installation:

The gotcha in the above guide is the SDL-devel package without which ffplay will not build.

#! /bin/bash
yum install autoconf automake cmake freetype-devel gcc gcc-c++ git libtool make mercurial nasm pkgconfig zlib-devel SDL-devel

mkdir ~/ffmpeg_sources

cd ~/ffmpeg_sources
git clone --depth 1 git://
cd yasm
autoreconf -fiv
./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin"
make install
make distclean

cd ~/ffmpeg_sources
git clone --depth 1 git://
cd x264
PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin" --enable-static
make install
make distclean

cd ~/ffmpeg_sources
hg clone
cd ~/ffmpeg_sources/x265/build/linux
cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="$HOME/ffmpeg_build" -DENABLE_SHARED:bool=off ../../source
make install

cd ~/ffmpeg_sources
git clone --depth 1 git://
cd ffmpeg
PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/ffmpeg_build" --extra-cflags="-I$HOME/ffmpeg_build/include" --extra-ldflags="-L$HOME/ffmpeg_build/lib" --bindir="$HOME/bin" --pkg-config-flags="--static" --enable-gpl --enable-libx264 --enable-libx265 --enable-ffplay
make install

cd /usr/lib
ln -s /home/username/bin/ffplay fffplay

Play Video With Low Latency Using ffplay

This is the fastest I can make ffplay:
ffplay -threads 2 -flags low_delay -fflags nobuffer -rtbufsize 32768 -fast -probesize 800000 -analyzeduration 800000 -ss 1 -framerate 50 udp://

Play Video With ffmpeg

This also quite fast, just to show that you can play video directly with ffmpeg using SDL output:
ffmpeg -i udp:// -f sdl -

La Voila!


Friday, November 13, 2015

Windows Insanity

Rusted Sieve

Windows 10 is about as secure as a rusted sieve.

Examples of data we collect include your name, email address, preferences and interests; browsing, search and file history; phone call and SMS data; device configuration and sensor data; and application usage.

Recently, the Diagnostics Tracking Service (DiagTrack) service was renamed to the Connected User Experiences and Telemetry service.  Sigh...

Here is a list of utilities that can be used to clean Windows 10 and try to prevent it from blabbing to all and sundry about everything you do.




Evil Addresses

A list of evil hard coded addresses that should be blocked in a router:

Domain Names

My hosts file that I made after looking at packets with tcpdump:

How Many More?

If the above lists haven't convinced you to shun this super quality spying system, then consider that there may be many more leaks that we haven't found yet and as soon as Microsoft figures out that most holes are plugged they are sure to add new ones to keep the data coming - as evidenced by the recent rename and rework of their networked sniffing service.  It is bound to keep happening to throw off the defenders.

Therefore I still think that the only secure way to use Windows 7, 8 and 10 is in a virtual machine with networking disabled.  The trouble is that you cannot stop encrypted tunnels with deep packet inspection in a router, so you have to unplug the network cable so to speak.

The only real solution is UNIX - buy a Mac, install Linux Fedora or PC-BSD.  Anything else is futile.



Tuesday, November 10, 2015

Dropbear - Embedded SSH Daemonology

A Bear of Very Little Brain

The name Dropbear is intriguing since it makes me think of grizzlies and gummy bears.  I love sugar - who doesn't - but I should not eat it anymore.  I found that cinnamon makes a good substitute in most things, but I digress, this is not supposed to be a treatise on sugary treats.

The Dropbear SSH daemon can be compiled with various options, but when one is faced with an existing system that cannot easily be changed, then one has to make do with what one got.

I was trying to download log files and video off an ARM based system and it took forever.  So I experimented with the SSH encryption and compression options to speed it up.  Since a typical embedded system has a dinky little processor, selecting a simpler encryption algorithm can make a huge difference.

AES vs Blowfish

The standard copy command "scp user@target:~/data ." ran at all of 6 Mbps.   I could see the grass growing, which is really special, since I live in a desert.

The default algorithm is AES256, which has special instructions on a X86 type processor to speed it up, but not on an embedded ARM based target.  When I tried Bruce Schneier's Blowfish "scp -c blowfish-cbc user@target:~/data ." it immediately ran at more than double the speed, clocking 13.5 Mbps.  It felt like flying compared to the previous.

Arcfour would run even faster, but Dropbear doesn't have it by default and some people are concerned that Arcfour is not secure anymore, though that is actually a Windows implementation problem.  IMHO Arcfour is not much worse than Blowfish - sorry Bruce...

Header Compression

I then tried header compression (the -C option) and it halved the speed again.  So this poor ARM processor really doesn't like the Zip algorithm either.

Process Control

Finally, I checked to see what the target processor was doing with 'top' and found that one running process was consistently sapping 25% of the processor power, so I thought I would hit the jackpot if I simply suspend that process while downloading.

A bit of remote job control using pidof and kill:
$ PID=$(ssh user@target "pidof -s processname")
$ ssh user@target "kill -SIGSTOP $PID"

Then I did my download test again and disappointingly found that the resulting speed-up was only 5% from 13.5 Mbps to 14 Mbps - where did the other 20 go?  Oh well, I'll take that little bit too thanks.

After the download one can resume the suspended task with:
$ ssh user@target "kill -SIGCONT $PID"

SSH Password Scripting with ssh-askpass

Another disappointment with this version of Dropbear was that it doesn't seem capable of public key authentication, only passwords and typing a password all the time gets boring really quickly, but OpenSSH is not particularly script friendly.

Fortunately there is a utility on Red Hat systems called ssh-askpass.  Users of Debian distributions will have to compile it from source, since it is not in the repositories, due to some misplaced concerns with protecting evil users of SSH against themselves.

Save your target password in a variable called SSHPASS, then use a command like this:
$ SSHPASS=password
$ ssh-askpass -e ssh user@target "remotecommand"

Zenity has a password entry dialogue that is useful for this type of problem.  Later in a script, I'd blank out the password so it doesn't hang around in memory too long, to salve my conscience.

La voila!


Saturday, October 24, 2015

Nothing to Fear

Armand Jean du Plessis, Cardinal-Duke of Richelieu and Fronsac, 17th century Prime Minister of France, reputedly said something to the effect of:

"Give me six lines written by an honest man and I will find something in it to hang him with".

There are probably many things he said, that he didn't say, but it is a nice quote.

Beware of a Man in a Dress

The French encrypted communications up to the early 19th century, using simple ciphers known as petits chiffres. These were short notes, based on 50 numbers. Later, they began to write letters using a combination of 150 numbers, known as the Army of Portugal Code.  By 1812, new cipher tables were sent from Paris based on 1400 numbers and derived from a mid-18th century diplomatic code.

 18th Century Paris Cipher

Many people think that 'If you have nothing to hide, then you have nothing to fear', or simply state 'I have nothing to hide', meaning that they don't care about ubiquitous government surveillance.  Well, if you are that boring, then no-one would want to talk to you!

In reality, even the worst, boring, dullards try to keep their bank account, tax return, passport and credit card private and even Germans usually at least wear budgie smugglers when they go for a swim...

To me, the biggest problem is organized criminals who can drain a bank account in seconds, ruin your credit rating and state security agents acting like the infamous Cardinal, who can plant incriminating evidence on your IT systems to hang you with.  Just watch what is happening to Hillary Clinton and the Aussie PM who ran private email servers, which were legal at the time, but which are now being systematically stuffed with 'new discoveries' of 'classified information', much like the tiny mountain cabin of Ted Kazinsky, that was searched multiple times and each time delivered a treasure trove of new 'evidence' - there must have been a rather large cave under O'l Ted's shack.

Encrypt everything!

Encrypt your phone, your tablet, your laptop PC, your desktop PC, your email.  Buy a Black Phone.  Use the Red Phone.  Install RetroShare.  Use Free Software.

Use Keepass to save the passwords and keep the master password in your wallet if you don't trust your memory.

Do not make it easy for online miscreants hiding in their mother's basement on the other side of the world, to destroy your life.

Upon the death of the most revered Cardinal, Francois Marie Arouet (Voltaire), who wrote rather more than six lines criticizing the dear Cardinal said:

'He was a kind and generous man, 
provided of course, that he is really dead.'

Ecrasez l'Infame.


Sunday, October 4, 2015

Reading and Parsing Data from Serial Ports

Read, Echo, Octal Dump, Head, Cat and Serial Ports

Anyone who tried to parse data from a serial port in a Bash script will have run into trouble really quickly.  This post explores a few different methods.

Some people prefer using minicom and expect, others just want to read a prompt from an embedded target such as an Arduino and send it a file without spending too much time on learning new tricks.  The best way to do it is the way that works for you!

Rather than fighting with an actual serial port (/dev/ttyUSB0), most of these examples use echo and pipes to send binary and ASCII data to a parsing utility to show what it does and how to use it.

In a nut shell, if you need to parse human readable ASCII data, use read.  If you have to parse unreadable binary data, use od or head.  If it has to be very fast, use cat.  Read has a built-in timeout that you can use to keep it from getting stuck waiting forever.  The others, you have to kill with a timer when something goes wrong.

Octal Dump and Head

This example uses echo to print out binary data to a pipe as a simulated serial port. Here shown with octal dump (od) to make the binary visible on screen:
$ echo -en "\x02\x05\x00\x01\x02\x0a\x0b\x0d\x0e" | od -tx1

0000000    02  05  00  01  02  0a  0b  0d  0e                            


Reading a number of data bytes with head works, but it doesn’t have a built-in timeout feature:

$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | head -c5 | od -tx1

0000000    40  41  00  42  01                                            


Reading a few data bytes directly with Octal Dump works, but it also has no built-in  timeout:
$ echo -en "\x40\x41\x00\x42\x01\x02\x0a\x0b\x0d\x0e\x0f\x00\x01\x02\x03\x41" | od -N5 -tx1

0000000    40  41  00  42  01                                            

Read is best in a loop

Using read in a while loop on a mix of binary and ASCII data with od for debugging, shows the following funky behaviour:
#! /bin/bash

while read -t1 -n1 CHAR; do

echo $CHAR | od -tx1

done < <(echo -en "\x02\x05\x00\x41\x42\x02\x0d\x0a\x43")

0000000    02  0a                                                        


0000000    05  0a                                                        


0000000    0a                                                            


0000000    41  0a                                                        


0000000    42  0a                                                        


0000000    02  0a                                                        


0000000    0d  0a                                                        


0000000    0a                                                            


0000000    43  0a                                                        


So the 00H and 0AH gets absorbed as delimiters and a new 0AH added at the end of each token.

The 00H is especially bad to read and causes a reset from which it only recovers at the next 0AH, unless n=1.

Therefore, use od or dd or even head, to parse binary data and use read for human readable ASCII data.

It is important to put read in a loop, since it is very slow with opening the port, so you should not call read repeatedly - it will then likely drop characters.  In a while loop as above or below, it works better.

This works OK with read:
#! /bin/bash

while read -r -t1 ; do

echo $REPLY | od -tx1

echo $REPLY

done < <(echo -en "\x02\x05\x01NO CARRIER\x0d\x0aOK\x0d\x0a\x40”)

0000000    02  05  01  4e  4f  20  43  41  52  52  49  45  52  0d  0a    



0000000    4f  4b  0d  0a                                                



So read will parse ASCII tokens from garbage, provided that the garbage doesn’t contain 00H.

Sleep Timeouts

Here is an example to read binary data and put it in a file, with od and an error timeout:
#! /bin/bash

# Read 20 bytes with a 1 second error timeout

od -N20 -tx1 < <( echo -en "\x02\x05\x00\x01\x02\x03\x04\x0d\x0a" ) > /tmp/data.txt &


echo "PID=$PID"

sleep 1

kill $PID

cat /tmp/data.txt

0000000    02  05  00  01  02  03  04  0d  0a                            


In this case, od will either finish reading the data, or get killed  when the sleep times out, so your script will not hang if the device on the other side of the wire is dead.

Raw or Cooked Sushi

For working with actual serial ports, it is important to check whether to use use raw mode (turn buffering off) or cooked mode (buffering on):
echo "Set serial port USB0 to 9600N81"
stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0 9600

Pussycat to the rescue

If the device under test is very fast and you experience dropped characters between a command and response, then you may need to use cooked mode with cat to read the port to a temporary file, then parse the file afterwards like this:

echo "Set serial port USB0 to 9600N81"
stty -F /dev/ttyUSB0 cooked
stty -F /dev/ttyUSB0 9600

echo -en "AT&V\r" > $PORT; cat < $PORT > $FILE &
sleep 1
kill $PID

if cat "$FILE" | grep "ERROR"; then
   let "ERRCNT+=1"

The /tmp file system is a RAM disk, so it is much faster than writing to a hard disk.

If you need the received data in a Bash variable for further processing, do this:
DATA=$( cat $FILE )

La voila!


Monday, September 21, 2015

Zenity Progress Dialogue

Zenity is a nice and simple way to prettify Bash scripts, similar to kdialog, xdialog and a few others, but on the whole I think the best of the lot.  Though there is nothing preventing you from mixing them up if one has a feature that would be more useful for something.

I started using Zenity years ago, when there wasn't much documentation and consequently didn't use the progress dialogue the right way.  The result being that some scripts eventually broke.  For example, there was no information on how to make the progress bar progress - so I only used the whirr-whirr pulsate effect.

No Progress

In the past, I simply started a pulsater, pushed it into the background, saved the PID and later killed it when necessary, like this:

zenity --progress &
echo "Do something"
kill $PID

That used to worked fine, but recently on Fedora 22, it just sits there and does nothing.  It won't even blink.  Dead as a doornail.  Bah, humbug...

When All Else Fails...

Eventually, I found the documentation and RTFMed:

The problem is that the progress dialog has a new bug and it expects to receive something over stdin in order to get going.  If it never gets anything, it does nothing - in previous versions it worked fine in the background, but no-more.  When it is pushed into the background with &, it just sits there and sulks.

The correct way to make a progress pulsater that must do something and exit based on a condition, at which point one has to stop the pulsater, is like this:
while TRUE, do
  echo "This Will Do Nothing"
  echo "#Change The Text"
  sleep 5
  let "CNT+=1"

  if [ "$CNT" -eq 10 ]; then
    # Close the progress dialog with 100%
    echo "100"
    exit 0

  echo "Continue doing nothing"
) | zenity --progress \
--width=350 \
--title="The Title"
--text="The Default Text" \
--no-cancel \
--auto-close \

The --auto-close parameter will cause the progress dialogue to stop when you echo "100" percent.

An echo statement that starts with a # will change the text and one without, will be ignored.  The --no-cancel will suppress the Cancel button, since it doesn't make sense.

La voila!

Saturday, September 19, 2015

Version Control For Scripts

Running a full blown version control system such as subversion or CVS for Bash scripts is a bit of overkill.  Most people don't use any version control at all on scripts and revert back to the ancient way of saving multiple copies of the files.

The inevitable result is a mess of files called script1, script2, script.ok, script.bad... and then after a few days, you cannot remember which is which.

Fortunately, there is an ancient, light weight versioning system that is very easy to set up and which is perfect for the single user case: RCS.

Setup RCS

It is probably already installed (try whereis rcs), otherwise do:
# yum install rcs

In your script directory, make a directory to keep the archives:
$ mkdir RCS

Edit each script and add the $Id$ keyword to the top, right underneath the familiar #! /bin/bash command:
#! /bin/bash

That will substitute your name, date, time and version number of the file upon checkout, so you always know which version of the script you got.

Check In

As soon as you made an important change, check the file in, with ci:
$ ci scriptname

Type a proper description of the change, it is for your own benefit, when you have to look at a problem, possibly years later.

The file will disappear!

Check Out

Check it out again with co:
$ co scriptname 

By default you get the latest version.  You can also check out older versions using the -r parameter and switch between versions to see which one works best, without losing anything, since they are all in the archive file.

Important Tricks

If you are the only person working on the scripts, then remove strict locking.  This will make your life much easier:
$ rcs -U *

Now you can check files in with the -u option, so they don't disappear and gets checked out again immediately:
$ ci -u scriptname
You can also set an alias (add to the bottom of ~/.bashrc), to save you some more typing, so that you never have to use the co command again, just ci filename:
$ alias ci='ci -u'

That is pretty much all there is to it.

Do read the man pages of ci, co and rcs for details on what to do when the inevitable happens and you want to look at the logs and revert to an older version of a script.