Monday, March 13, 2017

Troubleshooting Video Streaming

Casting About

Video multicast routing usually 'Just Works'.  Occasionally it doesn't work at all, or the video streams for a few minutes only and then one is left scratching one's head...

To find out what is wrong, one has to know how to analyze a playback streaming problem when it happens.

Here is something to read on streaming, multicasting and the Internet Group Management Protocol (IGMP):


Multicast Version

Sometimes is is necessary to use a specific version of IGMP to get things to work. One can force the IGMP version in the /proc/sys/net/ipv4/conf/eth0/force_igmp_version to 0 (auto), 2 or 3.

Multicast Routing

The multicast address subnet is the 224.x.y.z address range.  For routing to work at all, you need a multicast route, set with a command like this:
# route add -net netmask dev eth0

That is if your ethernet port is eth0.  Mine is enp0s26u1u1 at the moment - I'm using a USB ethernet device on this machine, as the RJ45 socket has a solder problem, resulting in a Welsh device name.

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         UG    1024   0        0 enp0s26u1u1   U     0      0        0 enp0s26u1u1 UGH   1      0        0 enp0s26u1u1       U     0      0        0 enp0s26u1u1

The last line above is the multicast route and that being missing/wrong is 99.999% of the time the actual problem.


Multicast Groups

For multicasting to work, your machine needs to be a member of the multicast group.  This is set at start-up using the IGMP protocol if your route is correct. 

You can list group memberships with netstat:

$ netstat -gn

IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1
em1             1
wlp3s0          1
enp0s26u1u      1
enp0s26u1u      1
lo              1      ff02::1
lo              1      ff01::1
em1             1      ff02::1
em1             1      ff01::1
wlp3s0          1      ff02::1
wlp3s0          1      ff01::1
enp0s26u1u1     1      ff02::1:ff00:10e
enp0s26u1u1     1      ff02::1
enp0s26u1u1     1      ff01::1

Group Subscription

For streaming to work, the client (video player) has to subscribe to the stream with an IGMP Join message.  The routers listen to the IGMP messages (IGMP snooping) and use the data to send multicast traffic only to members of the group.  Otherwise the LAN will be flooded with streaming data.

The network switches keep track of the MAC addresses used by multicast subscribers and this list expires after 500 to 600 seconds.  The subscription therefore needs to be renewed periodically by  IGMP Query and IGMP Report messages, otherwise the switch will stop forwarding the stream and the video player will stop.

To confirm that the subscription is happening and that the video player is configured right, start tcpdump on the server and then start the video player.  You should see something like this:

# tcpdump -i eth0 igmp
tcpdump: listening on eth0
00:20:09.007094 switch-a.stage > ALL-SYSTEMS.MCAST.NET: igmp query v2 [max resp time 20] [ttl 1]
00:20:09.241946 > igmp v2 report (DF) [tos 0xc0]  [ttl 1]
00:20:10.472159 > igmp v2 report (DF) [tos 0xc0]  [ttl 1]

Multicast Ping

You can also ping all members of the group, if the machines are configured to respond to broadcast/multicast pings:

$ ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=2.03 ms
64 bytes from icmp_seq=1 ttl=64 time=2.03 ms (DUP!)
64 bytes from icmp_seq=2 ttl=64 time=2.05 ms
64 bytes from icmp_seq=2 ttl=64 time=2.05 ms (DUP!)

The /proc filesystem variables /proc/net/ipv4/igmp* are useful to see the groups that your host is currently subscribed to. So you can go and dig there yourself if you don't have netstat installed.

The proc File System

Broadcast ping responses are usually turned off, to prevent Denial Of Service attacks.  Ping responses are controlled via /proc as well and can be turned on with:

echo 0 >/proc/sys/net/ipv4/icmp_echo_ignore_all

echo 0 >/proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

or on machines with systemd:
sysctl net.ipv4.icmp_echo_ignore_broadcasts=0

Next time you experience a problem, check your IP device and route setup, check the groups and do a ping to see what is going on.

Phun With Streaming

Now let's have some fun with other streaming tools, ffmpeg and gstreamer.

Record Your Laptop PC Camera With ffmpeg

There are many ways to do this, so let's explore one of the roundabout ways and record the camera to a file, while also playing it to see what is going on.

Use ffmpeg to record the camera:
$ ffmpeg -f v4l2 -s 800x600 -i /dev/video0 -r 30 -f matroska -vcodec h264 /tmp/camera.mkv &

Give the recorder a head start, wait a few seconds then play the file:
$ ffplay /tmp/camera.mkv

Now you have a slow mirror - yeah, da...

To stop, run 'killall ffmpeg' and 'killall ffplay'.

Create a Video Test Pattern With Gstreamer

It can be very useful to generate a test pattern, just to get started with gstreamer:
$ gst-launch-0.10 videotestsrc ! ffmpegcolorspace ! autovideosink

You can make a test pattern with gstreamer as explained above, or you can play a video. Here is the jackpot with every test pattern you may ever want and few more: 

Send and Receive Audio with Gstreamer

Streaming audio between two machines can be very useful for an intercom or radio over ethernet.  Here is something to get you started, by sending and receiving audio on the same machine.

Gstreamer transmit in one terminal:
$ gst-launch-0.10 -v audiotestsrc do-timestamp=false ! audioconvert ! audio/x-raw-int,channels=1,depth=16,width=16,rate=8000 ! rtpL16pay ! udpsink host=localhost port=1234

Gstreamer receive in another terminal:
$ gst-launch-0.10 -v udpsrc port=1234 ! "application/x-rtp,media=(string)audio, clock-rate=(int)8050, width=16, depth=16, encoding-name=(string)L16, encoding-params=(string)1, channels=(int)1, channel-positions=(int)1, payload=(int)96" ! rtpL16depay ! audioconvert ! alsasink sync=false

Talking to yourself gets boring quickly though.

To ensure that two different machines will work together without an ever increasing time delay due to slight clock differences, make the consumer slightly faster than the producer by using a clock rate of 8050 instead of 8000 Hz.

To send audio between two machines, use the other machine IP address instead of localhost and to make an intercom, you need to make both streams bidirectional - by combining the two examples using another ! to concatenate them.

La voila!


No comments:

Post a Comment

On topic comments are welcome. Junk will be deleted.