Saturday, 18 May 2013

Raspberry Pi and the HD PVR

I had a spare USB Hauppauge HD PVR, and a Raspberry Pi.. and had meant to get around to trying this for quite some time. There were even a few posts in the Hauppauge forums where people were wondering if the device would work well with the Pi.

The HD PVR is a component video capture device, good for up to 1080i, with optical or stereo audio input, it has a hardware encoder onboard, so the video is sent to the Pi already encoded as an h264 transport stream. I'm using the original HD PVR 1212 here, there are newer Gaming oriented versions, and an HD PVR2 now.. but I don't know the state of the linux drivers for those.

Once the Pi has the data, it can stream it, record it, or possibly use it as a tuner for a pvr backend.. Read on if you want to follow what I've tried so far..

Basic Setup & Testing...

The good news is quite how simple it is to setup, starting from the latest Raspbian image, the HD PVR was recognised immediately when connected, and created the /dev/video0 node.

Note: This assumes the HD PVR is the only video device connected, if you want to be sure.. use   dmesg  after connecting the HD PVR, and use the name from the end of the line like..
  hdpvr 1-1.3:1.0: device now attached to video0

From there, a great simple test is..
  cat /dev/video0 > test.ts

Which should result in the blue lights of awesomeness engaging, and a file spooling out to disk.. leave it going a few seconds, then kill the process.. the resulting test.ts file should playback ok.

If you are using spdif audio, then you'll need the v4l-ctl tool to toggle the input.. get that installed with       
   sudo apt-get install v4l-utils
And then select the spdif input using the amazingly easy to remember command.. 
   sudo v4l2-ctl -d /dev/video0 --set-audio-input=2
The "2" at the end selects SPDIF, "0" would be the back stereo input, and "1" will be the front stereo input.

The same tool can also be used to configure the HD PVR into VBR mode, and set the bitrates..
  sudo v4l2-ctl -d /dev/video0 -c video_peak_bitrate=16500000
 sudo v4l2-ctl -d /dev/video0 -c video_bitrate=10000000
 sudo v4l2-ctl -d /dev/video0 -c video_bitrate_mode=0

If you are not using the Component input, there'll be more fun to have.. 
   sudo v4l2-ctl -d /dev/video0 --set-input=0
The "0" selects the component input, "1" will select Svideo, and "2" will select composite on the front.

With all that out the way, the Pi should now be creating test.ts files that playback with audio & video. 

I'm just using VLC on Windows to playback, transferring the created file using WinSCP. Being H264, the files should playback pretty well on the Pi with the GPU doing the work.. but I haven't tested that yet..

Reading around a lot online, I find people suggesting to avoid using cat as that will exit if the device produces any errors.. which will mean an interrupted recording.. the general suggestion seems to be to use dd instead like this..
   dd if=/dev/video0 of=test.ts conv=noerror ibs=1024
The noerror bit should mean the recording will continue even if the device does something oddball.

So now I had the device recording.. I wanted a way to watch the stream live..

Live Streaming...

Since /dev/video0 outputs a nice transport stream, this should be as simple as magically getting the data from the device on the Pi, into my VLC on Windows.. I'm deliberately not looking to transcode, or do any processing on the stream Pi side, as I want to keep that as lightweight as possible.

The general consensus online is to use Netcat, a wonderful little tool that lets you pipe data in on one host, and pipe it out the other.. In theory, this should work like..
   Pi:  cat /dev/video0 | nc 192.168.0.X 6666
   Pc: nc -l 6666 | "c:\program files(x86)\videolan\vlc\vlc.exe" -
For this to work, you'll need to install netcat on the pi, and install it on windows, and have more luck than I did.. Every time the vlc player started getting data, the blue lamps would go out on the HD PVR, and the stream would stop. I tried reversing the netcat direction, tried using mplayer instead of vlc, tried using dd instead of cat, and eventually gave up.

So, here's my alternative approach.. it's a little more entertaining.. but it gets there ;p

First, we'll be needing apache, so
   sudo apt-get install apache2

Then we'll be creating an amusing small cgi script..
put these lines into a new file at /usr/lib/cgi-bin/video.cgi

echo -ne "content-type: video/h264\n\n"
dd if=/dev/video0 conv=noerror ibs=1024

I'm using dd if=/dev/video0 conv=noerror ibs=1024 here instead of cat as I found that otherwise the stream would interrupt & drop every so often. The lack of an output "of=" argument means the data goes to standard output.. which from a cgi script is the remote process, VLC in our case.

This script will read from the video device and splat the data back to VLC.. or it would if it were allowed to read the video device, by default the permissions deny this.. so fix them with 
  sudo chmod 777 /dev/video0

Now make sure apache is started sudo /etc/init.d/apache start and tell VLC to access the stream at http://ip.address.of.your.pi/cgi-bin/video.cgi

Bing! magic live video =) Now.. what if we want to record AND stream?

Here I cheat a little.. we can't have 2 processes trying to read from /dev/video0 at the same time, so we'll copy the data from the device so the webserver can use it, and write it to disk at the same time.. I want to use dd for this, but the regular dd will only output to one place at a time, and while we can solve that with some usage of pipes, and use of tee there's an alternative version of dd that supports multiple outputs.. dcfldd so we'll get that.. 

sudo apt-get install dcfldd

Now if we use dcfldd, we could read from the video device, and write to disk, we'll write into a fifo for the webserver, so we don't eat space.. and the webserver won't touch our real storage file.. 

mkfifo ~pi/video.ts

This is a magic not-quite-a-file-really that lets people read from it while you push the data in the other end.. it's use is just like the regular dd, except with multiple "of=" arguments. Here we're outputting to the fifo, and to a file called myrecording.ts (Remember the HD PVR creates large files, you can easily fill the sdcard, consider storing the recording on a usb disk, or network drive). We run this from a spare terminal.. as it will need to keep running relaying the data from the video device to the outputs.. when we want to stop recording, we can kill the process.

dcfldd if=/dev/video0 conv=noerror ibs=1024 of=~pi/video.ts of=~pi/myrecording.ts

And edit the last line of that cgi to be 

dd if=~pi/video.ts conv=noerror ibs=1024

As it now needs to read from the fifo, not from the the video device.

Finally.. reconnect from VLC, and there's your live video feed, while it's storing the data out to disk =)

I'm still seeing some odd drop outs, which I need to debug further.. I suspect even with all the dd conv=noerror usage, that something can still cause it all just to 'stop'. Eventually I suspect this means I'll need to write my own small bit of C that can read from the dev/video, and write out to disk / stdout / a socket .. but that's work for another day.. I'm playing with other ways of doing this.. if I find one that works.. I'll post an update.

Update: still needs more testing.. but using this instead of dcfldd seems to help a bit.. (this is all just one line to enter.. )
  dd if=/dev/video0 conv=noerror ibs=1024 | tee >(dd of=~pi/video.ts conv=noerror) | dd of=~pi/myrecording.ts


  1. This looks interesting. I'm curious, have you tried using the IR blaster in the HD-PVR from this Raspbian setup? I'm a user of SageTV, and would love to be able to use an HD-PVR with a Raspberry strapped to it as a network based tuner for SageTV. Could be located somewhere with a cable box. It would need to be able to change channels on the box with the IR blaster, and would need to be able to store the recorded stream onto a windows network share on the server.

  2. As you've probably spotted by the amazingly almost total lack of updates on this in about 2 years.. I've been busy in other directions ;p I really need to get back into writing blog posts.. I've been messing with ESP8266's, using a Teensy 3.0 to read floppy disks, putting a CD32 online with a homebrew plipbox, and playing with my open bench logic sniffer (that I finally got round to figuring out the software stack for)..

    I really should dig out the old hd pvr and a spare Pi tho, and give it a blast again.. I was part way through coding up a fake tuner for tvheadend for it, although I wasn't planning on using the ir blaster in the hd pvr, I dimly remember from my days of using it as a tuner under windows that using the ir blaster on the device was a quick way to end up with it being unstable / locking up.

    Although now, I suspect I could easily wire a relay off a pi GPIO pin to reset it every channel change.. so it might be doable.. it appears there are now linux drivers for the hd pvr 2, so maybe it's time for me to find one of those & have another bash at this =)

    1. I dug into this a bit, but moved from the pi to the PogoPlug v4. Less power, considerably less cost, but actually better suited to this task. Using a perl script running as daemon to interface with sage and present itself as a tuner. Using ffmpeg to stream from the /dev/video0 directly to the sagetv server's samba shares.

      I'm now just working out the ir blaster. The built-in hdpvr driver that is in the archlinux build for the pogoplug is older, and doesn't support the blaster - so I'm muttling through the very long process of compiling a new hdpvr driver... compiling goes REALLY slow on an 800MHz Kirkwood. I'm very much NOT a linux guy, so this is fun.

  3. Just be careful using tools that merely stream data from /dev/video0 .. as I found above even using dd with appropriate flags set can still cause it to just give up for no apparent reason. (Although rumor has it, that it's something to do with a somewhat small fixed sized buffer inside the unit, that if you allow to fill to capacity, shuts the stream down.. so you have to empty it faster than it fills it to prevent it, and dd/ffmpeg don't always manage that)

    I ended up writing a simple app that pulled data off the device node using one thread, and requeued it into an internal buffer that was then written out to smb shares etc.. during testing in worst case scenarios I could see it use almost all the available memory on an original 256mb pi just acting as a buffer while waiting on the smb write to complete.

    I did make a little headway into coding a backend for tvheadend that allowed the device to act as a tuner (with oob channel changes via a script).. but then I changed countries and project priorities shifted away from playing with grabbing hdmi, towards other more fun things.. =)

    Have a read of the other hd pvr posts I put up tho.. I think they all culminated at this one..

  4. This comment has been removed by the author.

  5. Can this be used to record (onto USB-drive or SD-card) an already-encoded Transport Stream? If so, at what bit rate?