Sunday 2 June 2013

Raspberry Pi and the HD PVR.. streaming and recording, Part II

It's working =)

Finally got to the bottom of the drop outs / glitches.. Some were my fault in the code, some were due to a dodgy cable feeding the HD PVR from the set top box. With the replacement cable in place, I had it stream for 12 hours straight with no reported dropouts.. that's close to a record for this unit =).

With the new cable in place however, I could still make it glitch, but only when recording.. the very time you didn't want glitches ;p This was while writing data to a network share, mounted via cifs, and the glitches became more frequent if the same network share was used via another client.. all of which got me thinking...



I was only using a single thread to handle the read from the device, and write out to all socket clients, and to the recording file.. maybe the writes to the mounted share were taking too long, and the extra delay meant too much data built up at the device node, causing the connection to be dropped..

I had a chat with the MythTV developers in their IRC channel, figuring they must have hit stuff like this when adding support for the HD PVR, that gave a few ideas.. and I added a buffered writer thread (It's at this point you really miss Java). With a little careful use of locking, and buffer allocation though, I've got a buffered writer thread that writes buffers from a write list, and puts them back onto a free list to be reused.

(Ok, if you're reading the code, they are arrays, not lists, and thus are pre-allocated to a max length, but the array is only holding pointers to the buffers, not the buffers themselves, so the overall overhead isn't too bad).

With this new buffered writer in place, the glitches during write disappear.. and I discovered I needed to update the code to large file support (oops, should have seen that one coming!!), thankfully that was as simple as adding a few #defines.

Lastly, since I now had a buffer pool style solution running, I added a way to monitor the buffer usage..
   http://ip.address.of.pi:1101/status
will report the number of connected streaming clients, the number of buffers in the free pool, the number of buffers with data waiting to be written to disk, and the total number of buffers. In an ideal world the total number of buffers will stay small, but it will start climbing rapidly if the network gets busy.. Currently the code is set to allow a max of 4096 buffers, that seems enough to cope with my network, although the amount may need tweaking..

A quick check on the pi says that a char * pointer takes 4 bytes, so that's 32k for the free/write array storage, and holding 4096 buffers of 4k each, will eat 16MB of ram.. even on a 256MB pi that seems ok =). I've seen my total buffer usage creep up toward 11MB so I might even consider allowing 32MB (8192 buffers) if it helps guarantee those writes.

Now it's onto the coding of the java xmltv/imdb channel changing recording scheduler... I made pretty good progress on that so far (after getting myself temporarily blocked on Google & Yahoo ;p)

And watch out for those URL auto suggestions! it seems Chrome actually sends a speculative HTTP GET for a url it offers you, before you select it and hit enter.. obviously, this is bad when you want to request 'status' and it hits the 'stoprec' url ;p

Update: the device restarts are gone with this version.. but it still seems to write bad data if streaming clients join while recording to disk.. I'll try moving the streaming client writes to a buffered writer too.. see if that helps.. although it'll get fun with 2 consumers of the buffer data ;p 

Update 2: have moved the socket writes to the buffer output thread, and added another thread to reclaim excessive free buffers over time.. this has fixed the joining client issue.. also tidied up a few bits .. new code to go soon when I get the http parser rewritten.

Source:

11 comments :

  1. Hi, I was very excited to find your blog, because you seem to be already successfully doing exactly what I wanted to do, that is to record with my HD PVR through the pi to a smb share. I often encounter repeated skips in my TS files when I record through my pc (even when recording to the local harddrive), which is very frustrating. So, now that I have a pi, I thought that the pi might be a lot stabler for this.

    So I did all you described, including compiling your code above, and first it seemed like I was lucky, since my file started growing, but then when I also opened VLC (which only gave screen updates every few seconds, and no sound), the file stopped growing in size.

    Since that moment, I have not been able to record properly anymore. It creates a zero sized file, and that's it. Nothing else happens. I carefully checked my script, but I don't see anything I could be doing wrong... Below is the output from my script. As you can see, I opened http://127.0.0.1:1011/startrec from within a pi ssh shell, to avoid that it's a browser problem. But also when I try to open it from chrome on my pc, the problem remains the same: only a zero sized file is created, but no errors are displayed...

    pi@raspbmc:~$ ./start video/Movies/_new/_raw/
    Video folder is mounted as /home/pi/video

    HD PVR is present. Will set it up now.
    Audio input set to 2
    Video input set to 0 (Component: ok)
    ~/video/Movies/_new/_raw ~
    waiting for a connection
    -0--------------------
    Received connection from 127.0.0.1
    waiting for a connection
    Received bytes 105
    Received string "GET /startrec HTTP/1.1
    User-Agent: curl/7.26.0
    Host: 127.0.0.1:1101
    Accept: */*
    Content-Length: 0

    "
    Sent bytes 105

    Would you have any idea what I could be doing wrong? I restarted both the pi and the hdpvr, but nothing helps :-(

    Thanks in advance!

    Regards,
    Ronny

    ReplyDelete
  2. Hi.. VLC only updating every few secs seems like VLC might be running on a system that can't handle the decoding.. And with the current version, if the socket client is slow/unlucky, it can glitch the recording stream. The next version I've moved the socket i/o to the buffered output thread, which seems to have helped. I'm running my Pi headless, and using VLC from a PC (i5 2500k) .. in theory the pi should support h/w decoding of it, and that's on my list of things to try out eventually..

    It's interesting your log above has some data, but not all, it's not saying if the code recognized your request (which it should do, even for an unknown one).. it's also missing the timestamps.. which hints quite strongly you are running the code from the last post (27 may), not the code from this one (Giveaway is the 'Received string'... line that was only in the last version. The version you are running doesn't do buffered i/o to disk at all, so struggles on SMB.

    In the meantime, check you can actually read/write from the device to the file on the SMB mount using cat /dev/video0 > /home/video/pi/test.ts and check that file does get bigger.. if that doesn't work, then the code I've written won't either. I notice you've selected audio input as Spdif, which is definitely trickier to get working than straight stereo rca, as the HD PVR needs a valid audio signal to be able to 'lock on' (blue bling lights start glowing solid).. try stereo to rule out if that's the problem.

    I'll put up a post with my latest build soon.. I'm just adding scheduled event support to allow me to tell it to start/stop at programmed times.

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. (Oops, I wrote a comment but only after it got published, your reply showed up, so I quickly removed that comment.) So I don't know how I managed to copy and paste the first code instead of the newest, but apparently somehow I did :-( Thanks VERY MUCH for your good tips, I'm going to try them and let you know. Looking forward for your latest build :-)

    About the scheduling you are working on, even just a parameter with startrec that tells how long the recording should be (in minutes), would be totally sufficient for me. Or otherwise, if it is somehow possible, some kind of scripting language where one can interact with the box one is recording from (through infrared signals), that would be really something... But that is quite a challenge. So far I've been manually starting the things I wanted to record, that's quite alright. The biggest concern is definitely that the recording should be without any interruptions.

    Thanks again, the prospect that this will actually work, is very encouraging :-)

    ReplyDelete
  5. So I compiled the right code this time, but unfortunately still no luck... I tried using an analog input, didn't help.

    pi@raspbmc:~$ ./start video/Movies/_new/_raw/
    Mounted video folder as /home/pi/video

    HD PVR is present. Will set it up now.
    Audio input set to 0
    Video input set to 0 (Component: ok)
    ~/video/Movies/_new/_raw ~
    2013-06-23 11:03:17 Waiting for a connection on port 1101
    2013-06-23 11:03:39 Received connection from 127.0.0.1
    2013-06-23 11:03:39 Waiting for a connection on port 1101
    2013-06-23 11:03:39 Start recording request...
    Writer woken up..
    2013-06-23 11:03:56 Received connection from 127.0.0.1
    2013-06-23 11:03:56 Waiting for a connection on port 1101
    2013-06-23 11:03:56 Start recording request...
    ^C~
    pi@raspbmc:~$

    BTW, it says "Start recording request" even when stoprec was sent, but I guess you've noticed that already.

    Then i tried the dd command (after switching back to the spdif input), let it run for 20 sec, and this time it did create a file.

    The weird thing is that it has actually worked once, the very first time I tried it. In all my later attempts, I didn't even have VLC open to preserve cpu time, I have the recorded source straight on my TV. Is that the cause of my trouble?

    ReplyDelete
  6. I did have a bug where I had to have at least one streaming client running to record.. Not sure when I fixed that, but it's gone in my current versions, the streaming should be totally smooth if the pc is able to handle the cpu load to decode the stream. Can your pc playback a ts file recorded with cat, over smb using VLC ? if that's totally smooth, then there's something odd going on.

    It's good that cat is working =) I'm assuming the device stays on /dev/video0 (it won't if you mess around hot plugging it after crashing it ;p) .. so if yours isn't on video0 then you'll need to edit the code..

    Might want to check if it needs sudo to launch.. I think I tweaked the perms on the /dev/video0 device to let my pi user play with it directly.. maybe it worked the 1st time because you were root?

    My usual usage is fire up the program.. connect with VLC, once I have a nice solid blue light & VLC is playing, send the start/stop requests, leaving VLC running so I know when the programme ends.

    I've put my current code (which doesn't yet do scheduled recording, although it's pretty close) at https://gist.github.com/anonymous/565ceededcc814c958b5 see if that version helps out..

    ReplyDelete
  7. I see. VLC is not working reliably on my pc, which is a very lightweight Atom, that's why I avoid using it.

    I do launch your program with sudo indeed, it needs it even if only to be allowed to write in the smb folder.

    But your latest code works (even without VLC)!! Wonderful. Thank you so much!! Now I only have to figure out what VBR I was using on the pc, the image quality of the test recording seems way too soft now.

    PS. It still says "Start recording request" for /stoprec.

    ReplyDelete
  8. Glad to hear it's working =) I put up the info on how to change bitrates etc in my first hd pvr post, although most of that info came courtesy of the myth-tv pages on the hd pvr.

    I'll update that stop message next time I'm in the code.. do try the /status url now tho, as it has a bit more info.. it's set to use 8192 buffers in the new code, which should be enough to ride out a fairly busy network share, should be able to increase that if you see via /status that the no of in use buffers is regularly approaching that value..

    The new code has the worlds most trivial webserver embedded in it now too.. any url starting /web is served from the current dir the prog was run from (eg /web/fred.html will serve fred.html from the current dir when the prog was launched). It's also most likely the worlds least secure server, so I wouldn't go making it internet facing ;p

    You can mount the folder to allow the pi user to write to it, in the long run it's safer than letting my random code run as root.. I usually mount with a command like.. sudo mount -v -t cifs //ip.of.my.server/share -o file_mode=0777,dir_mode=0777,uid=1000,gid=1000,rw,username=MyUserName /home/pi/mysmbfolder

    If you are using win8 or win7 as the server, you may also need to set some magic registry flags if you discover the server starts reporting "not enough resource" errors as mine did.. I'll try to remember to post about it next time =)

    Thanks for taking the time to reply though.. it's nice to know it was worth writing all this up, if it means just one other person gets some value from it!!

    ReplyDelete
  9. Hi, and sorry for responding so late. I didn't have time during the week to actually record anything, but today I tried some more.

    Thanks for the advice on mounting, indeed this way I don't need sudo to call your program.

    I even managed to set up a cron job to call "stoprec" at a certain time to stop the recording.

    One thing I could not figure out is how to reliably run it with "&" and redirect output to a file, so that I can exit the ssh command shell after I started a recording. Now I need to leave my pc on, otherwise the ssh session will close, and your program will be killed, I suppose. So, it would be nice if there were a way to run it in the background, and also a way to exit it through a web command.

    I could of course try to run your program from the console for now. That way it should remain running and I could probably redirect output to a file and access it from an ssh session when I wish to see if there were any interruptions in the recording.

    About the status on the free buffers, isn't it possible to print a message if the number of free buffers is near zero? That would be more practical than having to call the status command.

    Anyhow, thanks again for putting so much time into researching and developing this, and especially sharing it with the rest of the world. You are a very kind soul.

    I'm certainly looking forward to your version with built-in scheduling. Or a way to specify how many minutes a recording should last. Thanks!

    ReplyDelete
  10. PS. I guess you have been thinking about that already, but you could use your mini web server part to make an "HD PVR recording settings" webpage, if you could figure out how to accept the posted values and turn them into v4l2-ctl commands...

    Although personally I don't really need that desperately, the only thing I miss is being able to pass the number of minutes to be recorded along with the start command... And the possibility to run it in the background.

    ReplyDelete
  11. Instead of redirecting.. install screen (apt-get install screen) which is like a virtual console you can connect/disconnect to.. to connect to it, screen -r and once inside it launch whatever, then disconnect with "ctrl-a d". There's a lot more to it, (man screen), but that's enough to use it as a task backgrounder.

    It does have a message when it runs out of buffers..
    "Out of write buffers!! - reusing last buffer.. "
    .. which will be issued for every buffer it has to do that for, meaning the last buffer is lost forever, and the recording will now have a glitch.

    The good news is I altered the write strategy in my latest version, and it's looking a lot more tolerant of busy networks =) I can even write two files at the same time =) .. Hoping to post about it as soon as I create a basic webpage to manage it all =)


    ReplyDelete