Busy busy busy

Well.. life presses on.. and somehow I find it's almost a month since I posted here last..

I mean.. just how? it's not like I visited another planet, or entered a tech free lifestyle growing corn & making bread.. is it?

Well, here's some of what I've been up to..


I got a bit sidetracked with postcode data, as the last blog post will say =).. was trying to get the FTTC info for uk postcodes into a nifty blog post with a google map (yay), and other such goodness. BT having placed the files on their site for people to read, and linked them & discussed them in their own forums, I figure it was worth a bash, since the data wasn't all that hidden any more.

After playing with my area, (I'm one of the few cabs on an FTTC enabled exchange, that BT has elected to totally ignore.. roads all around me can have FTTC, but I get to sit here & drink beer instead.. ).. I started seeing about putting it in a blog post.

Why ? well, I don't have my own host, and figured that a post with an embedded app would be kinda nifty, except blog posts need to be about 1mb or so, and the database of info was around 48mb.. ouch. Zip compressed thats still around 4mb, and my first attempt at domain compression left it around 2.3mb, but getting lower than 2mb looked tough.

I remembered a trick from way back, coding games for the amiga & spectrum, I'd stored some info within the pixels of the gfx data. Info hiding in gfx isn't new, that's steganography, and a quick google search showed a few people in the pc demo scene had tried using PNG's for storage (lossless compression kinda important for data ;-) so jpeg is less than ideal). They were more interested in good PNG compression though, and had found some neat stuff about encoding data just in one channel (eg, the red channel) rather than using all the range.

I tried it, and my 1 channel PNG came out pretty awesomely small. But blogger wouldn't accept it, apparently images hosted by blogger have a max of 1600x1600. Mine was about 3800x3800, that didn't bode well, except that I'd only used 1 channel, what if I encoded using all 3 (yes, I know there's sort of 4).

A quick rehack later, and I find yes, I'd need an image of 1800x1800 or so to store in 3 channels.. not quite there.. I did try to use the alpha channel, but clearly I need to understand png encoding better, as that just caused all my data to be corrupt on decode. So I tweaked my data content.. I'd been trying to store neat javascript, but if I removed some of the less important formatting & handled that during the image decompression.. yup.. below the 1600x1600 limit =)

Now I needed to decode it in javascript.. easy enough, load the image, rip the pixel data, store in string, eval etc.. except the images don't come from the same host as the blog text, and security stuffs in javascript say Noooooo if you try that (ok, they actually say some obscure error that needs google to tell you why it went wrong..)

Jump in some google app engine thing that can load your image, and squirt back the data as json !

And that's roughly where I'm at.. far enough along to prove the concept, I _can_ read all the data from a blogpost, with enough room left over for the map, the query api, etc.. and one day real soon now.. I'll put it up =) right after I stop looking at.. oh look.. there's a castle!!