big data

Very good talk from Hilary Mason from a recent Lucene Revolution keynote.

Excellent talk from Pat Helland, “Immutability Changes Everything” :

via igrigorik's twitter

Velocity was on last week, and I was following along enviously on twitter - I'm making a promise to myself that I'll be along in person next year!

Quite a few of the talks now appearing online - here's a couple I've watched so far..

the legendary Mike Hoover turned me on to Backblaze a few months back - they're a cloud storage provider, who opened up the design for their chassis and storage pod solution so you can build your own “Storage Pod 2.0: a 135-terabyte, 4U server for $7,384″ (blog post here)

I've been trying to track down problems with really slow network transfer speeds between my servers and several DSPs. I knew it wasn't local I/O, as we could hit around 60Mb/s to some services, whereas the problematic ones were a sluggish 0.30Mb/s; I knew we weren't hitting our bandwidth limit, as cacti showed us daily peaks of only around 500Mb/s of our 600Mb/s line.

I was working with the network engineer on the other side, running tcpdump captures while uploading a file and analysing that in Wireshark's IO Graphs - stream looked absolutely fine, no lost packets, big non-changing tcp receive windows. We were pretty much stumped, and the other engineer recommend i look into HPN-SSH, which does indeed sound very good, but first i started playing around with trying different ciphers and compression.

Our uploads are all run via a perl framework, which utilises Net::SFTP in order to do the transfers. My test program was also written in perl and using the same library. In order to try different cyphers i started testing uploads with the interactive command line SFTP. Boom! 6Mb/s upload speed. Biiiig difference from the Net::SFTP client. I started playing with blowfish cipher and trying to enable compression with Net::SFTP - it wasn't really working, it can only do Zlib compression, which my SSHD server wouldn't play with until i specifically enabled compression in the sshd_config file.

After much more digging around, i came across reference to Net::SFTP::Foreign, which uses the installed ssh binary on your system for transport rather than relying on the pure perl Net::SSH.

Syntax is very similar, so it was a minor rewrite to switch modules, yet such a massive payback, from 0.30Mb/s up to 6Mb/s.

(It turns out the DSPs i mentioned earlier who could achieve 60Mb/s were actually FTP transfers, not SFTP)

Found this Facebook engineering video quite fascinating, a nicely detailed platform overview of FB's new real-time analytics system:

In the comment section, someone posted a link to a somewhat similar presentation from Twitter about their real-time solution::

(hello Johan!)

yow!

from cisco blog