perl

Super nice demo of algorithmic sound generation with Perl and Portaudio…

from here

I've been trying to track down problems with really slow network transfer speeds between my servers and several DSPs. I knew it wasn't local I/O, as we could hit around 60Mb/s to some services, whereas the problematic ones were a sluggish 0.30Mb/s; I knew we weren't hitting our bandwidth limit, as cacti showed us daily peaks of only around 500Mb/s of our 600Mb/s line.

I was working with the network engineer on the other side, running tcpdump captures while uploading a file and analysing that in Wireshark's IO Graphs - stream looked absolutely fine, no lost packets, big non-changing tcp receive windows. We were pretty much stumped, and the other engineer recommend i look into HPN-SSH, which does indeed sound very good, but first i started playing around with trying different ciphers and compression.

Our uploads are all run via a perl framework, which utilises Net::SFTP in order to do the transfers. My test program was also written in perl and using the same library. In order to try different cyphers i started testing uploads with the interactive command line SFTP. Boom! 6Mb/s upload speed. Biiiig difference from the Net::SFTP client. I started playing with blowfish cipher and trying to enable compression with Net::SFTP - it wasn't really working, it can only do Zlib compression, which my SSHD server wouldn't play with until i specifically enabled compression in the sshd_config file.

After much more digging around, i came across reference to Net::SFTP::Foreign, which uses the installed ssh binary on your system for transport rather than relying on the pure perl Net::SSH.

Syntax is very similar, so it was a minor rewrite to switch modules, yet such a massive payback, from 0.30Mb/s up to 6Mb/s.

(It turns out the DSPs i mentioned earlier who could achieve 60Mb/s were actually FTP transfers, not SFTP)

I have a speedtest perl script i wrote - nothing complicated, takes a file and uploads it to a remote FTP or SFTP server, while calculating how long, then gives you a a measure of the MB/per second bandwidth between two sites.

I want it available on a selection of machines so it can run from wherever, so I thought i'd package it up as a .DEB file and stick it in our local repo. Nothing complicated in that, and there are a number of online tutorials about building your own debs. The main drawback with most I found was that they assume you are actually building from source rather than just distributing a script, although I also found a relevant Ubuntu thread which is pretty simple and to the point.

However, even using these tutorials it still took me a few hours to figure out. There are just a couple of non-obvious points, so i figure writing out my own steps is worth recording -

So first, grab your required packages:

apt-get install dh-make dpkg-dev debhelper devscripts fakeroot lintian

You will need to build from a directory with the name of your script in the form packagename-version, so for mine i created /tmp/speedtest-1.0, then copied in my script ‘speedtest‘ and it's data file 25MBFLAC.file ( which i could have created with dd on the box rather than copy over, but downloading the file is actually quicker in this situation ).

The first step is to run:

dh_make -s --indep --createorig -e thor@valhalla.com
(dash-s means create a single binary .deb - i.e. no source version; indep means architecture-independent; and createorig is to indicate you are the original maintainer)

this creates a top-level ‘debian‘ directory containing all the necessary config files.
The main one you need to edit is debian/control - you prob only need fill in “section”, “homepage” and “Description”

Mine looks like:

Source: speedtest
Section: web
Priority: extra
Maintainer: Thorsten Sideboard <thor@valhalla.com>
Build-Depends: debhelper (>= 7.0.50~)
Standards-Version: 3.8.4
Homepage: http://github.com/sideboard/speedtest.git

Package: speedtest
Architecture: all
Depends: ${misc:Depends}
Description: Test Upload Speeds


One of the things which baffled me for a while, which was answered in the askubuntu link above, was how to specify where something is installed — it goes in a file ‘debian/install‘ which isn't created for you. The format of the file is ‘filename location/to/be/installed” (without the initial slash)

so in my case, i ran:
echo "speedtest usr/local/Scriptz/" > debian/install
echo "25MBFLAC.file usr/local/Scriptz/" >> debian/install


At this point, you should then be able to run:
debuild -us -uc

and you should have a deb file built. but..

First i ran into :

dpkg-source: error: can't build with source format '3.0 (quilt)': no orig.tar file found

As the above-mentioned askubuntu post says, you can

echo "1.0" > debian/source/format

then re-running the debuild -us -uc i ran into

dpkg-source: error: cannot represent change to speedtemp-1.0/25MBFLAC.file: binary file contents changed

This error is due to leftover build-cruft from my last run - if you check the directory one step up from where you are, you'll see debuild has already built some files for you, typically a tar.gz, a .dsc and a .build file. Delete all them, then re-run debuild -us -uc — now it should build properly!

ah!

dh_usrlocal: debian/speedtemp/usr/local/Scriptz/speedtest is not a directory

This one also caught me out for a while - turns out this is caused by my specifying “/usr/local/Scriptz” as my install location -

Most third-party software installs itself in the /usr/local directory hierarchy. On Debian this is reserved for private use by the system administrator, so packages must not use directories such as /usr/local/bin but should instead use system directories such as /usr/bin, obeying the Filesystem Hierarchy Standard (FHS).

(from here)

So, yeah, i changed my debian/install file to be “speedtest usr/bin”

and finally! running debuild -us -uc completes properly, outputting a /tmp/speedtest_1.0-1_all.deb which can then be installed via
dpkg -i /tmp/speedtest_1.0-1_all.deb

One last note — there are four useful scripts to also know about — preinst, postinst, prerm, postrm — these should be in the debian/ directory - pretty self-explanatory - pre- and post- install and remove scripts - if these exist, they will be run exactly as they are named, so for example, i wanted my 25MBFLAC.file still to be installed under /usr/local/Scriptz, so i listed it to be installed in the debian/install file as “25MBFLAC.file tmp” and then in my postinst file, i added:

#!/bin/sh
mv /tmp/25MBFLAC.file /usr/local/Scriptz/


Really enjoyed programming along with this perl Moose introduction at Linux Journal - The piece and code are nice and concise.

I put together a quick perl script for comparing installed CPAN modules between two hosts. Find it here.

Quite easy to use:
Usage: ./CompareHostCpanModules.pl login@host1 login@host2

The script ssh's into both hosts (so it's easier if you have your ssh-keys setup) and grabs a list of installed CPAN modules and versions, then outputs the differences - it returns two lists - one of modules installed but having different versions, and another list of modules missing from the second host.

I've been working pretty extensively with Xen and Puppet in my new job, really loving it! I've been creating a whole load of Xen hosts, most of which are cloned from an initial image I built using Xen-tools. I've just finished a script which is over on my github page, which basically automates what was previously a manual process.

Basically, it copies your existing disk.img and swap.img, generates a new xen.cfg file based on some interactive input (desired hostname, IP, memory and number of vCPUs) plus a random Xen mac address, then mounts the disk.img file and changes some appropriate system files - /etc/hostname, hosts, and network/interfaces.

All quite simple and straight forward, but quite nice to have automated.

GenXen

Here's the README:

GenXen #
#############################

A script for automating Xen VM deployment.

It requires that you have a base disk.img and swap.img already created.
I created mine with:
xen-create-image -pygrub -size=50Gb -swap=9Gb -vcpus=2 -memory 6Gb -dist=squeeze -dhcp -passwd -dir=/var/virt-machines -hostname=xen-squeeze-base

Fill in some of the variables at the top of GenXen.pl before running, then simply:
./GenXen.pl

The interactive part will ask for hostname, memory size, vCPUs, IP address, then generate a unique Xen mac address, and write these all to a xen config file which will be saved in /etc/xen/

It'll copy your disk.img and swap.img to destination dir, mount the disk.img and create appropriate files for:
/etc/hostname
/etc/hosts
/etc/network/interfaces

After that you should be good to launch with:

xm create -c /etc/xen/whatever-your-hostname-is.cfg