unix

The Tragedy of systemd

Mon, Aug 13, 2018

Awesome talk from BSDCan 2018, a balanced look at systemd’s history and ideas..

K&R

Sun, Aug 10, 2014

Sad circumstance, lovely stories…

perl parp parp

Sun, Sep 2, 2012

I updated the IP address for both my Name Servers tonite, and was monitoring to see how quickly the new addresses were propagating. First stop was the exceptionally useful Whats My DNS

At the host level I also wanted to track the incoming DNS queries using tcpdump. I could see them streaming into the new host, and visually you could see an obvious difference when viewing the output of the same command on the old host. I googled around for a timer utility which run a command for a given time, so i could quantify the difference. Perfect answer was here, a simple perl wrapper function.

Here's how to use it to run tcpdump command for sixty seconds, and count the packets seen:

# doalarm () { perl -e 'alarm shift; exec @ARGV' "$@"; }
# doalarm 60 tcpdump -u -i eth0 port 53 -n |wc -l
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
19504

Like Treacle, Solving Perl Net::SFTP Slow Transfer Speeds

Sun, Feb 19, 2012

I've been trying to track down problems with really slow network transfer speeds between my servers and several DSPs. I knew it wasn't local I/O, as we could hit around 60Mb/s to some services, whereas the problematic ones were a sluggish 0.30Mb/s; I knew we weren't hitting our bandwidth limit, as cacti showed us daily peaks of only around 500Mb/s of our 600Mb/s line.

I was working with the network engineer on the other side, running tcpdump captures while uploading a file and analysing that in Wireshark's IO Graphs - stream looked absolutely fine, no lost packets, big non-changing tcp receive windows. We were pretty much stumped, and the other engineer recommend i look into HPN-SSH, which does indeed sound very good, but first i started playing around with trying different ciphers and compression.

Our uploads are all run via a perl framework, which utilises Net::SFTP in order to do the transfers. My test program was also written in perl and using the same library. In order to try different cyphers i started testing uploads with the interactive command line SFTP. Boom! 6Mb/s upload speed. Biiiig difference from the Net::SFTP client. I started playing with blowfish cipher and trying to enable compression with Net::SFTP - it wasn't really working, it can only do Zlib compression, which my SSHD server wouldn't play with until i specifically enabled compression in the sshd_config file.

After much more digging around, i came across reference to Net::SFTP::Foreign, which uses the installed ssh binary on your system for transport rather than relying on the pure perl Net::SSH.

Syntax is very similar, so it was a minor rewrite to switch modules, yet such a massive payback, from 0.30Mb/s up to 6Mb/s.

(It turns out the DSPs i mentioned earlier who could achieve 60Mb/s were actually FTP transfers, not SFTP)

building a DEB package from a perl script

Thu, Feb 16, 2012

I have a speedtest perl script i wrote - nothing complicated, takes a file and uploads it to a remote FTP or SFTP server, while calculating how long, then gives you a a measure of the MB/per second bandwidth between two sites.

I want it available on a selection of machines so it can run from wherever, so I thought i'd package it up as a .DEB file and stick it in our local repo. Nothing complicated in that, and there are a number of online tutorials about building your own debs. The main drawback with most I found was that they assume you are actually building from source rather than just distributing a script, although I also found a relevant Ubuntu thread which is pretty simple and to the point.

However, even using these tutorials it still took me a few hours to figure out. There are just a couple of non-obvious points, so i figure writing out my own steps is worth recording -

So first, grab your required packages:

apt-get install dh-make dpkg-dev debhelper devscripts fakeroot lintian

You will need to build from a directory with the name of your script in the form packagename-version, so for mine i created /tmp/speedtest-1.0, then copied in my script ‘speedtest‘ and it's data file 25MBFLAC.file ( which i could have created with dd on the box rather than copy over, but downloading the file is actually quicker in this situation ).

The first step is to run:

dh_make -s --indep --createorig -e thor@valhalla.com
(dash-s means create a single binary .deb - i.e. no source version; indep means architecture-independent; and createorig is to indicate you are the original maintainer)

this creates a top-level ‘debian‘ directory containing all the necessary config files.
The main one you need to edit is debian/control - you prob only need fill in “section”, “homepage” and “Description”

Mine looks like:

Source: speedtest
Section: web
Priority: extra
Maintainer: Thorsten Sideboard <thor@valhalla.com>
Build-Depends: debhelper (>= 7.0.50~)
Standards-Version: 3.8.4
Homepage: http://github.com/sideboard/speedtest.git

Package: speedtest
Architecture: all
Depends: ${misc:Depends}
Description: Test Upload Speeds

One of the things which baffled me for a while, which was answered in the askubuntu link above, was how to specify where something is installed — it goes in a file ‘debian/install‘ which isn't created for you. The format of the file is ‘filename location/to/be/installed” (without the initial slash)

so in my case, i ran:
echo "speedtest usr/local/Scriptz/" > debian/install
echo "25MBFLAC.file usr/local/Scriptz/" >> debian/install

At this point, you should then be able to run:
debuild -us -uc

and you should have a deb file built. but..

First i ran into :

dpkg-source: error: can't build with source format '3.0 (quilt)': no orig.tar file found

As the above-mentioned askubuntu post says, you can

echo "1.0" > debian/source/format

then re-running the debuild -us -uc i ran into

dpkg-source: error: cannot represent change to speedtemp-1.0/25MBFLAC.file: binary file contents changed

This error is due to leftover build-cruft from my last run - if you check the directory one step up from where you are, you'll see debuild has already built some files for you, typically a tar.gz, a .dsc and a .build file. Delete all them, then re-run debuild -us -uc — now it should build properly!

ah!

dh_usrlocal: debian/speedtemp/usr/local/Scriptz/speedtest is not a directory

This one also caught me out for a while - turns out this is caused by my specifying “/usr/local/Scriptz” as my install location -

Most third-party software installs itself in the /usr/local directory hierarchy. On Debian this is reserved for private use by the system administrator, so packages must not use directories such as /usr/local/bin but should instead use system directories such as /usr/bin, obeying the Filesystem Hierarchy Standard (FHS).

(from here)

So, yeah, i changed my debian/install file to be “speedtest usr/bin”

and finally! running debuild -us -uc completes properly, outputting a /tmp/speedtest_1.0-1_all.deb which can then be installed via
dpkg -i /tmp/speedtest_1.0-1_all.deb

One last note — there are four useful scripts to also know about — preinst, postinst, prerm, postrm — these should be in the debian/ directory - pretty self-explanatory - pre- and post- install and remove scripts - if these exist, they will be run exactly as they are named, so for example, i wanted my 25MBFLAC.file still to be installed under /usr/local/Scriptz, so i listed it to be installed in the debian/install file as “25MBFLAC.file tmp” and then in my postinst file, i added:

#!/bin/sh
mv /tmp/25MBFLAC.file /usr/local/Scriptz/

CPAN Diff script

Fri, Dec 16, 2011

diff

I put together a quick perl script for comparing installed CPAN modules between two hosts. Find it here.

Quite easy to use:
Usage: ./CompareHostCpanModules.pl login@host1 login@host2

The script ssh's into both hosts (so it's easier if you have your ssh-keys setup) and grabs a list of installed CPAN modules and versions, then outputs the differences - it returns two lists - one of modules installed but having different versions, and another list of modules missing from the second host.

80s hair unixz

Sat, Jun 11, 2011

History of BSD - this is awesome:

Brief history of NCP -> IP/TCP transition, also awesome:

time travelling with Unix

Sun, May 1, 2011

I discovered Unix back in 1998. At the time i was a lowly Windows desktop support engineer when i started dabbling with Red Hat Linux v 4.2. Compared to the process of fixing Windows 3.1 systems ( i.e. reinstall drivers and reboot - if that doesn’t work, reinstall OS and drivers - repeat until it does work), playing with Linux was a joy of knowledge. Every part of the installation process was a problem hiding a wormhole of required knowledge. Eventually, we managed to start X - very literally, all we had was a grey screen with an ‘X’ to represent the mouse pointer. However, it was addictive - every time you had to fix something, there was a reason for it, and you learned from that experience, and once fixed, it would stay fixed.

So, fast forward 13 years, and I find myself in a wiki loop, reading up on Bell Labs, then early iterations of Unix, and especially Version 7 Unix - http://en.wikipedia.org/wiki/Version_7_Unix - Back in 1998, i had my head full learning Linux and the newly released SunOS 5.7, so even though i was surrounded by older Vax, SGI, and Digital, I didn’t explore much historical software at the time even though i did have a bit of a fixation with the PDP-11.

What caught my eye on the wiki page was this line - “Bootable images for V7 can still be downloaded today, and can be run on modern hosts using PDP-11 emulators such as SIMH.”.

whut??

Yeah! So, this is my step-by-step instructions for compiling SIMH on OS X (Snow Leopard 10.6.7), running the PDP-11 emulator and getting a V7 Unix image up and running on it. There are other HOWTOs out there and the documentation with SIMH does explain all this, but even so, I found it took me several hours of reading and searching to get things working, so I figure theres still room on the internets for what hopefully is a bit more of a concise howto..

You can find the SIMH homepage at:
http://simh.trailing-edge.com/

The SIMH project consists of emulators for a bunch of historical machines, load of DEC machines from the VAX, through to the PDP range, plus IBMs, Honeywell, and even the Altair 8080.

“The Computer History Simulation Project is a loose Internet-based collective of people interested in restoring historically significant computer hardware and software systems by simulation. The goal of the project is to create highly portable system simulators and to publish them as freeware on the Internet, with freely available copies of significant or representative software.”

(You can also find a precompiled OS X binary at http://homepage.mac.com/mba/simh/index.html however compiling wasn’t hard, and its all part of the fun, right?!)

First, if you don't already have it, you'll need to install Libpcap for network support, which you can get from http://www.tcpdump.org/ - Install libpcap with the usual ./configure, make, make install dance, then download the simh source.

To install on OS X you need to export the OSTYPE environment variable.
(Or you'll get:
ld: library not found for -lrt
collect2: ld returned 1 exit status
make: *** [BIN/pdp1] Error 1)

echo $OSTYPE

export OSTYPE
make USE_NETWORK=1

I encountered my first failure here, no BIN directory. Easily fixed with:

mkdir BIN
make USE_NETWORK=1

boom!
cd BIN
./pdp11 

After compiling, you can fire up any of the emulators and you’ll find yourself in a sim> shell.

“Now what the hell do i do?” was my first thought.

It took me a bit of reading to understand how to use the simulators, basically they are composed of the devices such as the CPU, disk, memory etc, plus Registers for which you set values. They are quite fully explained in - http://simh.trailing-edge.com/pdf/simh_doc.pdf -

In order to load up any of the software, you have to set some device options, map your software image to a device then load it up. You’ll find the software at:

http://simh.trailing-edge.com/software.html

(and for more thorough explanation, read the doc at http://simh.trailing-edge.com/pdf/simh_swre.pdf)

The correct V7 Unix to run on the PDP-11 is :
http://simh.trailing-edge.com/kits/uv7swre.zip

The instructions given in the documentation are:

sim> set cpu u18
sim> set rl0 RL02
sim> att rl0 unix_v7_rl.dsk
sim> boot rl0
@boot
New Boot, known devices are hp ht rk rl rp tm vt
: rl(0,0)rl2unix
#

They are absolutely correct, however, a gotcha for me was that after i typed “boot rl0” I only got an “@” sign, and nothing more. I thought perhaps i had a corrupt image or something was different with my emulator. But no, i was just being a numpty, you actually need to type “boot” and press return, then type in the “rl(0,0)rl2unix ” line too.

I also had to create a /tmp directory so i could view man pages, but yeah, apart from that, you should be good to go!

I also had fun playing with 2.11 BSD which i got from :
http://ftp.fibranet.cat/UnixArchive/PDP-11/Boot_Images/2.11_on_Simh/

This is a really well put together package, with a clear README file on how to use it.

download, unzip and run as:

./pdp11 211bsd.simh

You can even telnet into your virtual host:

rar!