One theme I’ve noticed recently is a move to User Space networking - not for normal application use per se, but for specific high throughput cases, where the kernel’s general network stack’s overhead is too much. This isn’t a new thing, but something I’ve never gotten around to exploring. This vid, from last year, provides an overview of three different implementations..
Solomon Hykes, creator of Docker, speaking at Dockercon - paints a nicely detailed overview of all the new Docker ecosystem libraries released recently - Libcontainer, Libchan, and Libswarm - basically all middle layer abstractions which seem to have buy across all the main platforms and providers. He starts talking about 10mins in..
I updated the IP address for both my Name Servers tonite, and was monitoring to see how quickly the new addresses were propagating. First stop was the exceptionally useful Whats My DNS
At the host level I also wanted to track the incoming DNS queries using tcpdump. I could see them streaming into the new host, and visually you could see an obvious difference when viewing the output of the same command on the old host. I googled around for a timer utility which run a command for a given time, so i could quantify the difference. Perfect answer was here, a simple perl wrapper function.
Here's how to use it to run tcpdump command for sixty seconds, and count the packets seen:
# doalarm () { perl -e 'alarm shift; exec @ARGV' "$@"; }
# doalarm 60 tcpdump -u -i eth0 port 53 -n |wc -l
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
19504
I use tcpdump a lot, but mostly at a reasonably high level, only really restricting the capture to host and port info, then pulling the dump back into Wireshark for nicer visualisation and easier filtering.
A couple of months back I read Moonwalking With Einsten, which is a nice pop-science history of the importance of memory in previous societies, alongside the contemporary phenomena of competitive memory competitions. The book is great, and explains how feats of memory are achieved via the technique of memory palaces, a technique dating back to Roman times - spatial memory relationships. I've been using the technique a lot since I read this book, and truly, no magic to it, it really works. Basically, when you have a list of items to remember, you weave each item, in order, into a spatially focussed narrative.
So, last night, I get out my copy of TCP/IP Illustrated, Volume 1*, one of my most-returned-to tech books - I've always wanted to have a more encyclopedic knowledge of the lower level details of TCP/IP, and last night applied the Memory Palace Technique to the structure of a TCP packet.
(( read the wikipedia article for more details))
In my memory palace I was walking down the path towards the house where I grew up, and seeing a ‘SoRCerer/Src Port‘ battling with ‘Dick DaSTardly and Mutley/Dst Port‘, then walk into my mothers front hallway with a Sequence Number along the front hall, then my Grandfather sitting in a chair in the living room saying “ACK!” because the soccer is on the television and he's complaining about the Header Length … you get the idea - but yeah, you need to make your own memory palace.
Now that I have a complete image of this TCP packet in my head suddenly expressions like :
tcpdump -ni en1 tcp[13] == 18 and host 172.16.1.200 and port 80
are way easier to understand and use - the tcp[13] part refers to the 13th Octet of the packet, which is the Flags octet, then the 18 part is a simple decimal representation of the binary flags, in the order they are in the diagram above - i.e the Flags are
CWR | ECE | URG | ACK | PSH | RST | SYN | FIN
so in my example 18 refers to having both the ACK and SYN flags set - 00010010 which if you're used to dealing with netmasks math is quite an easy translation. My example, then, will only capture the first response packet from the server, as it would be the only part of the conversation to have both an ACK and SYN flag set. (I used a separate memory palace for the flags themselves)
To capture all SYN packets, including the ACK/SYN ones, you would use:
tcpdump -ni en1 'tcp[13] == 18 or tcp[13] == 2' and host 172.16.1.200 and port 80.
Memory Palaces are pretty damn useful!
** Most Engineers are aware of TCP/IP Illustrated, however a lot of people I've spoken to aren't aware there was a 2nd Edition published in November of 2011, updated by a guy named Kevin R. Fall - I would absolutely recommend it, an amazing book and especially with the updates, just seems an essential addition to any Engineer's library..
A month or two back, i saw an interesting figure on Bram Cohen's blog:
“The speed of light in a fiber optic cable around the earth’s circumference is about 200 milliseconds.”
(from here)
I clipped it for my ever expanding Evernote tech tips, thinking it's one of those useful metrics to know. I've referred to it a few times now, but I always like to verify things myself, so this morning I looked up the relevant data -
So - speed of light in a vacuum is 186,000 miles per second. However according to this wikipedia article, the index of refraction for the cladding of an optical fiber is 1.52. “From this information, a good rule of thumb is that signal using optical fiber for communication will travel at around 200 million meters per second”.
Ok, so 200, 000,000 meters / second = 200, 000 meters / ms
“The circumference of the earth at the equator is 24,901.55 miles (40,075.16 kilometers).” // from here
40,075.15km = 40,075,000 meters
With all figures then, Earth Circumference is 40,075,000 meters, and the speed of light in fiber is 200,000 meters per ms:
40,075,000 /2 00,000 = 200.375 ms
// or to be even smarter, I could have just followed the Wolfram Alpha link from Bram's blog here - gotta love the Wolfram //
via Google Blog
[[ image half-inched from here]]
I started studying Sociology a few years back with the Open University, but never managed to complete my course as I got a job here in the States and turned my study-time back towards practical technology. I did however study it long enough for it have quite a profound effect on my understanding and conceptualisation of networking, and felt especially influenced by the works of Manuel Castells and Stephen Graham.
I just came across a paper i wrote almost a year ago which covers a lot of these ideas and ties in quite well with the general theme of this blog, so i thought i'd post it up here instead of languishing on my hard drive…
Why is it important to understand how a city’s fortunes are shaped by its connections?
The City is our personal gateway to the wider world. It resonates with a dense polyrhythm constructed of the flow and foci of innumerable networks. From the physical infrastructure under and around us: the travel and power networks, water, gas and telecommunications lines; to the more ephemeral flows of culture, people, information, finance, and commodities. These structures, relationships and their interconnections are the very essence of a city, connecting the local to the global. In order to fully comprehend the economic and social wellbeing of an urban spatiality, it is essential to look at how a city is positioned within a wider global system, and conversely to examine how these global flows connect to local networks. I would like to answer the above question by first looking at the role of connections in a city’s formation and then explain how these connections extend out to form a global network of influence, including the historically new form we now find ourselves in, the Network Society. Within this framework, we shall then examine how these flows connect locally, explaining the new forms of social division created by a combination of technology and ideology, and why now, more than ever, it is critical to understand how a city’s fortunes are shaped by its connections.
Although we have no concrete proof of the origins of the first cities, Jane Jacobs has a particularly convincing argument for their establishment as trading centres, locations of some geographic or social convenience that became a permanent market place. According to the theory, as more people settle in one place, more opportunities for connections are easily made, and local networks of cooperation and competition would grow and drive innovation. Initially they would be trading with local neighbouring lands, but as trade would increase, these local connections would stretch out more and more to form part of a larger network. Trade would quickly diversify through the division of labour, where the commodity itself would be “one export. The other export is a service: the service of obtaining, handling and trading goods that are brought in from outside and are destined for secondary customers who also come from outside” (Jacobs, 1970, p21).
Chicago is a good example of one such city whose growth was predicated upon positioning itself within emerging trade networks. A small trading centre since the late seventeenth century, its growth was assured when it was connected to the first rail and telegraph networks in 1848. With travel time between Chicago and the East Coast cut from over two weeks to two days, and the near instantaneous messaging of the telegraph, “the pace of life had speeded up and the distances covered by flows of goods, people and information were ever greater”. Chicago managed to place itself at the centre of a voluminous pan-American trade and travel network, thereby establishing itself as a thriving and vibrant financial and social hub. As easy it would be to equate this fortune with Chicago’s geographic positioning, the reality was that it achieved and maintained this dominant position through political and economic manoeuvring - before the construction of the railway, Chicago was competing with St Louis, which had a more capable waterway system. Chicago’s good fortune was due to some canny businessmen who realised they needed an alternative network to compete with the waterways, and who set up a railway company by persuading the local farmers to invest in them. (Pile, 2010, p24-35)
Competition over trade routes is a recurring story throughout the development of the modern interstate system. The early capitalist city-states of fifteenth century northern Italy, of which Venice was the most prominent, established their wealth and power through a monopolistic control of trade routes to India and China. Other northern European countries, mainly Spain and Portugal, tried to find alternative routes to bypass the Venetian monopoly, and it was through this process of exploration that Columbus “discovered” the Americas, thereby creating a whole new network of connections. The sixteenth century saw a great deal of change as Spain, Portugal and the mini-empires of France, England and Sweden all vied for world power through territorialist expansion of their respective networks (Arrighi, 2010). Although I digress here and talk of nation states, the unit of power and of management remains that of the city. We can see this in the conquest of Mexico City by the Spanish in 1521, which transformed the Aztec City, then known as Tenochititlan. Until that point, its dominant network of social relations and trade was confined around Mexico, but with the arrival of the Spanish, that network changed, as the flow of power now came from Madrid, and Mexico City’s “local dominance was now in turn subordinated to an even greater power, a new imperial capital across the Atlantic” (Massey, 2010, p105).
The Dutch Republic, operating from its capital, The Hague, managed to shape its own fortune and usurp Spain’s might by creating a new network of connections atop trade routes, a level of abstraction beyond the trading of physical goods: financial networks. It became the hegemonic power on the world stage by innovating forms of financial speculation based on capitalist expansion rather than territorialist expansion. “These networks encircled the world and could not easily be bypassed or superseded” (Arrighi, 2010, p46). Over successive centuries we have seen the balance of world power shift through manipulations in these network of connections, with first the United Kingdom and then the United States, leveraging themselves into subsequent positions of economic strength.
Immanuel Wallerstein’s World System Theory provides a useful framework for understanding the nature of these global flows of power, conceiving the world not as separate nations with separate economies but as one interlinked capitalist world economy. His conceptualization of the modern world system distinguishes between the Core (developed) and Periphery (developing) countries, with the core countries exploiting the resources of the periphery through monopolistic control of network connections (Arrighi, 2010). Within this widescreen view the relationship between a city’s fortunes and its connections to the network of power becomes clear, that it has to be actively aware of, and strive to maintain it’s position within the global “’power geometry’ – different cities have their own trajectories and there is a constant process of the making and unmaking of connections” (Massey, 2010 p124).
Society is now entering a new form of space-time experience – the Informational Age, a new form of network and thus, new forms of connection for a city. Although today’s main network of power is still finance, Manuel Castells points out a crucial difference: Although we have had interconnecting networks of influence and a world economy going back now to the sixteenth century, it is only now due to computer networks that we have a truly global economy “with the capacity to work as a unit in real time, or chosen time, on a planetary scale” (Castells, 2010, p101). Following similar societal changes as the train and the telegraph, the mass adoption of broadband Internet connectivity in many aspects of our everyday life is drastically changing our lived experience of space and time. The Internet has its roots in military scientific work of the 1960s and 1970s, however the mass adoption of the technology into everyday life only began in the mid 1990s with the advent of the World Wide Web, essentially an easy to understand and use interface to the Internet; In just over ten years, Core countries of the West have migrated whole areas of life onto this digital network, affecting everything from work and education to banking, government, leisure, travel, media, relationships and much more. A whole new network space of power, from its physical fibre-optic network infrastructure, to the new virtual realms it allows.
Since the Haussmannisation of Paris in the mid-nineteenth century, the overarching ethos behind urban planning was a scientific-minded belief in a comprehensive and unified infrastructure. Guided by Keynesian welfare states, the modern infrastructural idea for most of the 20th century was based on universal access and cross-subsidized provision provided by a government or private monopoly, such as the railway or telephone networks. Since the 1980s, this way of thinking has become eroded as ideas of privatisation and liberalisation of the markets gained popularity. The monolithic conception of a city as a coherent unified machine no longer fit with postmodern ideas of identity while technological advances allowed for the creation of tiered and premium network services. Technological control now allows secessionary network infrastructure such as private tolled highways, gated communities, enclosed malls, and Business Improvement Districts - an idea originating in the US, but now “found in Europe, the Caribbean, Australia and South Africa” (Hannigan, quoted in Graham/Marvin, 2009 p261). Business Improvement Districts take over the running of their own network connections: street cleaning, lighting, garbage, policing etc., leaving remaining areas to deal with their own problems. Although in theory the market should provide for all who have a requirement, according to the work of Castells what we are seeing is a new social division, “structurally irrelevant people”, people who have no economic power and who therefore, the market can simply ignore because the “architecture of global networks connects places selectively, according to their relative value to the network” (Castells, 2010 pXXXV)
Stephen Graham and Simon Marvin call this phenomenon “Splintering Urbanism”. They outline the physical geography of the network society by looking at the powers behind, and embodied in the urban infrastructure that services the network. They highlight that this area is often overlooked due its technical nature, often dismissed by architects, sociologists and geographers as a politics-neutral engineering problem, outwith their area of expertise. Graham and Marvin demonstrate a rise in “premium networked spaces of the splintering metropolis” across the globe, a new geography transcending ideas of Core and Periphery countries, centring on cities from Shanghai to Manhattan, Sao Paolo to Montreal, Dubai, London, Bombay, and beyond. We see express highways and train routes connecting business centres to international airports, bypassing surrounding local areas to create virtual network topologies. (Graham, Marvin, 2009). We should strive to see that not all inhabitants of a city feel the benefits of these global connections of power and financial flows equally, a problem exasperated by the dual effects of government ideology - the prevailing idea to allow the market to manage all aspects of society - combined with the technological sophistication to allow very selective and granular unbundling of network services.
However, we must also be aware that the situation is more complex that this binary description of the Included and Excluded. Doreen Massey adds definition to the Network Society concept, using examples of poverty in Bombay and Los Angeles to show how these dominant spaces of capital flows are contested urban areas, with differing rhythms sharing a shaky co-existence. There are a multiplicity of flows within a city which stretch beyond it boundaries, and though certain people may be excluded from certain flows, “they are all the products of complicated interweavings of networks of social relations” (Massey, 2010, p 130)
Drawing together the various threads of this topic, we can see that a city’s fortunes are one and the same as its connections to larger network flows. Originally a city’s connections would be with neighbouring regions, but society has evolved over the centuries into one worldwide flow of influences and trade. Now, as an Informational Society, we see the emergence of a singular realtime global economic network, yet in sharp contrast we see a greater division of wealth and power. As we enter this new stage of Society the old systems and language for understanding structure and inequality are no longer adequate to express this new historical reality. An understanding of the mechanics and flows of this new virtual geography and multi-tiered network society becomes an absolute necessity for anyone involved in the planning and governance of urban space, or indeed for anyone simply living or working in a city.
References:
Allen, J., Massey, D., and Pile S. (2010) Understanding Cities: City Worlds, Oxon, The Open University.
Arrighi, G. (2010), The Long Twentieth Century: Money, Power and the Origins Of Our Times London, Verso
Castells, M., (2010) The Rise Of The Network Society, Oxford, Wiley-Blackwell
Jacobs, J. (1970) The Economy Of Cities, New York, Random House
Graham, S., and Marvin, S., (2009) Splintering Urbanism: Networked Infrastructures, Technological Mobilities and The Urban Condition, New York, Routledge
I started reading James Gleick's “The Information” last week and haven't been able to put it down yet - so good!
I just found this video of a talk he presented at Google last year on the book, looks ace, i'll save it for watching this evening.
I've been trying to track down problems with really slow network transfer speeds between my servers and several DSPs. I knew it wasn't local I/O, as we could hit around 60Mb/s to some services, whereas the problematic ones were a sluggish 0.30Mb/s; I knew we weren't hitting our bandwidth limit, as cacti showed us daily peaks of only around 500Mb/s of our 600Mb/s line.
I was working with the network engineer on the other side, running tcpdump captures while uploading a file and analysing that in Wireshark's IO Graphs - stream looked absolutely fine, no lost packets, big non-changing tcp receive windows. We were pretty much stumped, and the other engineer recommend i look into HPN-SSH, which does indeed sound very good, but first i started playing around with trying different ciphers and compression.
Our uploads are all run via a perl framework, which utilises Net::SFTP in order to do the transfers. My test program was also written in perl and using the same library. In order to try different cyphers i started testing uploads with the interactive command line SFTP. Boom! 6Mb/s upload speed. Biiiig difference from the Net::SFTP client. I started playing with blowfish cipher and trying to enable compression with Net::SFTP - it wasn't really working, it can only do Zlib compression, which my SSHD server wouldn't play with until i specifically enabled compression in the sshd_config file.
After much more digging around, i came across reference to Net::SFTP::Foreign, which uses the installed ssh binary on your system for transport rather than relying on the pure perl Net::SSH.
Syntax is very similar, so it was a minor rewrite to switch modules, yet such a massive payback, from 0.30Mb/s up to 6Mb/s.
(It turns out the DSPs i mentioned earlier who could achieve 60Mb/s were actually FTP transfers, not SFTP)
Ever since reading Neal Stephenson's Mother Earth Mother Board article, I've been quite fascinated with the undersea cables which physically connect the land masses of the world. This map, linked above is an amazing view into this part of the Internet's current infrastructure.
(via flowingdata.com)
One of the tools i've been using a lot of recently is iperf - a really simple and sleek tool for measuring bandwidth between two hosts - rather than write up a full tutorial myself, i'll simply point you at this one by Jayson Broughton
more soon!
yow, this is some hardcore indepth port scanning shizzle -
Interesting article on GigaOm this morning about Facebook benchmarking a 64-core chip from Tilera.
The X86 processor architecture has been in a dominant for so many years, not based upon any inherent sophistication, but more based upon chance and circumstance (see this vid for more details around the 48min mark), so it does feel about time for a new advancement in hardware architecture.
Be interesting to watch this space and see what develops..
Watch it here
Found this Facebook engineering video quite fascinating, a nicely detailed platform overview of FB's new real-time analytics system:
In the comment section, someone posted a link to a somewhat similar presentation from Twitter about their real-time solution::
(hello Johan!)
I keep coming across mentions of node.js, but wasn't sure what it was. This morning I've been watching some tutorials and reading up a little, and from what I understand it's basically a network server framework built on top of Google's V8 JavaScript engine, really an abstraction for socket programming. It's main advantage is speed and scalability, due to it being based on an event driven I/O model, rather than threaded, like most other languages or frameworks.
This video from node's creator, Ryan Dahl is a pretty funny and very informative introductory video. I'd recommend programming along with watching it:
Here's some further links:
http://nodejs.org/docs/v0.4.8/api/synopsis.html
http://howtonode.org/
This podcast is also a good source of information:
http://herdingcode.com/?p=299