verzweiflung – A story about how I hacked the internet

TL;DR
verzweiflung (GitHub) is a highly pragmatic solution to packet loss in China (and other places) as well as sharing bandwidth with other users altogether – though at a cost.

A class based system
About three years ago my employer started sending me to China, a magical place where the internet is strangely different. Besides not being able to reach some services like Google, which is quite interesting in itself, I discovered the classes of Chinese internet. For example, China’s leading telecommunications provider, China Telecom, offers ChinaNet and CN2. Domestically, ChinaNet is most sold. In its blend there’s a generous amount of domestic bandwidth with a hint of foreign traffic – just enough to justify calling it “internet”. CN2 is essentially the real deal, minus what the Great Firewall protects your mind from. Sadly, though, it’s hard to obtain and you’ll never see it in public places. (All public WiFi is ChinaNet.) Unlike its premium version, ChinaNet has high to extreme packet loss to Western locations. (I’ve regularly seen >40%.) This is actually really effective in convincing people to switch to domestic services – you simply can’t rely on anything else. Analysing the connection further I noticed that the loss was mostly asymmetric – inbound traffic often was extremely lossy, while outbound was better by about an order of magnitude. I assume the actual links are symmetric, but the Western interest in Chinese content isn’t great.

All TCP connections are equal…
Piecing together the whole picture, I was both baffled and somewhat intrigued at the same time. Packet loss isn’t something you often have to deal with these days, and if you do it’s usually something you can fix. Though as a foreigner in China their peerings and upstream providers are utterly out of reach. After ruminating about the issue for days, I concluded playing by the rules wouldn’t get me any further. I remembered TCP/IP congestion avoidance back at university, and how exponential back-off seriously impacts throughput in the presence of loss. It is supposed to be the internet’s egalitarian solution to packet loss – everyone slows down to the same speed in order to collectively minimise loss. So, naturally, I hacked the Linux TCP stack to get rid of it. The story could have ended here, but sadly I didn’t see the results I was hoping for. With TCP stalling and retransmitting packets at 350ms roundtrip latency and 50% loss, it still slowed to a crawl.

The diminishing returns of different levels of verzweiflung.

The Nuclear Option
After watching SSH connections time out one too many times, verzweiflung (the German version of despair) set in. I knew that unlike dropped packets, duplicate packets wouldn’t do any harm to TCP connections, though they can seamlessly take the place of a preceding dropped packet. Sending every packet twice at 50% loss results in only a 25% chance for both to be dropped, that’s 12.5% if you send them thrice and so on. Some quickly stitched together iptables rules confirmed my theory. This time it looked promising! Initially the results were still a bit worse than the math would suggest, which was caused by short loss spikes. But once I started mixing the packets in time I got a close match. In honour of its origin, I named my work verzweiflung.

One man’s loss is another man’s gain
The more I started playing with verzweiflung, the more I realised that I got way more than I bargained for. SSH connections were finally reasonably stable, SFTP was pushing several megabits a second again, but the real surprise was how it affected other things. I shared it with my colleagues and they noticed how communicating with verzweiflung in a crowded public WiFi resulted in considerably better speeds, while having everyone else come second. One of my friends reported his 25 Mbps connection turned into what felt like dial-up, when he ran a backup of one of my servers. (Oops.) Given enough bandwidth verzweiflung oversaturates your link way beyond where loss starts, meanwhile its redundancy chips in for what gets lost. And while verzweiflung thrives in a lossy environment, everything else backs off. It’s win-win, except for everyone else, for them it’s more like lose-lose. But that just turned into somebody else’s problem.

And why all of this is a really bad idea
verzweiflung is a prime real life example of the tragedy of the commons, the more people use it, the less effective it becomes. And in the end, everyone is dependent on it, wasting huge amounts of traffic, for less total throughput than before. And last but not least, verzweiflung actively violates internet standards – which is bad.

sigrok for OS X

pulse
Last weekend I spent some time compiling the open source signal analysis software sigrok and its GUI pulseview on OS X. While it still isn’t running perfectly smooth and there’s a major bug preventing saving ones captures, I still found it quite interesting to play with. It supports a variety of devices for data acquisition and if one doesn’t happen to own any of those, there’s always the demo mode to try. Loading firmwares for supported devices works as well, though they have to be dropped inside the application bundle. (PulseView.app/Contents/share/sigrok-firmware)

I’ll make sure to build a new one once the bugs are gone, but this is it for now: PulseView.dmg

Blobby Volley 2 for OS X

blobby
For quite a while I felt like I missed out while my colleagues were competing in a fun little game called Blobby Volley 2. While the game itself is actually open source software, I wasn’t able to find any build for OS X. So I took the Linux version of the source code and ported it over to OS X. In addition I’ve also come up with an icon of higher resolution since the default one wasn’t up to par.

Download me!

Source code on github: https://github.com/EliasOenal/blobby

PS:
You get around the Apple Gatekeeper warning by right-clicking “Blobby Volley 2” and selecting “Open”.

di-di-di-dah di-di-di-dah multimon-ng just learnt morse!

multimon-ng_morse
(Showing off the timing detection capabilities. Only the first few characters are wrongfully decoded while adjusting from 24WPM to 12WPM.)

Two weeks ago I thought writing a morse (aka CW) decoder would be a fun weekend project, and so I did it. As an additional challenge I decided to only rely on integer math, which forced me to rewrite core parts of multimon-ng. The resulting decoder also has basic threshold detection, as well as the ability to automatically adjust the timing. Note that there are several new parameters to adjust these options. In preparation for a morse encoder which I might write as well, I’ve revived multimon’s counterpart “gen” and gave it a nice qmake-file called “gen-ng.pro”. Building it works analog to multimon-ng.
Get it at GitHub: https://github.com/EliasOenal/multimon-ng

Update 7/9/2014:
The CW decoder has a bunch of parameters worth tweaking. Also make sure to bandpass the signal as closely as possible. If you plan on using it, have a look at Matt’s blog: Matt’s RF Stuff

low_pass
(I just felt like my low pass implementation was quite decorative.)

Press ‘L’ to loot Pandora

Given that Pandora is being Nazi about people’s backgrounds, (Or rather their IPs) I haven’t been able to use the service so far. Today though, I decided to hide behind a proxy and have a look at their offer. My client of choice was pianobar which is really lightweight, yet loaded with all of the features you could ever need – all but one, that is. Inspired by an older patch that sadly stopped working, I’ve added a feature that utilizes curl to download your favorite music. It integrates with pianobar’s history, meaning that you can fetch songs even after they’ve stopped playing. https://github.com/EliasOenal/pianobar

Screen Shot 2013-08-14 at 11.37.56 PM
The only downside I found is their actual service: you only get 64kbps for free. While their prices seem quite reasonable, paying isn’t an option for me. Given that I’m in Germany and they don’t own the distribution rights over here, they couldn’t legally license the content to me – even if they wanted to.

If you’re in need of a proxy to use Pandora, look no further! I’ve used the mediahint.com Chrome extension to sign up and then used their proxy for pianobar as well. Just grab their configuration file from here: https://mediahint.com/default.pac (The IPs are represented as 32bit numbers)

Update:
In addition to downloading songs from Pandora, my branch now also supports replaying previous songs, as well as restarting playback for the currently playing one.

RTL-SDR transformer mod

Some time ago I’ve tried the HF Mod (aka Direct Sampling Mod) on one of my dongles. The simplest way of doing it is by connecting one of the RTL2832U’s ADC (pin 1 or 2) inputs to an antenna which is exactly how I started. Shortly after that mikikg told me about using a transformer in order to generate a differential signal for the ADCs differential input, as well as boosting the signal and matching impedance. As a quick and dirty solution I desoldered a H16105DF 10/100Mbps Ethernet decoupling transformer from an old ADSL modem and used it. It actually matches the frequencies we care about quite well and in addition serves as a low pass filter. At the moment it is used in a 1:1 configuration, though 1:2 will probably perform better, thus I plan to try that as well. Using it together with an antenna made from a few meters of litz I receive many AM radio stations as well as lots of other signals.

Update:
Since someone over at reddit claimed that mikikg originally came up with using Ethernet transformers for the mod, implying I wouldn’t credit him properly, I feel like clarifying. On 10/3/12 I sent him a mail explaining that I had tried a variation of the HF-Mod using an Ethernet transformer and that the results were promising, I also attached a picture to that mail. Seemingly we then both blogged about it quite a while later, yet he did so before I did.

Update #2:
I have tried using the transformer in a 1:2 configuration and the signal actually got worse. That makes me wonder about the RTL2832U’s input impedance.

Power supply failures can be pretty annoying to find

Recently one of the four drives in my HTPC kept dropping out, first it happened once every few days, then after several hours. The weird thing was that it always came back after losing the SATA link and allowed me to mount it again. I first thought it was a problem of the SATA controller or cables so I switched the drives around, yet the same drive was still affected. I then assumed it was related to the drive which got me sidetracked for a while, till I routinely decided to reset the BIOS. Doing so I randomly had a look at the supply voltages, the 12V rail had dropped to around 10.2V.
Gladly I had a spare ATX PSU which I then switched in resolving all the problems. The PSU that failed was a one hung low 75W high efficiency one that came with a miniITX case I bought a while ago. It consists of a AC/DC power brick supplying 12V and a DC/DC switch mode board generating the whole variety of ATX voltages. Some of the DC/DC boards electrolytic capacitors were bulgy and I couldn’t even find a datasheet for their brand so I decided to replace them all with fresh low ESR ones. Needless to say that solved the problem and it now works fine (11.83V) again.
It really was an annoying experience and the symptoms were highly misleading. After all the drive was fine, and it was just the manifestation of a completely different problem. From now on I will routinely check lm-sensors for voltages as well.

Continue reading ‘Power supply failures can be pretty annoying to find’ »

The power of TNT is at your disposal

TNT stands for Thumb2 Newlib Toolchain, a GCC/Newlib based toolchain project I started quite a while ago. Trying various other toolchains like CodeSourcery, summon-arm-toolchain and YAGARTO, I realized none of them really seemed to have the smallest ARM Cortex-M microcontrollers in mind; They were wasteful with RAM as well as flash memory. TNT has been optimized for small unhosted (bare-metal) microcontroller targets and thus size optimizations have been the primary goal. Especially the C-library Newlib has been configured for the least possible size which makes using it (e.g. printf()) much more affordable. I have seen size reductions of over 30% compared to other GCC based toolchains when compiling the TNT_Example project.

The following architectures are supported:
armv6s-m (cortex-m0/cortex-m1) armv7-m (cortex-m3) armv7e-m (cortex-m4/cortex-m4f including FPU and DSP instruction support)

The TNT_Example project is targeted at beginners using STM32 microcontrollers, it plays well with TNT and other GCC based toolchains. It has an example on how to implement the Newlib stubs to get syscalls like malloc() and printf() working. For the STM32F4 discovery board you can just compile it and get the lights blinking. For other targets you’ll have to slightly adjust the makefile. https://github.com/EliasOenal/TNT_Example

TNT precompiled downloads:
Linux x64
OSX x64

Source:
https://github.com/EliasOenal/TNT

Continue reading ‘The power of TNT is at your disposal’ »

OS X port of the awesome gqrx SDR software [Update 07/01/2013]

A while ago I already managed to compile gqrx on OS X, but then Alexandru Csete decided to move on to a new audio backend called pulseaudio. Sadly there is no proper pulseaudio port for OS X. Thus I bring you my branch of gqrx 2.1 which runs quite well on OS X.

This release bundles all the rtl-sdr, gr-osmosdr, GNU Radio, libUSB, boost and Qt dependencies allowing it to work out of the box. Though feel free to contact me in case you encounter problems. It has been reported to run fine on Snow Leopard (10.6), OS X Lion (10.7), OS X Mountain Lion (10.8) and OS X Mavericks (10.9). Sadly Leopard (10.5) isn’t supported due to Apple breaking binary compatibility on 64bit binaries for some reason.

Download the ready to use DMG/APP here:
gqrx_8.dmg – (07/01/2013) Latest GIT. Some people are reporting FFT issues with this build, in case you encounter those try the one below and leave a comment. (shasum: 66b9f6952aff5cb0bc995556c633b3cc7f5d990e)

Old versions:
gqrx_7.dmg – (02/03/2013) Up to date with GIT. I’ve added a deadlock prevention feature in case your settings keep gqrx from starting up properly. After a crash/forced-close it asks you whether you want to reset all your settings to default. This should help you guys with invalid rtl_tcp settings and the like. I have also fixed rtl_fm to now properly work on OS X (It used to only output noise), as well as multimonNG which now supports umlauts for POCSAG decoding. So check out the bundled command-line utilities! 🙂 (e.g. /Applications/Gqrx.app/Contents/MacOS/rtl_test -t)
Continue reading ‘OS X port of the awesome gqrx SDR software [Update 07/01/2013]’ »