bert hubert finally blogs

The general mediocrity of the world

Posted by bert hubert Thu, 13 Apr 2006 14:39:00 GMT

I like statistics, and I disagree with the famous line “lies, damn lies and statistics”. Without statistics, all we have are data points, and no facts.

One of the better books if you really want to get into this kind of thing is Statistics for Technology.

Ok, why do I mention this? Have you ever noticed how mediocre many things are? Or even, many people? And specifically, programmers and their code! A lot of it only just passes the ‘not obviously broken’ mark.

At first I found this very upsetting, as if people simply don’t want to do a better job. And although this is partially true, we shouldn’t get mad because it is just.. statistics. And why? Because the most common skill level is bound to be somewhere around the median, which is to say mediocre. By definition in fact. Hence all the crappy code and products out there.

(I know there are distributions where this wouldn’t be the case, see the fine book mentioned above).

My grandfather used to say, never get angry at things. And as an addendum, this holds to an even greater extent for numbers. So, let go of your anger. I know I should.

Here goes.

Go read the RFC already

The ‘fine’ company E-Tech just cost me three days of my life in worry. Like I don’t worry enough already. As you may know, the PowerDNS recursor powers the internet access of hundreds of thousands of people these days. And while we’ve tested PowerDNS by sending it billions of recorded packets, and verifying the output against Other Nameservers, some problems cropped up.

Slowly a pattern emerged - some customers who relied on their (ADSL) routers of varying brands to relay their DNS traffic lost the ability to resolve domains the instant we switched them over to PowerDNS, cutting them off from the internet.

Sometimes rebooting the router helped, but often not, or not for long.

So we tried to get test data.. and there was none. Much searching ensued, tcpdump running on heaps of servers. And there was no data. When customers reported the problem, there were no packets to inspect.

But one thing became clear - PowerDNS did not compress the first part of the DNS answer. Compression is an optional part of DNS that allows more data to fit in a packet, and PowerDNS does its best to do that.

However, a programming oversight meant that this first part was not compressed - wasting a few bytes.

And lo, several DNS proxies in crappy routers had decided that compression was not just optional but mandatory. And would crash upon receiving a non-compressed first answer. And not relay any DNS packets anymore.

Which explained why we didn’t find any packets to analyse.

So somewhere in the dreaded E-Tech router, there is code that probably doesn’t even understand compression, but assumes that the magic bytes ‘c0 0c’ are a sort of “start of answer” marker. Which is the impression you might get if you’ve only looked at a few packets and decided to parrot your way out.

Oh well.

Thanks to Paul Duivestein for helping debug this problem, which also affects older firmware revisions of Thompson 510 modems.

Other news

Ok, it is not all debugging. I added a small hack to make the PowerDNS recursor benefit from an additional processor, and boy, does it work well. I implemented no locking, no concurrency, nothing whatsoever. What basically happens is that you run two wholly separate recursors that share one socket. This does double your memory needs, but also completely doubles performance. I’ve tried to max out a dual 3GHz Xeon, but I need to improve my replaying code. Some extrapolation appears to show you could run a sustained 15 to 17,000 dns questions/second on this machine.

Oh, and about my own mediocrity :-) The PowerDNS recursor ran into a lot of broken answers trawling the net for data, which generated heaps of debugging output. Seeking to silence this flood I investigated the broken answers coming from the authoritative servers on the internet. And 95% of them came from.. other PowerDNS deployments. It turns out that the authoritative PowerDNS server simply truncates answers at 512 bytes, and sets the ‘truncated’ bit. This is within the RFC as the RFC doesn’t specify how to truncate.

But the recursor code balked at the partial record at the end of this truncated packet. So I fixed that. I previously harboured angry thoughts at the servers emitting packets that ugly. Oh well.

And it doesn’t stop there. At one point moving over the recursor to the wonderful MOADNSParser architecture, I needed some code to parse incoming answers to figure out their DNS ID, label and status, so I could wake up the proper MThread, which would then fully parse the packet. And what did I use to extract these small bits of information? The full blown MOADNSParser of course. This in turn means that each answer packet gets parsed twice.

I now no longer use the full blown parser to extract this information and this alone reduced CPU load by 20%. See, I know all about mediocrity.

Oh well.. I hope I pass the ‘not obviously broken’ mark.