Posted by bert hubert
Thu, 08 Jun 2006 21:33:00 GMT
Ok, I’m back from Egypt. Had a stellar time, if you have a group that wants to travel to Egypt, drop me a line, the people that organized this for us are very good.
We’ve visited the pyramids, the sphinx, the egyptian museum (all in Cairo), the unfinished obelisk, Philae, the high dam (all in Aswan), Kom Ombo, Edfu (en route to Luxor), valley of the kings and Karnak (in Luxor). All very impressive.
While I was away, PowerDNS 3.1.1 has held up well, although some minor bugs are now known about. Most of these involve the new zone forwarding, and authoritative zone hosting features, which are not in wide use.
Each consecutive PowerDNS recursor release has attracted the attention of larger internet providers, but we’ve reached the end of the line now. There are none bigger than those trialling PowerDNS now. Additionally, the previous weeks have seen actual deployment to over ten million internet users.
This is a very humbling thought. The PowerDNS company and I have worked very hard at making PowerDNS the most secure, simple and capable recursor out there. In the brief time since XS4ALL funded us to add the features a modern ISP needs, the takeup has been tremendous.
It is good to be back!
Posted in Linux, PowerDNS, Life | 3 comments | no trackbacks
Posted by bert hubert
Wed, 24 May 2006 04:41:00 GMT
PowerDNS 3.1 turned out to contain a brown paper bag bug that in retrospect should not hit too many people, but still. So I rushed out 3.1.1, which always leaves me with a bad feeling.
Furthermore, I’m off to Egypt for two weeks. While other people do work on PowerDNS, development will come to a nearly complete halt.
So here’s to hoping that 3.1.1 fixed more bugs than it caused..
See you in two weeks!
Posted in Linux, PowerDNS, Netherlabs, Life | 4 comments | no trackbacks
Posted by bert hubert
Fri, 12 May 2006 22:08:00 GMT
Talk about embarrassing. You may know I’m busy working on a draft-draft RFC (it becomes an ‘Internet-Draft’ once submitted) about making DNS safer through some implementation and operational guidelines (see dns-anti-spoofing.html and
dns-anti-spoofing.txt).
While writing this document, I decided to add a section on the ‘birthday paradox attack’. Reported in 2002, this is a curious mathematical phenomenon that makes spoofing a nameserver vastly easier to do.
So I wrote down the specification:
Given the above, a recursor MUST:
* Use a new random source port from its available
range for each outgoing query
* Make full use of all 16 bits of the ID field
* Assure that its choices of port and ID cannot
be predicted by an attacker having knowledge of
its (pseudo-)random generator
* Take measures to prevent having multiple equivalent
questions outstanding to any authoritative server
Which is all fine. Except that PowerDNS did not adhere to the bit about equivalent outstanding questions! PowerDNS contains a general system that prevents heaps of identical queries from leaving the server, but that doesn’t translate well into ‘standardese’, you’d get something like ‘recursors MUST have a system that sort of prevents most of the identical queries’.
So, I added ‘query-chaining’ to PowerDNS, which detects this situation and puts an MThread to sleep when it tries to send out a duplicate question. When the answer to the initial question arrives, they all get woken up.
Due to the throttling code already in place, and the source port randomisation, this does not improve our security significantly, but at least I’m now in compliance of my own draft-draft RFC :-)
Code is linked here.
Posted in PowerDNS, Netherlabs | 1 comment | no trackbacks
Posted by bert hubert
Tue, 09 May 2006 20:15:00 GMT
I’ve long been a somewhat active member of the relevant DNS mailing lists, ‘namedroppers’ and ‘dnsop’, both affiliated with the IETF DNS workgroups.
I consider myself a bit of an outcast in the DNS community as I don’t sing the praises of DNSSEC, nor BIND, but I suspect this is not entirely fair as there are quite a number of people who are far more outcast than I am. So I suspect I’m on the fringe of the DNS community in the sense that I incidentally take part in useful email discussion, either on list or privately with relevant parties.
I recently called upon nameserver authors and operators to either upgrade their nameserver so it performs adequate anti-spoofing measures, or switch to a nameserver implementation that does (like tinydns or of course PowerDNS).
This call fell on very deaf ears it appears. The BIND people promised to look into it but as noted then, without an apparant sense of urgency. Not a lot has happened since, except that I’ve reiterated my recommendation privately to a number of relevant people.
In the meantime, I’ve been told the Microsoft nameserver is about 4 times easier to spoof than BIND, but I’ve been unable to verify this.
So, I did what I never thought I’d do, I wrote something intended to be an RFC. In short, this RFC specifies that a recursor MUST implement adequate anti-spoofing measures, and details what this entails.
Read all about it as old school text or rendered as pretty HTML. The RFC-compliant output is made possibly by the interesting but quirky tool xml2rfc.
I’ll spend some more time polishing the document before submitting it as an Internet Draft. I also need to figure out the correct procedure to set things in motion.
I sincerely hope nameservers that are easy to spoof clean up their act quickly, hopefully even before my draft hits the standards track.
Posted in PowerDNS, Netherlabs | 12 comments | no trackbacks
Posted by bert hubert
Sat, 06 May 2006 21:58:00 GMT
Welcome back after this 9-day hiatus from my Blog!
Ok, what has happened. I had two good experiences with local electronics stores here in Delft. Goris was unable to provide me with the proper cable to hook up my shiny new WiFi directional antenna, but they referred me to HEC, which did have the components to make the cable. My skills with the soldering iron are humorous at best. However the people at HEC kindly offered to make the cable for me! So now I finally have a working combination of antenna, cable and adapter. And to make things perfect, Goris allowed me to test my new WiFi card to verify Linux compatability. Luckily it all works. I hope to hook up pahu tomorrow.
Slight damper on today is that I was fined for driving my bicycle through a street here in Delft that turned out to be for pedestrians only. 30 euros too. I normally am all in favour of the rule of law but this makes little sense. It is fortunate therefore that the actual fine contained a number of errors which I am sure invalidate it, so I wasted no time in drafting a written protest. I’m not usually like this but I was pissed of at the inanity of this fine.
PowerDNS & Windows
As staunch a supporter as I am of Open Source, my technology wants to go places. So, I downloaded the ‘free’ version of Visual Studio Express 2005 from Microsoft. And a fine compiler it is! I had fixed a bunch of initial incompatabilities using the (also fine) Minimalistic GCC for Windows. I think this is the first Microsoft C++ compiler that can really be taken seriously. VC++ debugging mode found two real bugs in PowerDNS, which motivated me to turn on the ‘debugging mode’ of the G++ libstdc++ as well, which uncovered two further bugs!
This strengthens my feelings that porting to different platforms helps uncover bugs which aren’t (yet) a problem but might be.
Ahu’s quick guide to porting to windows:
- Use VC++ 2005, earlier versions have a lot more problems with constructions g++ accepts. It also appears that VC++ 2005 is smart with respect to UNIX/DOS line endings.
- Separate the really different things to different files, which share one header file. Don’t make #ifdef soup!
- Make a single include file that includes OS-dependent include files (like
windows.h).
- On windows, one can only write and read from sockets using
send(to) and recv(from). As these functions work for UNIX as well, use these functions exclusively on sockets.
- To close a socket under windows, you need
closesocket() and not close. Candidate for the file mentioned under 1.
- Windows has different
errno traditions. All network (‘winsock’) related errors need WSAGetLastError(). See here.
- Use ‘Tortoise’ Subversion for revision control, integrates really well with both Windows and UNIX. Also smart about line endings.
- If, as for me, your prime development platform is UNIX, install the MINGW crosscompiler so you can easily verify the code at least compiles for Windows. This helps prevent code-rot at an early stage.
- Get a Windows buddy :-) Many thanks to Michel Stol, who is far more at home in Windows than I am.
PowerDNS 3.1
I hope to release PowerDNS 3.1 shortly, and make things settle down a bit then. Since the previous blog post, I added full blown IPv6 outgoing support, with IPv6 achieving full parity - any IPv6 nameserves that are faster than their IPv4 partners will receive more queries.
The ‘–export-etc-hosts’ stuff also works fine now, which should allow many networks to simply run unconfigured, save for that option, and have everything Just Work.
For more, see here.
Posted in Linux, PowerDNS, Netherlabs, Life | 3 comments | no trackbacks
Posted by bert hubert
Thu, 27 Apr 2006 19:31:00 GMT
We’ve been looking for a new house lately, but this has not been easy. The Netherlands has been experiencing a housing bubble for the past decade, so even a small house costs an arm and a leg. A number of possible places were sold in the brief period between making an appointment and actually going round to visit. Very frustrating.
1.4 billion queries
I started an endurance test some time ago to really stress out the recursor. This test has now reached 1.4 billion queries. This means we still have 3 billion queries to go to hit the magic 2^32.
PowerDNS 3.0.1 appears to hold up well. There are some small problems on big endian platforms (ultrasparc), which are solved in subversion, and I uncovered an obscure form of misconfiguration (having a nameserver with multiple IP addresses, one of them being lame) we didn’t deal with. Other nameservers don’t either, so it doesn’t really matter. See if you can resolve ‘www.nl.netherlabs.eu’. If you can, chances are you are running a very recent PowerDNS :-)
Crossing over
The PowerDNS recursor is a pure recursor, or at least, used to be. I literally spent years thinking (on and off) about how to make PowerDNS authoritative and recursive at the same time without losing the clean design and today I figured out how to do it.
It turned out the proper way is to insert a hook in the call that figures out the best nameserver to ask a question. If we are authoritative for a domain, we send back an empty nameserver which means ‘we know, don’t go out’.
When the time comes to go ask that nameserver, the emptiness is recognized, and a call is made to the ‘out of band’ resolver. This delivers a vector of DNSResourceRecords, just like a remote nameserver would. The rest of PowerDNS does not ‘know’ it is parsing self-generated data.
This has the downside that we cache our own data. But compared to the elegance of keeping the rest of the nameserver unchanged, this is a small price to pay.
In the same place, we can also insert a ‘forwarder’ nameserver, whereby we can point a domain towards an external authoritative nameserver.
Finally, built on the authoritative infrastructure, I added ‘–export-etc-hosts’. Quite a number of people have asked me if there were an easy way to have their recursor serve a small number of domains. There is an obvious place to get this data, /etc/hosts. ‘–export-etc-hosts’ does the obvious and generates full zones for each entry in /etc/hosts, making them available for all your clients.
This in turn means that a large number of people now have no reason anymore not to run the PowerDNS recursor, and benefit from its performance and superior anti-spoofing measures :-)
To test, head to this posting to the mailing list and download away!
Posted in Linux, PowerDNS, Netherlabs, Life | 2 comments | no trackbacks
Posted by bert hubert
Sun, 23 Apr 2006 20:44:00 GMT
There has been a recent paucity of pizza related posts, but I did manage to employ my fine pizza oven yesterday, this time to make Nan bread. Nans are usually made in a tandoor, a blisteringly hot clay oven. And while my passion for good food is well known, having a clay oven (traditionally submerged in the earth if I understand correctly) is going a bit far. But I do have a bilsteringly hot pizza oven. And indeed, I can now finally procude Nans that are somewhat crisp on the outside, yet chewy on the inside, exactly the way I like them.
PowerDNS 3.0 release followup
Since the release of PowerDNS 3.0 last Thursday, some big users have switched over. This has led to a good trickle of tiny bugs which were all addressed quickly. To note:
- The Debian Sarge 3.0r1 default kernel, 2.6.8, claims to support epoll but in fact appears not to do so. The epoll multiplexer now does a better job of testing itself at runtime, and falling back to select if needed.
- No difference was made between NXDOMAIN and NXRRSET - probably not very relevant, but a technical violation. This was a regression caused by an earlier fix. No good deed goes unpunished!
- Filedescriptors 0, 1 and 2 are closed when going to the background, but I can’t 100% guarantee there is no logging output to fd 2 during operations. These descriptors are now dupped to
/dev/zero.
- Bad handling of malformed EDNS0 packets.
- Failed compilation on Solaris 10 i386, because Solaris decides to #define DS, messing up our support of the DS record type.
I’ll wait a few more days and do a 3.0.1 release with nothing but minimal changes that all address real problems. There is one report of an unexplained crash on Solaris around that I’d like to solve, but as there is only one report and it can’t be reproduced, this might be hard.
I have an endurance test running of a single PowerDNS instance which has processed half a billion packets so far, I intend to keep it running until it passes the 32-bit wraparound mark, just to check if my code is properly using the 64-bit variables I pass it.
PowerDNS Design and Engineering
I sat down for a few hours and documented the inner workings of the PowerDNS recursor here. I know I’ll be reading this documentation myself three months from now, I swap out memory really quickly. You don’t need to read this document in order to use PowerDNS, but if you want to contribute, it should be very helpful.
Posted in Linux, PowerDNS, Netherlabs, Life | 5 comments | no trackbacks
Posted by bert hubert
Fri, 21 Apr 2006 07:19:02 GMT
Well that came at a very bad time. Yesterday on the day of the PowerDNS Recursor 3.0 release, our cabinet over at Level3 dropped off the net. Much debugging later it appeared one of the customer hosts was compromised and filling the ethernet with tiny packets at line rate.
Apologies. The owner of said (Windows) machine will be coerced into providing an evening of drinks and entertainment to compensate our nightly labours.
Posted in Linux, PowerDNS, Netherlabs, Life | 1 comment | no trackbacks
Posted by bert hubert
Thu, 20 Apr 2006 14:27:00 GMT
PowerDNS Recursor version 3.0 has just been released, I’m relieved to say. Read all about it in the release notes.
To quote:
We consider this version of the PowerDNS recursor to be the most advanced resolver publicly available. Given current levels of spam, phishing and other forms of internet crime we think no recursor should offer less than the best in spoofing protection. We urge all operators of resolvers without proper spoofing countermeasures to consider PowerDNS, as it is a Better Internet Nameserver Daemon.
As mentioned previously, the new recursor is at least 64000 times harder to spoof than previous releases. Briefly, spoofing involves feeding a nameserver fake answers by making educated guesses about what questions it is asking. If one guesses right quickly enough, the nameserver believes the ‘spoofed’ answer, and email as well as web traffic can be redirected to malicious sites. This is obviously a big problem.
Previously, of major nameservers, only dnscache (tinydns) and presumably Nominum CNS offered decent protection against this phenomenon. PowerDNS has now joined this club, and goes further by not just being harder to spoof but by also detecting (and shielding) when an attempt is made.
I took this up with the people who program BIND, because it worries me that the dominant recursor is also the recursor that is easiest to spoof, and they are pondering improving BIND in this respect. But what I miss is the sense of urgency.
The internet has turned into a hostile place. Not too many years ago it was common for servers to heplfully relay other people’s email, these days such a server (known as an open relay) becomes a conveyor of gigabytes of spam within a few hours.
Running a spoofable recursor is the moral equivalent of running an open relay - it allows spammers and other miscreants to use the internet for their ends, on your dime. In fact, an open relay is less of a worry - relaying spam is simply a nuisance, but having your customers redirected to malicious sites, or letting their email be diverted is far worse!
So I hereby call on all nameserver operators and software authors to follow the lead of Dan Bernstein’s dnscache and now PowerDNS, and work dilligently on rooting out spoofing. Either change your recursor to PowerDNS or dnscache, or convince the authors of your vulnerable nameserver to clean up their act.
To read more, head over to this article on dns spoofing, or read my previous writings on this subject
Posted in PowerDNS | Tags pharming, phishing, spam, spoofing | no comments | no trackbacks
Posted by bert hubert
Wed, 19 Apr 2006 15:40:00 GMT
Powered up the Sun CoolThreads T2000 again today to work on first class Solaris support for PowerDNS. As mentioned previously, because of the new anti-spoofing measures, PowerDNS needs to listen to hundreds or sometimes thousands of sockets. The traditional UNIX approach was to tell the kernel which sockets hold your interest, call select(2), and look at the sockets it tells you are active. And then you have to do the whole thing again, ie, report all sockets to the kernel again.
All modern UNIXes come with a better solution: tell kernel which sockets hold your interest, ask kernel which are active, do work, ask kernel which are active. In other words, there is no need to setup everything for each packet.
I implemented epoll and kqueue for Linux and FreeBSD yesterday, today I did Solaris completion ports. Some things to note:
- A ‘Completion Port’ does not survive
fork(2). So create the port after forking.
- The
port_getn(2) function takes two parameters to specify how many events you want to receive, a minimum and a maximum. This is different from what kqueue and epoll do. The manpage does not make this overly clear. (UPDATE: ok, it does, I can’t read)
- Contrary to kqueue and epoll, once you’ve received an event from the port, you need to add back the socket if you are still interested. I think this is a slight optimisation for PowerDNS as the common case is indeed to remove a socket once an event has hit.
To really get all those cores and strands working, I split PowerDNS 24 ways today and tried to benchmark it. This appears to have worked as it fully saturated my fine xs4all ADSL connection :-) If I want to fully test this server I may need to get it hosted somewhere with real bandwidth.
I released the third 3.0 pre-release of PowerDNS today (including .deb and .rpm!), if you have a chance, please test it.
One large deployment has already moved to this release and I’m very pleased to note that due to the multiplexer work, the additional anti-spoofing sockets have no measurable impact on the CPU load.
Posted in Linux, PowerDNS, Netherlabs | Tags solaris, sun, t2000 | no comments | no trackbacks
