Posted by bert hubert Wed, 24 May 2006 04:41:00 GMT
PowerDNS 3.1 turned out to contain a brown paper bag bug that in retrospect should not hit too many people, but still. So I rushed out 3.1.1, which always leaves me with a bad feeling.
Furthermore, I’m off to Egypt for two weeks. While other people do work on PowerDNS, development will come to a nearly complete halt.
So here’s to hoping that 3.1.1 fixed more bugs than it caused..
See you in two weeks!
Posted by bert hubert Sat, 06 May 2006 21:58:00 GMT
Welcome back after this 9-day hiatus from my Blog!
Ok, what has happened. I had two good experiences with local electronics stores here in Delft. Goris was unable to provide me with the proper cable to hook up my shiny new WiFi directional antenna, but they referred me to HEC, which did have the components to make the cable. My skills with the soldering iron are humorous at best. However the people at HEC kindly offered to make the cable for me! So now I finally have a working combination of antenna, cable and adapter. And to make things perfect, Goris allowed me to test my new WiFi card to verify Linux compatability. Luckily it all works. I hope to hook up pahu tomorrow.
Slight damper on today is that I was fined for driving my bicycle through a street here in Delft that turned out to be for pedestrians only. 30 euros too. I normally am all in favour of the rule of law but this makes little sense. It is fortunate therefore that the actual fine contained a number of errors which I am sure invalidate it, so I wasted no time in drafting a written protest. I’m not usually like this but I was pissed of at the inanity of this fine.
As staunch a supporter as I am of Open Source, my technology wants to go places. So, I downloaded the ‘free’ version of Visual Studio Express 2005 from Microsoft. And a fine compiler it is! I had fixed a bunch of initial incompatabilities using the (also fine) Minimalistic GCC for Windows. I think this is the first Microsoft C++ compiler that can really be taken seriously. VC++ debugging mode found two real bugs in PowerDNS, which motivated me to turn on the ‘debugging mode’ of the G++ libstdc++ as well, which uncovered two further bugs!
This strengthens my feelings that porting to different platforms helps uncover bugs which aren’t (yet) a problem but might be.
Ahu’s quick guide to porting to windows:
windows.h).send(to) and recv(from). As these functions work for UNIX as well, use these functions exclusively on sockets.closesocket() and not close. Candidate for the file mentioned under 1.errno traditions. All network (‘winsock’) related errors need WSAGetLastError(). See here.I hope to release PowerDNS 3.1 shortly, and make things settle down a bit then. Since the previous blog post, I added full blown IPv6 outgoing support, with IPv6 achieving full parity - any IPv6 nameserves that are faster than their IPv4 partners will receive more queries.
The ‘–export-etc-hosts’ stuff also works fine now, which should allow many networks to simply run unconfigured, save for that option, and have everything Just Work.
For more, see here.
Posted by bert hubert Thu, 27 Apr 2006 19:31:00 GMT
We’ve been looking for a new house lately, but this has not been easy. The Netherlands has been experiencing a housing bubble for the past decade, so even a small house costs an arm and a leg. A number of possible places were sold in the brief period between making an appointment and actually going round to visit. Very frustrating.
I started an endurance test some time ago to really stress out the recursor. This test has now reached 1.4 billion queries. This means we still have 3 billion queries to go to hit the magic 2^32.
PowerDNS 3.0.1 appears to hold up well. There are some small problems on big endian platforms (ultrasparc), which are solved in subversion, and I uncovered an obscure form of misconfiguration (having a nameserver with multiple IP addresses, one of them being lame) we didn’t deal with. Other nameservers don’t either, so it doesn’t really matter. See if you can resolve ‘www.nl.netherlabs.eu’. If you can, chances are you are running a very recent PowerDNS :-)
The PowerDNS recursor is a pure recursor, or at least, used to be. I literally spent years thinking (on and off) about how to make PowerDNS authoritative and recursive at the same time without losing the clean design and today I figured out how to do it.
It turned out the proper way is to insert a hook in the call that figures out the best nameserver to ask a question. If we are authoritative for a domain, we send back an empty nameserver which means ‘we know, don’t go out’.
When the time comes to go ask that nameserver, the emptiness is recognized, and a call is made to the ‘out of band’ resolver. This delivers a vector of DNSResourceRecords, just like a remote nameserver would. The rest of PowerDNS does not ‘know’ it is parsing self-generated data.
This has the downside that we cache our own data. But compared to the elegance of keeping the rest of the nameserver unchanged, this is a small price to pay.
In the same place, we can also insert a ‘forwarder’ nameserver, whereby we can point a domain towards an external authoritative nameserver.
Finally, built on the authoritative infrastructure, I added ‘–export-etc-hosts’. Quite a number of people have asked me if there were an easy way to have their recursor serve a small number of domains. There is an obvious place to get this data, /etc/hosts. ‘–export-etc-hosts’ does the obvious and generates full zones for each entry in /etc/hosts, making them available for all your clients.
This in turn means that a large number of people now have no reason anymore not to run the PowerDNS recursor, and benefit from its performance and superior anti-spoofing measures :-)
To test, head to this posting to the mailing list and download away!
Posted by bert hubert Sun, 23 Apr 2006 20:44:00 GMT
There has been a recent paucity of pizza related posts, but I did manage to employ my fine pizza oven yesterday, this time to make Nan bread. Nans are usually made in a tandoor, a blisteringly hot clay oven. And while my passion for good food is well known, having a clay oven (traditionally submerged in the earth if I understand correctly) is going a bit far. But I do have a bilsteringly hot pizza oven. And indeed, I can now finally procude Nans that are somewhat crisp on the outside, yet chewy on the inside, exactly the way I like them.
Since the release of PowerDNS 3.0 last Thursday, some big users have switched over. This has led to a good trickle of tiny bugs which were all addressed quickly. To note:
/dev/zero.I’ll wait a few more days and do a 3.0.1 release with nothing but minimal changes that all address real problems. There is one report of an unexplained crash on Solaris around that I’d like to solve, but as there is only one report and it can’t be reproduced, this might be hard.
I have an endurance test running of a single PowerDNS instance which has processed half a billion packets so far, I intend to keep it running until it passes the 32-bit wraparound mark, just to check if my code is properly using the 64-bit variables I pass it.
I sat down for a few hours and documented the inner workings of the PowerDNS recursor here. I know I’ll be reading this documentation myself three months from now, I swap out memory really quickly. You don’t need to read this document in order to use PowerDNS, but if you want to contribute, it should be very helpful.
Posted by bert hubert Fri, 21 Apr 2006 07:19:02 GMT
Well that came at a very bad time. Yesterday on the day of the PowerDNS Recursor 3.0 release, our cabinet over at Level3 dropped off the net. Much debugging later it appeared one of the customer hosts was compromised and filling the ethernet with tiny packets at line rate.
Apologies. The owner of said (Windows) machine will be coerced into providing an evening of drinks and entertainment to compensate our nightly labours.
Posted by bert hubert Mon, 17 Apr 2006 08:23:00 GMT
We visited my father on holiday last Saturday, hoping to get his wireless internet up and running. Sadly it turns out there are around 3 different antenna connectors and I think around 3 on the computer side of things. The antenna is very spiffy though. Will investigate this week how to best provide wireless - the non plus ultra solution would be a repeater, which would save heaps of cables.
While there, I was tasked with getting some eggs from the chicken coop. Being a city person, I wondered if the chickens would be ok with that and it took some convincing to get me into the cage :-) It suddenly hit me that eggs do not come out of a box but out of the back of a chicken!!! Gross! Oh well.
Then we saw a stork, which is rare enough in The Netherlands. And then another one, until there were four. Impressive birds.
As mentioned previously, I implemented a lot of anti-spoofing technology in PowerDNS recently. This did come at some operating system cost - listening on so many sockets at the same time exposes shortcomings in the traditional unix select function. Luckily, modern Unixes like FreeBSD, Linux and Solaris all come with replacements, called kqueue, epoll and ports respectively. I wrote a simple multiplexer and implemented support for select, kqueue and epoll, and it all appears to work. I’ll do Solaris once I have a chance.
This also moved all different packet handlers to separate functions, everything previously was within one big loop in main().
With the major exception being the ability to serve from IPv6 (AAAA etcetera works of course), the PowerDNS recursor is now feature complete for 3.0, so I released version 3.0-pre2 (freshmeat).
Posted by bert hubert Fri, 14 Apr 2006 20:48:00 GMT
Went for dinner yesterday with a friend of mine at one of the Indian restaurants here in Delft. This place is marvelous. It has had a non-working phone number listed on its window for the past 8 years. The music is probably not agreeable even for people from India. The beer tends to taste a bit funny. The entrance is dark, and looks like it has been burgled repeatedly. The staff is clumsy. But the food! Oh my.
Ate too much and went home a bit sleepy.
One of the interesting bits about authoring an open source program is that you know both a lot and nearly nothing about your customers. Sometimes PowerDNS users share everything with me and other developers. I’ve been mailed more root passwords than I care to remember (I have a fully functioning PGP key btw, please use it if you trust me with passwords!).
On the other hand, there are a lot of ‘stealth users’ who don’t come out of the closet. I tend to hear from them only if they hit a problem - which is rare.
So imagine my surprise yesterday when one of the larger access providers in Europe, with a double digit market share in their large country, suddenly announced they’d switched all their nameservers to PowerDNS. 1.3 million additional homes served by my humble code.
I can tell you, that rattles me. Especially since DNS is absolutely 100% vital to using the internet.
So, that inspired me to take the last step in attempting to make PowerDNS the best recursor on the planet.
If you can fool DNS, you can fool a user. DNS is the phone book of the internet, if you manage to give out false data, browsers will head to the wrong servers. Same goes for email. All very bad.
The worse news is that DNS is a breeze to “spoof”, in other words, it is easy to slip in bad data. I set up a somewhat contrived network here today and I was able to spoof both BIND 9 and PowerDNS in less than two seconds. I must admit that the conditions I tested under were highly ideal, but nothing that can’t be achieved in the real world with concerted effort.
And given the huge number of people I now feel responsible for, this is unacceptable.
One of the brightest people I know, Dan J. Bernstein, also writes nameservers. He can be very stubborn and opinionated, but some of his ideas are first rate. You have him to thank (in part) for today’s more liberal cryptography research climate as well. So, I took a lot of inspiration from his work. Read more below.
To spoof a nameserver, one needs to know three things:
You can generally figure out 1) pretty easily, especially if you can force a nameserver to make queries. 2) is easy if the network end-point doesn’t change. 3) can be dealt with by scanning all 65536 ids.
I reduced all three factors today:
I made the PowerDNS recursor default to not accepting questions from the internet at large. This reduces the chances of a spoofer to force questions.
I copied Dan J. Bernsteins system of using a new random network end-point for each question, which means you’ll have to try to guess this end-point too, just like you have to guess the ID. This does put a heavy load on the OS as we now have to listen to perhaps thousands of ports! So I made this optional, but on by default.
If the recursor sees more than 20 failed guesses for the ID, it considers the whole query timed out. I spent a heap of time thinking how to do this elegantly, I had to lie down at one point and close my eyes briefly. This may look like a sinful mid-day nap but don’t let appearances fool you! The solution is to only do the accounting once a packet with a proper ID is in, and deal with it then, and not keep a list of failed guesses.
This was literally the last major piece of PowerDNS that was not ‘best of breed’. Now all I need to do is clean up the code a tad and integrate full IPv6 support, and it should be Perfect.
Wonder what I’ll do then though :-)
I haven’t mentioned my good friend and wife Mirjam enough, and she’s complained a bit. So for the record, Mirjam has been doing a fine job, or at least making a valliant attempt, at making me leave the computer every once in a while. And I think she still believes me a bit when I say PowerDNS is ‘nearly done’. Now I have to believe it too.
Posted by bert hubert Sun, 09 Apr 2006 22:18:00 GMT
Spent some time this weekend setting up my father’s holiday location, which was very good. Time flew by and we had to race to pick up my car, which was on a company property that would be locked up at night. His new place has Wifi, but he is too far removed from the access point. I ordered a very nice directional antenna from the Wifi Shop, hope it shows up soon. Aiming it should be hard work, as getting line of sight requires mounting the antenna on a large pole. Should be worthwhile though!
As regular readers of this blog will know, I’m currently working hard on making the PowerDNS recursor the fastest and most ablest nameserver on this planet.
To this end, I’ve done a lot of micro-optimisation, which means trying to make individual functions as fast as possible. Time and time again however, I’ve discovered that gcc 4.1 does a very fine job already.
Until I started finding very major optimisations, which remind me of one of my favorite movies, War Games, which has a famous quote, ‘Sometimes the best move is not to play’. This has turned out to be very apt.
The best way to speed up a program on a modern platform.. is by making it do less. So I spent some time doing that. DNS at its core is case-insensitive, so www.powerdns.com is equal to WwW.PoWeRdNs.CoM. There are basically two ways do go about this. The first is to first lowercase everything, the second is to make your comparison functions case-insensitive.
The problem with the first solution is that, while allowed, many DNS clients will react badly when they receive an answer for ‘www.google.com’ when they asked for ‘WWW.GOOGLE.COM’. So you have to keep the original question around. This costs memory, and time.
Previously PowerDNS followed approach 1. I’ve now moved it to true case insensitvity. I also solved the strange and murky world of the trailing dot, ‘www.google.com’ versus ‘www.google.com.’.
Al this proved to be a small speedup, but it feels a lot better now lowercasing everything all the time.
Even bigger news in the ‘not to play’ department was short-circuiting the whole wonderful MOADNSParser (mother of all etc) structure. This system is wonderful, a joy to program, and about as efficient as it can get while remaining safe.
However - to decode and encode A, CNAME and NS records, it is overkill. So, large parts of PowerDNS now do those records by hand, saving a lot of malloc/free (new/delete) calls. Especially for A records, which constitute the bulk of all queries, this makes a big difference.
I wonder very much why I did not do this earlier, but I think it is because I love the MOADNSParser too much. I should be honest however and know that it is overkill to use so much code for parsing the 4 bytes of an IP address!
PowerDNS appears to be 20-30% faster because of the work today. On FreeBSD, a loaded PowerDNS now spends 25% of its time in the kernel, which tells me we are doing all right.
Posted by bert hubert Fri, 07 Apr 2006 20:06:00 GMT
I should be looking snazzy again, spent a heap of money on clothing the past two days. I tend to neglect my appearance, which detracts from my engineering brilliance :-)
In other news, I used the Sun T2000 to do statistics on the .COM zone, and learned some performance related things. One trick is if you do stuff on the commandline, it pays to stretch out things over several processes using pipes, as more strands will be busy. This is the opposite of what you do on smaller systems.
Much calculating later, I now have some solid figures on PowerDNS use and I have to say I’m rather pleased. It appears there are far far more PowerDNS users than mailing list subscribers, which makes me a bit sad. These people might miss vital PowerDNS security announcements. So, if you are a PowerDNS user, head over to the mailinglists and subscribe, even if only to the announcements list.
I announced the open sourcing of PowerArchiver the day before yesterday and have since decided to change the name to PowerFiler as PowerArchiver is an existing product. I also bought powerfiler.net and powerfiler.org and brought them online, but there is no content yet. I’m pondering hosting the entire website on trac, but this is not perfect. But then again, what is.
I made PowerFiler compile on FreeBSD, which required all of a two line change. See the previous entry on how to get PowerFiler!
Ahh - big day again. You may recall a large Dutch ISP trialling PowerDNS. They are so confident they switched over 3 more of their most important nameservers to PowerDNS. I feel all warm and fuzzy.
Also, I’ve suspected this before, but there are things about PowerDNS I don’t know. Like TUPA, The Ultimate PowerDNS Admin. It looks cool. Then I hear rumours about Druid DNS releasing cool stuff, as well as Frontiernet. I’m very happy to see these projects but please drop me a line if you are developing things on top of PowerDNS, I might be able to help, or at least mention you to other PowerDNS users.
Posted by bert hubert Mon, 03 Apr 2006 20:33:00 GMT
Last week I received an invitation to become a member of a superstore. Due to planning idiocies, large supermarkets aren’t allowed outside of city centers (get this) here in The Netherlands, so you need to be a ‘member’, which is only possible if you have a business. Luckily I have several.
But the odd thing was that I already was a member. But they did send a 20 euro discount voucher! So I went there today to ‘apply’ for membership, and they duly discovered I already was a member. Could I still cash my discount voucher?
Much pondering, calling of supervisors, issuing of stamps later, it was decided I could. 20 euros is no small thing so I immediately splurged lots more on vital stuff like…
.. several kinds of ready-made pizza products. Avid readers of this blog will know that I have a worrying addiction to these ancient flat-breads, so why would I try pre-made stuff? Well, I can be stupendously lazy at times, that’s why. I’ve not yet met a good programmer who isn’t lazy, so this is good news.
On to the vital stats. The ‘Nestle’ perishable pizza bottom looks expensive, well made, and even _is_ expensive. When prepared in my pizza oven, it even looks perfect, very thin, very Neapolitan. It tastes likes carton though.
I also tried a smaller no-brand pizza bottom which is thicker and looks less professional, but which tasted a lot better. Nothing compared to my home made dough though.
I bought several bottles of varying kinds of ‘pasta sauce’ and it turns out they are all fine. My new pizzas don’t have a lot of sauce on them, I find that even the very cheap ready-made sauce is very good.
All in all, a worthwhile experience.
A day that didn’t really go anywhere. Spent quite some time fighting different-endian PCAP files. PowerDNS contains technology to replay recorded DNS streams for verification and analysis purposes, for which it needs to be able to parse tcpdump files. It turns out these come in both big- and little-endian flavours.
Furthermore, Solaris has a 2*64-bit struct timeval, whereas pcap files use regular 32-bit time_t values. So I had to abstract that all out. Didn’t commit it to SVN yet as part of the code doesn’t work yet.
Peter Zijlstra previously educated me on the use of ‘clock algorithms’ for cache pruning. PowerDNS currently prunes based on the TTL of records, which is probably not the best thing to do. A long-lived record has no need to outstay a shorter lived one if it is never queried.
My local sources now put a record in the back of a linked list every time it is accessed or created (many thanks to Joaquín Mª López Muñoz for explaining how this works). When we want to prune, we start with the least used records, which are at the beginning.
When the recursor tries to find a record in the cache and finds it to be expired, it can simply ignore it. It will be refreshed soon anyhow.
It would appear this could speed up PowerDNS a lot, and also enable us to limit ourselves to a fixed amount of memory used (see below).
Also, implemented RFC 4255, SSHFP, which took all of 30 minutes, counting the implementation of hex-encoded records. Without that infrastructure, it would’ve taken 3 minutes.
This does not do anything yet - the recursor does not need to know about SSHFP and the authoritative nameserver doesn’t use the innovative ‘MOADNSParser’ yet. I’ll probably change that before 2.9.21.
I also spent some time trying to get the linux implementation of getrusage fill out the ‘integral’ memory fields. Turns out no OS I have access to uses these fields as their definition has traditionally been crap. SuSv3 also doesn’t mention these fields at all. So that was some lost effort. It appears you’ll have to do something different on each unix to discover real memory usage.
In working on this, I managed to get UML compiled and working without too much work, which is a first. The UML defconfig is not very good though, will send Jeff Dike a patch.
I did discover mallinfo(2) today, which is present in all unixes it appears, and provides information on the memory allocation subsystem. The numbers nedmalloc output here appear to be bogus though.
