Posted by bert hubert
Sun, 26 Mar 2006 14:05:00 GMT
Ok, apologies to the people that syndicate me, the URL might have changed. The timestamps on the older posts are also a bit dodgy, and the 2 comments have definitely vanished.
I used to be into ‘layout’ a “lot” so I hope you appreciate the improved appearance of this blog, including the dreaded smart quotes.
Ok, onto the real content.
PowerDNS
You may recall the stunning bug I wrote about yesterday, and how I solved it. Later that day I thought of an old adage “A bug is never alone”, and indeed, it turned out that the negative-cache, where we store records that auhoritatively don’t exist, was also cleaned in reverse, whereby we continuously removed all new entries.
Fixing that bug raised the steady-state cache hitrate from 80% to 90%, which doesn’t sound like a lot, but means the amount of network traffic generated to the Internet has halved.
I did do something controversial and limited this negative caching to at most one hour. I’m pretty sure this is what people want, and it saves heaps of memory anyhow. After an hour PowerDNS will, on getting a new query, verify if the domain name or record exists again. Sue me.
I also moved the negative cache to Boost::multi_index_container, I can’t heap enough praise on this container. It slices, it dices.
I also used it to implement user initiated cache deletion, you can now use rec_control wipe-cache blog.netherlabs.nl to remove this beloved blog from your cache, in case it contained bad content. To study your cache, use rec_control dump-cache filename.
Pizza
No pizza news today. I’m trying to think of the human angle of this blog but there is not a lot to report :-)
Posted in Linux, PowerDNS, Netherlabs | 1 comment | no trackbacks
Posted by bert hubert
Sat, 25 Mar 2006 15:02:00 GMT
For far too long now we’ve been working on implementing custom features for a big internet service provider here in The Netherlands and it appears we are almost there.
But then again, I’ve thought so a number of times already. The recursor (or resolver) of a network is one of the most crucial components of providing good service.
Put simply, a broken nameserver is perceived as a broken network. A slow nameserver means a slow network. So providers are understandably nervous about migrating to PowerDNS!
Some events may be forcing their hands however. To help ease migration fears, I’ve written dnsreplay_mindex, a tool that replays recorded DNS traffic (which you should anonymise using dnswasher if you plan on shipping it to me!) against PowerDNS, and shows statistics relative to your original nameserver.
I’m now confident that the PowerDNS recursor performs, in many cases, thousands of times better than the competition. Ok, that sentence has a touch of marketing to it. Just a touch. But I’m currently benchmarking at three times the original speed and dropping 30000 times less packets than BIND 8.latest.
That does not mean to say the PowerDNS recursor is perfect. It isn’t, not by a long shot. Even yesterday it turned out one of the more unique features of PowerDNS, the ability to forego hammering broken nameservers with queries that time out, had a cache that was cleaned in reverse: all NEW entries were being removed each minute.
The stunning thing is that it worked fine anyhow, just ate heaps of memory and performed some needless queries - which other nameservers perform all the time in any case.
Furthermore, the recursor carries IP addresses around as full blown strings, for which there is no excuse.
Update: I fixed this here
So there is still work to do, but I’m confident we can migrate at least one of the target servers to PowerDNS on Monday.
In other news, it is a bit quiet on the Niagara (Sun T2000) front, I’m mostly reading up on the unique features of its CPU before delving in with code.
My current aims are to make PowerDNS really fast on T2000 and write a HOWTO about the process, allowing you to benefit from this architecture as well.
On the human interest front, it turns out that leaving the dough to rise in the fridge does indeed produce something that is more like the kind of dough I want, but I’m still not there! I think I’ll aproach my favorite pizza restaurant soon and hope they are willing to share. I already have a proper pizza oven.
Posted in Linux, PowerDNS, Netherlabs, Life | 3 comments | no trackbacks
Posted by bert hubert
Fri, 24 Mar 2006 15:01:00 GMT
I’ve managed to make the T2000 a comfortable place to live in, for a Linux person like me. John Levon of Sun pointed me towards Blastwave, an OpenSolaris community site that adds apt-get like abilities.
I fixed up the few remaining problems people faced compiling and using PowerDNS on Solaris, it now works out of he box.
Generally, compared to earlier revisions, things tend to ‘just work’ a lot more on Solaris 10, but there is yet a way to go. For example, a default install won’t allow you to generate new home directories, as /home is under control of an automounter.
But what is very good is that even the notoriously difficult programs autoconf, automake and libtool all function as intended. This is of vital importance for when actually having Solaris as your main development platform for Open Source, as these tools generate the ubiquitous ./configure scripts for most projects.
Casper Dik pointed me to the proper place for UltraSPARC T1 (aka Niagara) performance documents, I think I see how I can make the various PowerDNS components shine on this architecture.
The converse of this is that you actually need to work at making this new chip scream - quite a number of unmodified programs do not benefit from all the additional cores and strands.
For the recursor, I can probably get away with removing my beloved MTasker and instead use pthreads or Solaris native threads.
For the bind2 authoritive mode, more work will be needed, where I will be looking at finegrained locking to make zone loading fast.
In other news, tonight I continued my quest to make perfect pizza dough, but I’m still not there. I think the flour available in shops here is not entirely of the right kind. This in an attempt to add a ‘human interest’ angle to my new blog :-)
Posted in Linux, PowerDNS, Netherlabs | no comments | no trackbacks
Posted by bert hubert
Fri, 24 Mar 2006 14:59:00 GMT
Power maintenance was over earlier than expected so I was able to immediately plug in the ‘try and buy’ Sun Fire T2000. I am typing this from the Mozilla that comes with Solaris 10, it looks good.
I’ll write up more of the experience, I think there are some easy ways in which Sun could improve the first impression this machine makes.
Also, many thanks to my friend Ahhing who supplied vital equipment to make everything work!
Updated with picture:
Posted in Linux, PowerDNS, Netherlabs | no comments | no trackbacks
Posted by bert hubert
Wed, 22 Mar 2006 14:58:00 GMT
Just announced to the PowerDNS Mailinglists that a Sun T2000 will be arriving tomorrow on which I intend to squeeze the utmost of performance out of my favorite nameserver.
I’ve been reading up on ‘CoolThreads’ and how to best utilize them.
It may be some form of poetic justice that there will be power maintenance at my office tomorrow so chances are I’ll have a lot of time to unpack and look at the server.
Posted in Linux, PowerDNS, Netherlabs | 1 comment | no trackbacks
