Posted by bert hubert
Tue, 28 Mar 2006 20:48:00 GMT
Ok, this blog is not all about PowerDNS. Seriously. So, first some Physics. I used to be a physics student at Delft University of Technology, but I dropped out halfway through. That doesn’t mean I lost interest in hard science though.
I have a strong interest in ‘fringe science’. In my not so humble opinion, quietly shared by scientists I know, physics is focussed too much on confirming current ideas, whereas doing research into ‘interesting’ results is frowned upon.
I previously wrote a bit about this here.
Some of the things I keep an eye on are
- “Cold fusion”
- The gravity anomaly described in the link above
- Gravity shielding
Cold fusion
The cold fusion bit is interesting enough. There are literally thousands of results but none of them has proven able to convince mainstream physics. Partially this has been due to the experimenters, which have sometimes made huge fools out of themselves, or have even committed fraud.
However, even when people do come along with solid results, they are faced with incredible amounts of criticism. You might as well try to convince people child pornography is art. The results on your career in physics are highly similar.
I’m currently of the opinion that there is so much smoke surrounding cold fusion that there is bound to be some fire.
Gravity shielding
Has been interesting too. Realise that nothing, and I mean nothing affects gravity. It goes through everything. We can’t create it, we can’t stop it. The saga started out with measurements by the secretive Evgeny Podkletnov, who claimed to have observed a slight decrease in the force of gravity above a rapidly spinning superconducting disk. High temperature superconductors are excellent at conducting electricity but their mechanical properties are somewhat lacking, and people have had a hell of a time getting such a disk to rotate at speed without disintegrating.
NASA sunk a lot of effort in trying to reproduce his results, but sort of failed. The guy in charge, David Noever is currently nowhere to be found, after he also researched gravity anomalies during solar eclipses.
Then another scientist, Ning Li studied the effect and vanished, as far as I understand it. Popular Mechanics ran an article on her. In the mean time, Podkletnov is now supposed to be part of secret military research in Russia. The stuff of conspiracies!
This strand of interest appeared to be slowly dying off though when suddenly ESA and US Air Force sponsored scientists presented this paper, on the ESA website no less.
In this paper, they report finding a 1-in-10000 change in gravity above a ring of niobium or lead when, cooled to liquid helium temperatures, it is rapidly spun up or down.
They mention that they’ve spent three years trying to spot errors in their experiment, which has been run 250 times.
Well, why is this important? As I described in my own page linked above, quantum mechanics and (general) relativity collide. Gravity is firmly in the relativity domain, superconductivity is as quantum mechanic as it gets. Also, nothing else has ever changed gravity.
This discovery could quite literally put physics on its head - which is high time, things were getting decidedly boring.
PowerDNS
Ah, that thing. Well, not a lot to report. Everything ticking over just fine. Did discover that an important part of DNS, the ‘any query’ is completely unspecified by the RFCs. You can try to read what you have to do into the ancient writings of Mockapetris
& friends, but I’m not to sure. Decided to emulate BIND instead, which is also the easiest thing to do.
I’m trying to double the recursor performance (again), but this appears to be hard work. Perhaps DTrace on the Niagara can be of some help.
Life
Trying to relax a bit, worked too hard on PowerDNS and other projects. Working too hard makes me unfriendly and irritable, which is not a pleasant thing.
Posted in Linux, PowerDNS, Netherlabs, Life | no comments | no trackbacks
Posted by bert hubert
Mon, 27 Mar 2006 18:45:00 GMT
Today the big Dutch ISP migrated one of its three recursor nameserver IP addresses to PowerDNS, at first sight all appears well. In preparation of this event, over a billion packets were retransmitted and answers verified against incumbent nameservers.
One thing we missed is that that the verification code uses part of the same code as the nameserver itself. This in turn meant that some malformed packets never were replayed, which hid the fact that
the recursor logged these errors verbosely
these packets are rather common
I’ve made the recursor a little less strict with respect to packets with trailing garbage. This has reduced error reporting a lot and improved general customer satisfaction.
But after these things were addressed, things progressed swimmingly and there were happy faces all round.
PowerTOOLS
Additionally, I’ve made a tiny 61 kilobyte package of just the PowerDNS recursor. I enjoyed the large amount of control a raw Makefile gives one compared to penetrating the layers of cruft called configure.ac, Makefile.am and Makefile.in.
I’ve long had the urge to rout out the venerable autotools from my projects, now may be the time. To this end, I’ve started summarising why we actually need ./configure. So far I’ve found a few things.
There are basically three categories:
Where things come from, what we have
- Checking dependencies and:
- informing the user intelligibly of any missing ones
- Make proper use of detected libraries
- Configure ourselves to work around any missing ones that are not vital
- Allow user to specify the non-default location of any dependencies, overriding either improper defaults (this should be rare) or make it possible to choose between different versions of a dependency.
- Choose which capabilities should be compiled into the resulting programs
- Make other compile-time choices which cannot easily be changed at runtime.
Where things go
- Figure out where programs, documentation, configuration files should be installed, either by
- determining proper defaults for the target operating system
- allowing the user to override these sensible defaults, if needed
- On install move items to these places.
- Make tarballs of the source that contain the files needed to compile.
Build mechanics, dependencies
Allow programmer to easily specify the buildup of binaries (ie, which source files are part of a program), without duplication of work.
Abstract out the mechanics of building shared libraries. This has generally been the domain of libtool. Different operating systems have different rituals for making shared libraries, static executables etc etc.
Implementation
I’m pretty sure GNU Make, combined with perhaps some bash scripts, contains almost everything needed to implement the above without too much work.
This won’t be a bombastic process, but will probably evolve into enough to make building and releasing the PowerDNS recursor easy.
Posted in Linux, PowerDNS, Netherlabs | 2 comments | no trackbacks
Posted by bert hubert
Sun, 26 Mar 2006 14:05:00 GMT
Ok, apologies to the people that syndicate me, the URL might have changed. The timestamps on the older posts are also a bit dodgy, and the 2 comments have definitely vanished.
I used to be into ‘layout’ a “lot” so I hope you appreciate the improved appearance of this blog, including the dreaded smart quotes.
Ok, onto the real content.
PowerDNS
You may recall the stunning bug I wrote about yesterday, and how I solved it. Later that day I thought of an old adage “A bug is never alone”, and indeed, it turned out that the negative-cache, where we store records that auhoritatively don’t exist, was also cleaned in reverse, whereby we continuously removed all new entries.
Fixing that bug raised the steady-state cache hitrate from 80% to 90%, which doesn’t sound like a lot, but means the amount of network traffic generated to the Internet has halved.
I did do something controversial and limited this negative caching to at most one hour. I’m pretty sure this is what people want, and it saves heaps of memory anyhow. After an hour PowerDNS will, on getting a new query, verify if the domain name or record exists again. Sue me.
I also moved the negative cache to Boost::multi_index_container, I can’t heap enough praise on this container. It slices, it dices.
I also used it to implement user initiated cache deletion, you can now use rec_control wipe-cache blog.netherlabs.nl to remove this beloved blog from your cache, in case it contained bad content. To study your cache, use rec_control dump-cache filename.
Pizza
No pizza news today. I’m trying to think of the human angle of this blog but there is not a lot to report :-)
Posted in Linux, PowerDNS, Netherlabs | 1 comment | no trackbacks
Posted by bert hubert
Sat, 25 Mar 2006 15:02:00 GMT
For far too long now we’ve been working on implementing custom features for a big internet service provider here in The Netherlands and it appears we are almost there.
But then again, I’ve thought so a number of times already. The recursor (or resolver) of a network is one of the most crucial components of providing good service.
Put simply, a broken nameserver is perceived as a broken network. A slow nameserver means a slow network. So providers are understandably nervous about migrating to PowerDNS!
Some events may be forcing their hands however. To help ease migration fears, I’ve written dnsreplay_mindex, a tool that replays recorded DNS traffic (which you should anonymise using dnswasher if you plan on shipping it to me!) against PowerDNS, and shows statistics relative to your original nameserver.
I’m now confident that the PowerDNS recursor performs, in many cases, thousands of times better than the competition. Ok, that sentence has a touch of marketing to it. Just a touch. But I’m currently benchmarking at three times the original speed and dropping 30000 times less packets than BIND 8.latest.
That does not mean to say the PowerDNS recursor is perfect. It isn’t, not by a long shot. Even yesterday it turned out one of the more unique features of PowerDNS, the ability to forego hammering broken nameservers with queries that time out, had a cache that was cleaned in reverse: all NEW entries were being removed each minute.
The stunning thing is that it worked fine anyhow, just ate heaps of memory and performed some needless queries - which other nameservers perform all the time in any case.
Furthermore, the recursor carries IP addresses around as full blown strings, for which there is no excuse.
Update: I fixed this here
So there is still work to do, but I’m confident we can migrate at least one of the target servers to PowerDNS on Monday.
In other news, it is a bit quiet on the Niagara (Sun T2000) front, I’m mostly reading up on the unique features of its CPU before delving in with code.
My current aims are to make PowerDNS really fast on T2000 and write a HOWTO about the process, allowing you to benefit from this architecture as well.
On the human interest front, it turns out that leaving the dough to rise in the fridge does indeed produce something that is more like the kind of dough I want, but I’m still not there! I think I’ll aproach my favorite pizza restaurant soon and hope they are willing to share. I already have a proper pizza oven.
Posted in Linux, PowerDNS, Netherlabs, Life | 3 comments | no trackbacks
Posted by bert hubert
Fri, 24 Mar 2006 15:01:00 GMT
I’ve managed to make the T2000 a comfortable place to live in, for a Linux person like me. John Levon of Sun pointed me towards Blastwave, an OpenSolaris community site that adds apt-get like abilities.
I fixed up the few remaining problems people faced compiling and using PowerDNS on Solaris, it now works out of he box.
Generally, compared to earlier revisions, things tend to ‘just work’ a lot more on Solaris 10, but there is yet a way to go. For example, a default install won’t allow you to generate new home directories, as /home is under control of an automounter.
But what is very good is that even the notoriously difficult programs autoconf, automake and libtool all function as intended. This is of vital importance for when actually having Solaris as your main development platform for Open Source, as these tools generate the ubiquitous ./configure scripts for most projects.
Casper Dik pointed me to the proper place for UltraSPARC T1 (aka Niagara) performance documents, I think I see how I can make the various PowerDNS components shine on this architecture.
The converse of this is that you actually need to work at making this new chip scream - quite a number of unmodified programs do not benefit from all the additional cores and strands.
For the recursor, I can probably get away with removing my beloved MTasker and instead use pthreads or Solaris native threads.
For the bind2 authoritive mode, more work will be needed, where I will be looking at finegrained locking to make zone loading fast.
In other news, tonight I continued my quest to make perfect pizza dough, but I’m still not there. I think the flour available in shops here is not entirely of the right kind. This in an attempt to add a ‘human interest’ angle to my new blog :-)
Posted in Linux, PowerDNS, Netherlabs | no comments | no trackbacks
Posted by bert hubert
Fri, 24 Mar 2006 14:59:00 GMT
Power maintenance was over earlier than expected so I was able to immediately plug in the ‘try and buy’ Sun Fire T2000. I am typing this from the Mozilla that comes with Solaris 10, it looks good.
I’ll write up more of the experience, I think there are some easy ways in which Sun could improve the first impression this machine makes.
Also, many thanks to my friend Ahhing who supplied vital equipment to make everything work!
Updated with picture:
Posted in Linux, PowerDNS, Netherlabs | no comments | no trackbacks
Posted by bert hubert
Wed, 22 Mar 2006 14:58:00 GMT
Just announced to the PowerDNS Mailinglists that a Sun T2000 will be arriving tomorrow on which I intend to squeeze the utmost of performance out of my favorite nameserver.
I’ve been reading up on ‘CoolThreads’ and how to best utilize them.
It may be some form of poetic justice that there will be power maintenance at my office tomorrow so chances are I’ll have a lot of time to unpack and look at the server.
Posted in Linux, PowerDNS, Netherlabs | 1 comment | no trackbacks
