Posted by bert hubert
Mon, 14 Aug 2006 16:46:00 GMT
As previously noted, Sun is making a SunFire T2000 server available permanently for PowerDNS development, which should be good for all PowerDNS users, and probably for Sun as well. With a big PowerDNS user we are currently investigating an interaction between PowerDNS, Solaris and its Completion Ports, which may turn out not to be a PowerDNS bug. So everybody wins.
The server is arriving tomorrow at the PowerDNS offices, we hope to have it up and running shortly.
Ok, some more spiffy ‘before and after’ graphs, this time from a Solaris 10 user:
The lower graph lists the number of queries per 5 minutes. In the lowest graph, it can be seen that just before and after the maintenance period (the white bit around Mon), the number of processed queries went up substantially.
The upper graph is a plot of the load average of the server in question, which can be seen to drop visibly after this period. It is probably best to concentrate on Friday vs Wednesday. Friday, which is non-PowerDNS, did 200kqueries in 5 minutes in peak, at a load of 1.75 at peak.
The next Wednesday, we see a peak of 300kqueries in five minutes, with a load of 0.6 at peak.
If we combine these numbers, we see the efficiency (queries divided by cpu load) go up by a factor of 4. It should be noted that this is a dual CPU machine, which explains why the load can exceed 1 when running a single name server.
Thanks to Jan Gyselinck for these graphs.
Posted in Linux, PowerDNS, Netherlabs | no comments
Posted by bert hubert
Wed, 09 Aug 2006 21:04:00 GMT
(Update: I’ve upgraded my Ruby on Rails, thanks for warning me! See here)
Well, big news, we’ve decided PowerDNS needs a new homepage, and that it needs to tell you why you should run PowerDNS. All pretty obvious of course, but it took us some time to realise PowerDNS use is spreading purely based on word of mouth, and not because we promote it so well (which we don’t).
The main page currently projects a sort of post-dotbomb shareware image. The wiki is fine as it goes, but only suitable for hardcore developers. And finally, the documentation contains lots of gems on how to best use PowerDNS, but it is all very spread out.
So, until we have our new homepage, some promotion. A large PowerDNS deployment is set to make 120 servers redundant. In energy costs alone this should save around 100kW, continuously (Update: ok, perhaps a bit less. Allow me some artistic license here. If you include cooling, it is not that far off.) . For reference, that kind of power requires four of these to generate:
We might as well follow Sun’s lead and rename PowerDNS ‘The most Eco Friendly Nameserver’. EcoDNS has a nice ring to it.
Some more promotion. Switching to PowerDNS does not just save the environment, it also makes your mail go faster. A happy PowerDNS user sent us this graph:
This says, in Dutch, “average mail delivery time, in seconds”. Note the dramatic shift very early Thursday morning, from around 1.8 seconds to 0.8, and later around 0.65-ish.
The almost threefold speedup happened immediately after the switch to PowerDNS. This makes some kind of sense, with the massive amounts of spam these days, a mail server can spend an awful amount of time trying to resolve strange sender addresses, and traversing often very bad or weird reverse delegations. Spammers have also been known to try to make their DNS so misconfigured that DNS-based filtering attempts fail.
No matter what the exact cause is, there is a nearly threefold speedup. Made all the more spectacular by the non-zero based graph!
We’ll try to move the hype from here to serious white papers on the new homepage. But it feels good to share some of the improvements people are achieving by switching to PowerDNS.
Posted in Linux, PowerDNS, Netherlabs | no comments
Posted by bert hubert
Wed, 02 Aug 2006 21:54:00 GMT
Quick note. If you’ve sensed a disturbance in the force, it is because tens of millions of internet connections moved to PowerDNS just now.
I still have goosebumps.
Posted in Linux, PowerDNS, Netherlabs | 1 comment
Posted by bert hubert
Wed, 28 Jun 2006 20:51:00 GMT
PowerDNS
It’s been a period of contrasts. One the one hand I’ve had to deal with the fact that banks currently take the narrow minded view that I’m an unemployed bum (see previous post), while on the other hand the PowerDNS recursor appears to be taking the world by storm, and does bring in money.
A really really rough guestimate would be that around 40 million internet connections are now powered, or will soon be powered by PowerDNS. This includes deployments (or upcoming deployments) I know about, and a healthy 50% bonus for those people running the recursor without telling me, or having to report bugs.
The bonus is pretty conservative, I only hear from most PowerDNS users in case they have problems, and it appears many PowerDNS installations are mostly trouble free.
This also highlights a problem with generating PowerDNS, or more in general, open source, income. Being free software, we don’t make any direct money from PowerDNS sales.
Furthermore, if people really trust software, and experience nor expect any problems with it, they won’t feel the need to buy support. Nor do we really want to become a company mostly consisting of support personnel.
The upshot is that we are being punished for writing software people feel good about. Rather perverse.
The good news is that some big PowerDNS users have found it in their hearts to send money our way anyhow, either in the form of paid enhancement requests, or by simply taking out support which they don’t really expect to need.
Sun Niagara T2000
I’ve previously written about the ‘Try and Buy’ Sun ‘Coolthreads’ T2000, which was made available for us to test. More about this later, but Sun has decided to make such a server available permanently for PowerDNS development, which is good news. This is a big server with lots of processors and memory, with some special features to boot. We’ll make sure PowerDNS performs really well on this architecture, which should be good for everybody.
Mortgage
Wow, did I almost fall for it. I tend to want to understand things around me, which is why I studied physics. I’m good with numbers and I deal with complex systems all day.
Then I went to get a mortgage and decided to just trust my bank. I’ve rarely been shafted that badly. Thanks to my good friends Remco, Remco and Tsjoi, I barely escaped from a hideously complex and expensive mortgage.
It is like this (at least in The Netherlands). The bank has lots of savings from other people, and they need to make money to pay the interest those other people expect. So they decide to rent out that money to people who need it, and are willing to pay for it. A mortgage generally has a house to back it up, so their risks can be near-zero. If the mortgage is worth less than your house, perhaps because you are not an unemployed bum after all but have savings, the bank is in a happy position.
They can rent you the money for a nice fee, and if you can’t pay, they sell the house, take the money you owe them, and give you the rest. So almost no risk at all. They also demand that you insure your house, thus eliminating even the risk of it burning down.
Renting out money like this is not a high-margin product for a bank. Everybody with money to spare can offer such a deal, and competition is fierce.
So if you barge in with zero knowledge of mortgages, what happens? They sell you a monstrosity consisting of:
- an expensive life insurance (possibly including a thorough medical evaluation)
- an investment plan
- a loan you only pay interest on, and never pay back
They invest your savings for 30 years, make you pay for the life insurance, and after thirty years, in theory the life insurance pays out and your savings will have increased to an amount around the original loan, which is then paid back in one go.
Sounds fine. Except that you’ve just committed to a 30 year life insurance you could not select yourself, there is but one choice. If you die, the insurance pays the money to the bank, not to you.
Furthermore, you’ve bought a full 30 years of managed investment, so delivered by your bank. You can’t get anybody else to invest for you.
And most interestingly, in the end, if their investments didn’t work out you are screwed, because you’ll have a residual debt.
So, after my friends told me I was being screwed, I decided to get to the bottom of things. I now have a Gnumeric (Open Source equivalent of Excel) spreadsheet that is more spiffy than any of the mortgage software I’ve seen at banks, and in fact has caught a number of errors in mortgages that have been calculated for me.
Then I went back to the banks and saved a stunning 40% in total costs.
So it is really true that knowledge is power.
Posted in Linux, PowerDNS, Life | 5 comments
Posted by bert hubert
Thu, 08 Jun 2006 21:33:00 GMT
Ok, I’m back from Egypt. Had a stellar time, if you have a group that wants to travel to Egypt, drop me a line, the people that organized this for us are very good.
We’ve visited the pyramids, the sphinx, the egyptian museum (all in Cairo), the unfinished obelisk, Philae, the high dam (all in Aswan), Kom Ombo, Edfu (en route to Luxor), valley of the kings and Karnak (in Luxor). All very impressive.
While I was away, PowerDNS 3.1.1 has held up well, although some minor bugs are now known about. Most of these involve the new zone forwarding, and authoritative zone hosting features, which are not in wide use.
Each consecutive PowerDNS recursor release has attracted the attention of larger internet providers, but we’ve reached the end of the line now. There are none bigger than those trialling PowerDNS now. Additionally, the previous weeks have seen actual deployment to over ten million internet users.
This is a very humbling thought. The PowerDNS company and I have worked very hard at making PowerDNS the most secure, simple and capable recursor out there. In the brief time since XS4ALL funded us to add the features a modern ISP needs, the takeup has been tremendous.
It is good to be back!
Posted in Linux, PowerDNS, Life | 3 comments | no trackbacks
Posted by bert hubert
Wed, 24 May 2006 04:41:00 GMT
PowerDNS 3.1 turned out to contain a brown paper bag bug that in retrospect should not hit too many people, but still. So I rushed out 3.1.1, which always leaves me with a bad feeling.
Furthermore, I’m off to Egypt for two weeks. While other people do work on PowerDNS, development will come to a nearly complete halt.
So here’s to hoping that 3.1.1 fixed more bugs than it caused..
See you in two weeks!
Posted in Linux, PowerDNS, Netherlabs, Life | 4 comments | no trackbacks
Posted by bert hubert
Sat, 06 May 2006 21:58:00 GMT
Welcome back after this 9-day hiatus from my Blog!
Ok, what has happened. I had two good experiences with local electronics stores here in Delft. Goris was unable to provide me with the proper cable to hook up my shiny new WiFi directional antenna, but they referred me to HEC, which did have the components to make the cable. My skills with the soldering iron are humorous at best. However the people at HEC kindly offered to make the cable for me! So now I finally have a working combination of antenna, cable and adapter. And to make things perfect, Goris allowed me to test my new WiFi card to verify Linux compatability. Luckily it all works. I hope to hook up pahu tomorrow.
Slight damper on today is that I was fined for driving my bicycle through a street here in Delft that turned out to be for pedestrians only. 30 euros too. I normally am all in favour of the rule of law but this makes little sense. It is fortunate therefore that the actual fine contained a number of errors which I am sure invalidate it, so I wasted no time in drafting a written protest. I’m not usually like this but I was pissed of at the inanity of this fine.
PowerDNS & Windows
As staunch a supporter as I am of Open Source, my technology wants to go places. So, I downloaded the ‘free’ version of Visual Studio Express 2005 from Microsoft. And a fine compiler it is! I had fixed a bunch of initial incompatabilities using the (also fine) Minimalistic GCC for Windows. I think this is the first Microsoft C++ compiler that can really be taken seriously. VC++ debugging mode found two real bugs in PowerDNS, which motivated me to turn on the ‘debugging mode’ of the G++ libstdc++ as well, which uncovered two further bugs!
This strengthens my feelings that porting to different platforms helps uncover bugs which aren’t (yet) a problem but might be.
Ahu’s quick guide to porting to windows:
- Use VC++ 2005, earlier versions have a lot more problems with constructions g++ accepts. It also appears that VC++ 2005 is smart with respect to UNIX/DOS line endings.
- Separate the really different things to different files, which share one header file. Don’t make #ifdef soup!
- Make a single include file that includes OS-dependent include files (like
windows.h).
- On windows, one can only write and read from sockets using
send(to) and recv(from). As these functions work for UNIX as well, use these functions exclusively on sockets.
- To close a socket under windows, you need
closesocket() and not close. Candidate for the file mentioned under 1.
- Windows has different
errno traditions. All network (‘winsock’) related errors need WSAGetLastError(). See here.
- Use ‘Tortoise’ Subversion for revision control, integrates really well with both Windows and UNIX. Also smart about line endings.
- If, as for me, your prime development platform is UNIX, install the MINGW crosscompiler so you can easily verify the code at least compiles for Windows. This helps prevent code-rot at an early stage.
- Get a Windows buddy :-) Many thanks to Michel Stol, who is far more at home in Windows than I am.
PowerDNS 3.1
I hope to release PowerDNS 3.1 shortly, and make things settle down a bit then. Since the previous blog post, I added full blown IPv6 outgoing support, with IPv6 achieving full parity - any IPv6 nameserves that are faster than their IPv4 partners will receive more queries.
The ‘–export-etc-hosts’ stuff also works fine now, which should allow many networks to simply run unconfigured, save for that option, and have everything Just Work.
For more, see here.
Posted in Linux, PowerDNS, Netherlabs, Life | 3 comments | no trackbacks
Posted by bert hubert
Thu, 27 Apr 2006 19:31:00 GMT
We’ve been looking for a new house lately, but this has not been easy. The Netherlands has been experiencing a housing bubble for the past decade, so even a small house costs an arm and a leg. A number of possible places were sold in the brief period between making an appointment and actually going round to visit. Very frustrating.
1.4 billion queries
I started an endurance test some time ago to really stress out the recursor. This test has now reached 1.4 billion queries. This means we still have 3 billion queries to go to hit the magic 2^32.
PowerDNS 3.0.1 appears to hold up well. There are some small problems on big endian platforms (ultrasparc), which are solved in subversion, and I uncovered an obscure form of misconfiguration (having a nameserver with multiple IP addresses, one of them being lame) we didn’t deal with. Other nameservers don’t either, so it doesn’t really matter. See if you can resolve ‘www.nl.netherlabs.eu’. If you can, chances are you are running a very recent PowerDNS :-)
Crossing over
The PowerDNS recursor is a pure recursor, or at least, used to be. I literally spent years thinking (on and off) about how to make PowerDNS authoritative and recursive at the same time without losing the clean design and today I figured out how to do it.
It turned out the proper way is to insert a hook in the call that figures out the best nameserver to ask a question. If we are authoritative for a domain, we send back an empty nameserver which means ‘we know, don’t go out’.
When the time comes to go ask that nameserver, the emptiness is recognized, and a call is made to the ‘out of band’ resolver. This delivers a vector of DNSResourceRecords, just like a remote nameserver would. The rest of PowerDNS does not ‘know’ it is parsing self-generated data.
This has the downside that we cache our own data. But compared to the elegance of keeping the rest of the nameserver unchanged, this is a small price to pay.
In the same place, we can also insert a ‘forwarder’ nameserver, whereby we can point a domain towards an external authoritative nameserver.
Finally, built on the authoritative infrastructure, I added ‘–export-etc-hosts’. Quite a number of people have asked me if there were an easy way to have their recursor serve a small number of domains. There is an obvious place to get this data, /etc/hosts. ‘–export-etc-hosts’ does the obvious and generates full zones for each entry in /etc/hosts, making them available for all your clients.
This in turn means that a large number of people now have no reason anymore not to run the PowerDNS recursor, and benefit from its performance and superior anti-spoofing measures :-)
To test, head to this posting to the mailing list and download away!
Posted in Linux, PowerDNS, Netherlabs, Life | 2 comments | no trackbacks
Posted by bert hubert
Sun, 23 Apr 2006 20:44:00 GMT
There has been a recent paucity of pizza related posts, but I did manage to employ my fine pizza oven yesterday, this time to make Nan bread. Nans are usually made in a tandoor, a blisteringly hot clay oven. And while my passion for good food is well known, having a clay oven (traditionally submerged in the earth if I understand correctly) is going a bit far. But I do have a bilsteringly hot pizza oven. And indeed, I can now finally procude Nans that are somewhat crisp on the outside, yet chewy on the inside, exactly the way I like them.
PowerDNS 3.0 release followup
Since the release of PowerDNS 3.0 last Thursday, some big users have switched over. This has led to a good trickle of tiny bugs which were all addressed quickly. To note:
- The Debian Sarge 3.0r1 default kernel, 2.6.8, claims to support epoll but in fact appears not to do so. The epoll multiplexer now does a better job of testing itself at runtime, and falling back to select if needed.
- No difference was made between NXDOMAIN and NXRRSET - probably not very relevant, but a technical violation. This was a regression caused by an earlier fix. No good deed goes unpunished!
- Filedescriptors 0, 1 and 2 are closed when going to the background, but I can’t 100% guarantee there is no logging output to fd 2 during operations. These descriptors are now dupped to
/dev/zero.
- Bad handling of malformed EDNS0 packets.
- Failed compilation on Solaris 10 i386, because Solaris decides to #define DS, messing up our support of the DS record type.
I’ll wait a few more days and do a 3.0.1 release with nothing but minimal changes that all address real problems. There is one report of an unexplained crash on Solaris around that I’d like to solve, but as there is only one report and it can’t be reproduced, this might be hard.
I have an endurance test running of a single PowerDNS instance which has processed half a billion packets so far, I intend to keep it running until it passes the 32-bit wraparound mark, just to check if my code is properly using the 64-bit variables I pass it.
PowerDNS Design and Engineering
I sat down for a few hours and documented the inner workings of the PowerDNS recursor here. I know I’ll be reading this documentation myself three months from now, I swap out memory really quickly. You don’t need to read this document in order to use PowerDNS, but if you want to contribute, it should be very helpful.
Posted in Linux, PowerDNS, Netherlabs, Life | 5 comments | no trackbacks
Posted by bert hubert
Fri, 21 Apr 2006 07:19:02 GMT
Well that came at a very bad time. Yesterday on the day of the PowerDNS Recursor 3.0 release, our cabinet over at Level3 dropped off the net. Much debugging later it appeared one of the customer hosts was compromised and filling the ethernet with tiny packets at line rate.
Apologies. The owner of said (Windows) machine will be coerced into providing an evening of drinks and entertainment to compensate our nightly labours.
Posted in Linux, PowerDNS, Netherlabs, Life | 1 comment | no trackbacks
