Posted by bert hubert
Wed, 28 Jun 2006 20:51:00 GMT
PowerDNS
It’s been a period of contrasts. One the one hand I’ve had to deal with the fact that banks currently take the narrow minded view that I’m an unemployed bum (see previous post), while on the other hand the PowerDNS recursor appears to be taking the world by storm, and does bring in money.
A really really rough guestimate would be that around 40 million internet connections are now powered, or will soon be powered by PowerDNS. This includes deployments (or upcoming deployments) I know about, and a healthy 50% bonus for those people running the recursor without telling me, or having to report bugs.
The bonus is pretty conservative, I only hear from most PowerDNS users in case they have problems, and it appears many PowerDNS installations are mostly trouble free.
This also highlights a problem with generating PowerDNS, or more in general, open source, income. Being free software, we don’t make any direct money from PowerDNS sales.
Furthermore, if people really trust software, and experience nor expect any problems with it, they won’t feel the need to buy support. Nor do we really want to become a company mostly consisting of support personnel.
The upshot is that we are being punished for writing software people feel good about. Rather perverse.
The good news is that some big PowerDNS users have found it in their hearts to send money our way anyhow, either in the form of paid enhancement requests, or by simply taking out support which they don’t really expect to need.
Sun Niagara T2000
I’ve previously written about the ‘Try and Buy’ Sun ‘Coolthreads’ T2000, which was made available for us to test. More about this later, but Sun has decided to make such a server available permanently for PowerDNS development, which is good news. This is a big server with lots of processors and memory, with some special features to boot. We’ll make sure PowerDNS performs really well on this architecture, which should be good for everybody.
Mortgage
Wow, did I almost fall for it. I tend to want to understand things around me, which is why I studied physics. I’m good with numbers and I deal with complex systems all day.
Then I went to get a mortgage and decided to just trust my bank. I’ve rarely been shafted that badly. Thanks to my good friends Remco, Remco and Tsjoi, I barely escaped from a hideously complex and expensive mortgage.
It is like this (at least in The Netherlands). The bank has lots of savings from other people, and they need to make money to pay the interest those other people expect. So they decide to rent out that money to people who need it, and are willing to pay for it. A mortgage generally has a house to back it up, so their risks can be near-zero. If the mortgage is worth less than your house, perhaps because you are not an unemployed bum after all but have savings, the bank is in a happy position.
They can rent you the money for a nice fee, and if you can’t pay, they sell the house, take the money you owe them, and give you the rest. So almost no risk at all. They also demand that you insure your house, thus eliminating even the risk of it burning down.
Renting out money like this is not a high-margin product for a bank. Everybody with money to spare can offer such a deal, and competition is fierce.
So if you barge in with zero knowledge of mortgages, what happens? They sell you a monstrosity consisting of:
- an expensive life insurance (possibly including a thorough medical evaluation)
- an investment plan
- a loan you only pay interest on, and never pay back
They invest your savings for 30 years, make you pay for the life insurance, and after thirty years, in theory the life insurance pays out and your savings will have increased to an amount around the original loan, which is then paid back in one go.
Sounds fine. Except that you’ve just committed to a 30 year life insurance you could not select yourself, there is but one choice. If you die, the insurance pays the money to the bank, not to you.
Furthermore, you’ve bought a full 30 years of managed investment, so delivered by your bank. You can’t get anybody else to invest for you.
And most interestingly, in the end, if their investments didn’t work out you are screwed, because you’ll have a residual debt.
So, after my friends told me I was being screwed, I decided to get to the bottom of things. I now have a Gnumeric (Open Source equivalent of Excel) spreadsheet that is more spiffy than any of the mortgage software I’ve seen at banks, and in fact has caught a number of errors in mortgages that have been calculated for me.
Then I went back to the banks and saved a stunning 40% in total costs.
So it is really true that knowledge is power.
Posted in Linux, PowerDNS, Life | 5 comments
Posted by bert hubert
Sat, 17 Jun 2006 21:22:00 GMT
We bought a house! This was somewhat scary as I was in Egypt at the time, in a dreary resort (it looks nothing like this) surrounded by 60km of Red Sea and 60km of desert - and no internet. Even after fixing the network of the operator of the resort, there was no way to get online.
The following was even more scary - I don’t have a real job as such (or so the bank thinks), even though I make enough money from PowerDNS and other activities my company employs. Sadly, banks in The Netherlands still equate a normal job with income security and vice versa. So no real job, no income.
Luckily Mirjam holds a Real Job, which appears to have convinced the bank to fork over the money! The house would also not have been possible without the generosity/inheritance from my parents. House prices are crazy over here.
The ground floor of the house on the corner, it runs from just to the right of the door on the extreme left of the picture to the extreme right of the picture: 
Our upstairs neighbours walk through part of our garden.
Garden behind the house: 
Note that the realtor has a very powerful (‘lying’) lens - the garden is not as big as it appears here by far.
Same goes for the front: 
and:
.
The kitchen is the stuff of dreams:
The rest of the house is nice as well, but y’all don’t need to see my future bathroom and sleeping room :-)
Update: ok, Mirjam decided you do need to see it, read all about it here.
Posted in Life | 6 comments | no trackbacks
Posted by bert hubert
Thu, 08 Jun 2006 21:33:00 GMT
Ok, I’m back from Egypt. Had a stellar time, if you have a group that wants to travel to Egypt, drop me a line, the people that organized this for us are very good.
We’ve visited the pyramids, the sphinx, the egyptian museum (all in Cairo), the unfinished obelisk, Philae, the high dam (all in Aswan), Kom Ombo, Edfu (en route to Luxor), valley of the kings and Karnak (in Luxor). All very impressive.
While I was away, PowerDNS 3.1.1 has held up well, although some minor bugs are now known about. Most of these involve the new zone forwarding, and authoritative zone hosting features, which are not in wide use.
Each consecutive PowerDNS recursor release has attracted the attention of larger internet providers, but we’ve reached the end of the line now. There are none bigger than those trialling PowerDNS now. Additionally, the previous weeks have seen actual deployment to over ten million internet users.
This is a very humbling thought. The PowerDNS company and I have worked very hard at making PowerDNS the most secure, simple and capable recursor out there. In the brief time since XS4ALL funded us to add the features a modern ISP needs, the takeup has been tremendous.
It is good to be back!
Posted in Linux, PowerDNS, Life | 3 comments | no trackbacks
Posted by bert hubert
Wed, 24 May 2006 04:41:00 GMT
PowerDNS 3.1 turned out to contain a brown paper bag bug that in retrospect should not hit too many people, but still. So I rushed out 3.1.1, which always leaves me with a bad feeling.
Furthermore, I’m off to Egypt for two weeks. While other people do work on PowerDNS, development will come to a nearly complete halt.
So here’s to hoping that 3.1.1 fixed more bugs than it caused..
See you in two weeks!
Posted in Linux, PowerDNS, Netherlabs, Life | 4 comments | no trackbacks
Posted by bert hubert
Fri, 12 May 2006 22:08:00 GMT
Talk about embarrassing. You may know I’m busy working on a draft-draft RFC (it becomes an ‘Internet-Draft’ once submitted) about making DNS safer through some implementation and operational guidelines (see dns-anti-spoofing.html and
dns-anti-spoofing.txt).
While writing this document, I decided to add a section on the ‘birthday paradox attack’. Reported in 2002, this is a curious mathematical phenomenon that makes spoofing a nameserver vastly easier to do.
So I wrote down the specification:
Given the above, a recursor MUST:
* Use a new random source port from its available
range for each outgoing query
* Make full use of all 16 bits of the ID field
* Assure that its choices of port and ID cannot
be predicted by an attacker having knowledge of
its (pseudo-)random generator
* Take measures to prevent having multiple equivalent
questions outstanding to any authoritative server
Which is all fine. Except that PowerDNS did not adhere to the bit about equivalent outstanding questions! PowerDNS contains a general system that prevents heaps of identical queries from leaving the server, but that doesn’t translate well into ‘standardese’, you’d get something like ‘recursors MUST have a system that sort of prevents most of the identical queries’.
So, I added ‘query-chaining’ to PowerDNS, which detects this situation and puts an MThread to sleep when it tries to send out a duplicate question. When the answer to the initial question arrives, they all get woken up.
Due to the throttling code already in place, and the source port randomisation, this does not improve our security significantly, but at least I’m now in compliance of my own draft-draft RFC :-)
Code is linked here.
Posted in PowerDNS, Netherlabs | 1 comment | no trackbacks
Posted by bert hubert
Tue, 09 May 2006 20:15:00 GMT
I’ve long been a somewhat active member of the relevant DNS mailing lists, ‘namedroppers’ and ‘dnsop’, both affiliated with the IETF DNS workgroups.
I consider myself a bit of an outcast in the DNS community as I don’t sing the praises of DNSSEC, nor BIND, but I suspect this is not entirely fair as there are quite a number of people who are far more outcast than I am. So I suspect I’m on the fringe of the DNS community in the sense that I incidentally take part in useful email discussion, either on list or privately with relevant parties.
I recently called upon nameserver authors and operators to either upgrade their nameserver so it performs adequate anti-spoofing measures, or switch to a nameserver implementation that does (like tinydns or of course PowerDNS).
This call fell on very deaf ears it appears. The BIND people promised to look into it but as noted then, without an apparant sense of urgency. Not a lot has happened since, except that I’ve reiterated my recommendation privately to a number of relevant people.
In the meantime, I’ve been told the Microsoft nameserver is about 4 times easier to spoof than BIND, but I’ve been unable to verify this.
So, I did what I never thought I’d do, I wrote something intended to be an RFC. In short, this RFC specifies that a recursor MUST implement adequate anti-spoofing measures, and details what this entails.
Read all about it as old school text or rendered as pretty HTML. The RFC-compliant output is made possibly by the interesting but quirky tool xml2rfc.
I’ll spend some more time polishing the document before submitting it as an Internet Draft. I also need to figure out the correct procedure to set things in motion.
I sincerely hope nameservers that are easy to spoof clean up their act quickly, hopefully even before my draft hits the standards track.
Posted in PowerDNS, Netherlabs | 12 comments | no trackbacks
Posted by bert hubert
Sat, 06 May 2006 21:58:00 GMT
Welcome back after this 9-day hiatus from my Blog!
Ok, what has happened. I had two good experiences with local electronics stores here in Delft. Goris was unable to provide me with the proper cable to hook up my shiny new WiFi directional antenna, but they referred me to HEC, which did have the components to make the cable. My skills with the soldering iron are humorous at best. However the people at HEC kindly offered to make the cable for me! So now I finally have a working combination of antenna, cable and adapter. And to make things perfect, Goris allowed me to test my new WiFi card to verify Linux compatability. Luckily it all works. I hope to hook up pahu tomorrow.
Slight damper on today is that I was fined for driving my bicycle through a street here in Delft that turned out to be for pedestrians only. 30 euros too. I normally am all in favour of the rule of law but this makes little sense. It is fortunate therefore that the actual fine contained a number of errors which I am sure invalidate it, so I wasted no time in drafting a written protest. I’m not usually like this but I was pissed of at the inanity of this fine.
PowerDNS & Windows
As staunch a supporter as I am of Open Source, my technology wants to go places. So, I downloaded the ‘free’ version of Visual Studio Express 2005 from Microsoft. And a fine compiler it is! I had fixed a bunch of initial incompatabilities using the (also fine) Minimalistic GCC for Windows. I think this is the first Microsoft C++ compiler that can really be taken seriously. VC++ debugging mode found two real bugs in PowerDNS, which motivated me to turn on the ‘debugging mode’ of the G++ libstdc++ as well, which uncovered two further bugs!
This strengthens my feelings that porting to different platforms helps uncover bugs which aren’t (yet) a problem but might be.
Ahu’s quick guide to porting to windows:
- Use VC++ 2005, earlier versions have a lot more problems with constructions g++ accepts. It also appears that VC++ 2005 is smart with respect to UNIX/DOS line endings.
- Separate the really different things to different files, which share one header file. Don’t make #ifdef soup!
- Make a single include file that includes OS-dependent include files (like
windows.h).
- On windows, one can only write and read from sockets using
send(to) and recv(from). As these functions work for UNIX as well, use these functions exclusively on sockets.
- To close a socket under windows, you need
closesocket() and not close. Candidate for the file mentioned under 1.
- Windows has different
errno traditions. All network (‘winsock’) related errors need WSAGetLastError(). See here.
- Use ‘Tortoise’ Subversion for revision control, integrates really well with both Windows and UNIX. Also smart about line endings.
- If, as for me, your prime development platform is UNIX, install the MINGW crosscompiler so you can easily verify the code at least compiles for Windows. This helps prevent code-rot at an early stage.
- Get a Windows buddy :-) Many thanks to Michel Stol, who is far more at home in Windows than I am.
PowerDNS 3.1
I hope to release PowerDNS 3.1 shortly, and make things settle down a bit then. Since the previous blog post, I added full blown IPv6 outgoing support, with IPv6 achieving full parity - any IPv6 nameserves that are faster than their IPv4 partners will receive more queries.
The ‘–export-etc-hosts’ stuff also works fine now, which should allow many networks to simply run unconfigured, save for that option, and have everything Just Work.
For more, see here.
Posted in Linux, PowerDNS, Netherlabs, Life | 3 comments | no trackbacks
Posted by bert hubert
Thu, 27 Apr 2006 19:31:00 GMT
We’ve been looking for a new house lately, but this has not been easy. The Netherlands has been experiencing a housing bubble for the past decade, so even a small house costs an arm and a leg. A number of possible places were sold in the brief period between making an appointment and actually going round to visit. Very frustrating.
1.4 billion queries
I started an endurance test some time ago to really stress out the recursor. This test has now reached 1.4 billion queries. This means we still have 3 billion queries to go to hit the magic 2^32.
PowerDNS 3.0.1 appears to hold up well. There are some small problems on big endian platforms (ultrasparc), which are solved in subversion, and I uncovered an obscure form of misconfiguration (having a nameserver with multiple IP addresses, one of them being lame) we didn’t deal with. Other nameservers don’t either, so it doesn’t really matter. See if you can resolve ‘www.nl.netherlabs.eu’. If you can, chances are you are running a very recent PowerDNS :-)
Crossing over
The PowerDNS recursor is a pure recursor, or at least, used to be. I literally spent years thinking (on and off) about how to make PowerDNS authoritative and recursive at the same time without losing the clean design and today I figured out how to do it.
It turned out the proper way is to insert a hook in the call that figures out the best nameserver to ask a question. If we are authoritative for a domain, we send back an empty nameserver which means ‘we know, don’t go out’.
When the time comes to go ask that nameserver, the emptiness is recognized, and a call is made to the ‘out of band’ resolver. This delivers a vector of DNSResourceRecords, just like a remote nameserver would. The rest of PowerDNS does not ‘know’ it is parsing self-generated data.
This has the downside that we cache our own data. But compared to the elegance of keeping the rest of the nameserver unchanged, this is a small price to pay.
In the same place, we can also insert a ‘forwarder’ nameserver, whereby we can point a domain towards an external authoritative nameserver.
Finally, built on the authoritative infrastructure, I added ‘–export-etc-hosts’. Quite a number of people have asked me if there were an easy way to have their recursor serve a small number of domains. There is an obvious place to get this data, /etc/hosts. ‘–export-etc-hosts’ does the obvious and generates full zones for each entry in /etc/hosts, making them available for all your clients.
This in turn means that a large number of people now have no reason anymore not to run the PowerDNS recursor, and benefit from its performance and superior anti-spoofing measures :-)
To test, head to this posting to the mailing list and download away!
Posted in Linux, PowerDNS, Netherlabs, Life | 2 comments | no trackbacks
Posted by bert hubert
Sun, 23 Apr 2006 20:44:00 GMT
There has been a recent paucity of pizza related posts, but I did manage to employ my fine pizza oven yesterday, this time to make Nan bread. Nans are usually made in a tandoor, a blisteringly hot clay oven. And while my passion for good food is well known, having a clay oven (traditionally submerged in the earth if I understand correctly) is going a bit far. But I do have a bilsteringly hot pizza oven. And indeed, I can now finally procude Nans that are somewhat crisp on the outside, yet chewy on the inside, exactly the way I like them.
PowerDNS 3.0 release followup
Since the release of PowerDNS 3.0 last Thursday, some big users have switched over. This has led to a good trickle of tiny bugs which were all addressed quickly. To note:
- The Debian Sarge 3.0r1 default kernel, 2.6.8, claims to support epoll but in fact appears not to do so. The epoll multiplexer now does a better job of testing itself at runtime, and falling back to select if needed.
- No difference was made between NXDOMAIN and NXRRSET - probably not very relevant, but a technical violation. This was a regression caused by an earlier fix. No good deed goes unpunished!
- Filedescriptors 0, 1 and 2 are closed when going to the background, but I can’t 100% guarantee there is no logging output to fd 2 during operations. These descriptors are now dupped to
/dev/zero.
- Bad handling of malformed EDNS0 packets.
- Failed compilation on Solaris 10 i386, because Solaris decides to #define DS, messing up our support of the DS record type.
I’ll wait a few more days and do a 3.0.1 release with nothing but minimal changes that all address real problems. There is one report of an unexplained crash on Solaris around that I’d like to solve, but as there is only one report and it can’t be reproduced, this might be hard.
I have an endurance test running of a single PowerDNS instance which has processed half a billion packets so far, I intend to keep it running until it passes the 32-bit wraparound mark, just to check if my code is properly using the 64-bit variables I pass it.
PowerDNS Design and Engineering
I sat down for a few hours and documented the inner workings of the PowerDNS recursor here. I know I’ll be reading this documentation myself three months from now, I swap out memory really quickly. You don’t need to read this document in order to use PowerDNS, but if you want to contribute, it should be very helpful.
Posted in Linux, PowerDNS, Netherlabs, Life | 5 comments | no trackbacks
Posted by bert hubert
Fri, 21 Apr 2006 07:19:02 GMT
Well that came at a very bad time. Yesterday on the day of the PowerDNS Recursor 3.0 release, our cabinet over at Level3 dropped off the net. Much debugging later it appeared one of the customer hosts was compromised and filling the ethernet with tiny packets at line rate.
Apologies. The owner of said (Windows) machine will be coerced into providing an evening of drinks and entertainment to compensate our nightly labours.
Posted in Linux, PowerDNS, Netherlabs, Life | 1 comment | no trackbacks
