Posted by bert hubert
Thu, 12 Oct 2006 19:55:00 GMT
Many thanks to my brother who read my previous post and promptly offered to procure new disks for me, they are now in production. Thanks Jaap!
C & C++
One of the things that is easy to forget about C++ is that, while not (really) a superset of C, it does offer the ability to call C functions from C++, and makes some pretty strong statements about the abilty to exchange data between the two languages.
C++ does not come with a set of ‘foundation classes’, and while the “standard template library” is strong on data structures, and algorithms to manipulate them, nothing is offered in the way of network communications infrastructure.
Many attempts have been made to rectify this situation, but these tend to be somewhat heavy handed, or overly complex.
Enter the ComboAddress. This C++ union is laid out in memory just like the venerable struct sockaddr_in, and through its second member, also just like struct sockaddr_in6.
The upshot is that we have a C++ union with interesting methods, that allows us to specify destination addresses, either IPv4 or IPv6, with ease, but that can also be passed to the standard Berkeley C socket functions!
These functions promptly forget they are passed a C++ union, and interpret their argument as a struct sockaddr family member.
For example:
int sock = socket(AF_INET, SOCK_STREAM, 0);
ComboAddress ca("127.0.0.1", 6666);
if (connect(sock, ca) < 0)
unixDie("connecting to server");
‘unixDie()’ is a simple function that uses strerror to throw a runtime_error with a descriptive error message.
If you are really paying attention, you might have noticed that the ‘connection’ function above is not a real C function, and you would be right. It is a very thin wrapper that saves some typing:
inline int connect(int fd, const ComboAddress& remote)
{
return connect(fd, (struct sockaddr*) &remote, remote.getSocklen());
}
Another example:
int fd = accept(sock, &ca);
if(fd >= 0)
cout << "Connection from " << ca <<endl;
The tiny bit of code that makes up the ComboAddress can be found in the PowerDNS Recursor source code. I find that it nicely bridges the vast power of the Berkeley sockets API, while taking a lot of the tedium out of calling the host of functions needed to convert between printable IP addresses, port numbers, and the actual stuff the sockets API expects.
And this is all possible because a bunch of guys with serious ‘Unix beards’ decided that C and C++ should remain family members. Thanks!
Posted in Linux, PowerDNS, Netherlabs | 7 comments
Posted by bert hubert
Fri, 06 Oct 2006 22:52:00 GMT
Well, I reported previously that the server that powers this blog fell 9 feet, and appeared to have survived? Since that event, one of the disks reported odd errors every once in a while, but those appeared to point to a bad cable. I replaced it, but no joy, problems remained.
So tonight I decide to back up that disk completely, and take it out of use. And lo, during the backup it decides to pack up! It made a noise like a passing moped, and ceased to work. Backup was almost entirely done.
I restored the backup to another computer and mounted it via NFS (over wifi no less!), and things (including this blog) are back in production again. I’ll have to buy new disks ASAP though.
PowerDNS RIPE presentation
RIPE was lots of fun, although my presentation did not go as well as I’d hoped. I’ve been distracted by grave medical problems in my family, which mean that I spend a lot of my time in the hospital. It might’ve been better to not do the presentation. Some people did tell me they enjoyed it though. Oh well.
For the first time, I’ve had the pleasure of answering a question from a webcam viewer! RIPE offers the great service that remote attendants can ask questions over IRC or Jabber, and a RIPE employee will then relay the question. A tremendous service!
Lunch at RIPE was fantastic, and it was very nice to meet many friends again. All in all a good day.
Posted in Linux, PowerDNS, Life | 7 comments
Posted by bert hubert
Sun, 01 Oct 2006 12:47:00 GMT
Quick post to say that at RIPE 53, I’ll be presenting about the PowerDNS Recursor and specifically its implementation of my Internet-Draft (“Draft RFC”).
More details in this post.
If you are at RIPE, come and say hello, or have an excellent Krasnapolski lunch with me!
Long standing bugs
Over the past few weeks some very long standing “low level irritation” PowerDNS bugs have been fixed. One of the things you learn during the maturation of software projects is that things are good once you start to get reports of obscure bugs, as this means that the big problems are out of the way!
Predictably, the bugs were related to the handling of rare errors, which also reinforces my belief that error handling of rare bugs tends to be very buggy, as these paths rarely get exercised, and when they do, people often don’t even notice the problem is more in the handling than in the error.
Don’t try to be too smart when dealing with errors!
Posted in Linux, PowerDNS, Netherlabs | 8 comments
Posted by bert hubert
Thu, 21 Sep 2006 21:16:00 GMT
Quick update on some small things.
PowerDNS
I managed to release PowerDNS Recursor 3.1.3 which must rank as one of the most succesful releases of PowerDNS ever, as I have had zero feedback, despite a large number of downloads. Most big deployments have switched over. There is still a very small trickle of odd crashes though, but they are so rare it is hard to pin it down to anything.
Wireless
Our new house has a lot going for it, except wiring possibilities. It might be possible to improve this, but right now I want nothing but the best and I’m not prepared to soil my house with badly laid cables. So it has to be wireless, which for fixed computers mostly means USB. After some searching and experimenting, I can report that zd1211 derived devices work really well using the Linux zd1211rw driver. Wireless reception depends a lot on RF conditions, having a USB receiver on a cable means you can move it around for the best reception.
The nice thing about the ZD1211 derived devices (I have two 3Com OfficeConnect adaptors) is that the authors of the driver are very approachable and work well with (and are in fact part of) the Linux kernel community. Unlike some.
New house
It still rocks, although we haven’t had much time to empty the last boxes and buy furniture that matches the quality of the house. Sadly, we are spending a lot of time in the hospital and taking care of related things.
Posted in Linux, PowerDNS, Netherlabs, Life | 9 comments
Posted by bert hubert
Fri, 08 Sep 2006 13:36:00 GMT
Programming is a lot of things. One of them is writing good error messages. We tend to think that errors are rare and this should of course be so. However, sometimes they are not, and in that case, good reporting is vital to quickly resolve the problem.
So, even though we should make sure errors do not happen, if they do, our error messages should be top notch.
About error messages
Purpose
For operators, they are vital aids in configuring software
For system adminstrators, they show which external problems need to be resolved (disk full, network down, etc).
Should a program crash, the authors often only have error messages as clues to why this happened. Many crashes are preceded by errors that are reported. A good error can help generate a bug fix.
Taxonomy of error messages
- Configuration problems, for example, looking for a file in directory A while it resides in directory B;
- Unavailable resources (disk full, out of memory), connectivity problems;
- Should Never Happen.
Configuration problems
These commonly occur while software is being installed and setup. Good error reporting is of utmost importance here, as it serves two purposes:
- Educating the operator about how the program functions;
- Explaining what needs to be fixed.
Ad 1, an error like “Can’t start frobnicator because the discombulator is not running” teaches the operator that a frobnicator needs a discombulator. This knowledge can of course also be gleened from the documentation, but in this case, repetition is a good thing.
Compare this error to “Can’t start process: Connection refused”, for example, and think about how helpful that is.
Ad 2, a report like “Can’t connect to product database using connection string ‘dbuser=john, dbname=changeme’: No such database” immediately tells the operator what he needs to know.
Unavailable resources
These typically occur while a program is already running and installed, but are nonetheless important. Do not log ‘Disk full’, but report ‘Disk full writing to ….’ so the operator knows which disk filled up. If a server could not be reached, log the IP address and possibly the name of the server. Any discrepancy between the two may point out a DNS configuration error.
Out of memory is typically very hard to deal with, except when something really odd was going on. A typical example is trying to allocate a ridiculous amount of memory because of an earlier error, in which case logging what memory was being allocated for might help debug the problem. It typically will not help the user of a program.
Should never happen
These are errors of the program itself. Programmers quite often test for impossible conditions, especially if they sense these might conceivably happen one day. An example might be a module that can only connect to IPv4 servers that gets confronted by an IPv6 socket, which in turn leads calls to determine the IPv4 remote address to fail.
It is tempting to be quite rude in these messages, or say stuff like “should never happen!!”, but resist these urges. One day a ‘should not happen’ error is going to be a vital debugging clue. These errors are rare, but it pays to go, well, the extra few yards to perhaps report “unexpected address family accepting connection!”.
Guidelines
An error message should contain:
- Who is reporting the error (which subsystem, which program, which module)
- The action that failed
- The subject of the action (a directory, a server, a port number, an IP address)
- As good an indication of the actual error as possible.
- Optional - what the program is doing about it
An excellent error message is:
Webserver can't serve page, error opening file '/var/www/index.html': Permission denied, reporting HTTP 404 error
Ad 1, it is tempting to include filenames, function names, and line numbers here. OpenSSL does this a lot, for example. However, almost none of your intended audience will be able to extract useful information from the fact that the error occurred on line 123 of ‘multiplexer.c’. Make sure the module means something to the operator. It may be as simple as the name of your program.
Ad 2, this helps the operator determine if this error might be the explanation of observed problems. An error like “Webserver failed to increase TCP buffer size, continuing with default” can immediately be ruled out as an explanation for why people can’t log in to their mail.
Ad 3, an error like “Can’t open file” on its own can mean many things. One of which might be that it is not reading the configuration file you think it is, and trying to open your index.html in ‘/usr/local/www’, and not in ‘/var/www’ like you thought you configured.
Ad 4, self explanatory. Take the trouble to convert error codes into strings. Many programmers may know what ‘errno = 111’ means ‘Connection refused’, but don’t count on your users knowing this.
Ad 5, this is a fine counterpoint to item 4. “Giving a 404” is very clear for any operator of a web server. Not all errors need a followup, so reporting what the program is doing about the error is not mandatory.
Conclusion
Good error messages can save your users many days of problems. And, suprisingly, you might yourself gain even more time - how well do you know the internals of your program after a few months?
So please please, both as a user and a progammer, I ask you, spend time on error messages!
Posted in Linux, PowerDNS, Netherlabs | 12 comments
Posted by bert hubert
Mon, 14 Aug 2006 16:46:00 GMT
As previously noted, Sun is making a SunFire T2000 server available permanently for PowerDNS development, which should be good for all PowerDNS users, and probably for Sun as well. With a big PowerDNS user we are currently investigating an interaction between PowerDNS, Solaris and its Completion Ports, which may turn out not to be a PowerDNS bug. So everybody wins.
The server is arriving tomorrow at the PowerDNS offices, we hope to have it up and running shortly.
Ok, some more spiffy ‘before and after’ graphs, this time from a Solaris 10 user:
The lower graph lists the number of queries per 5 minutes. In the lowest graph, it can be seen that just before and after the maintenance period (the white bit around Mon), the number of processed queries went up substantially.
The upper graph is a plot of the load average of the server in question, which can be seen to drop visibly after this period. It is probably best to concentrate on Friday vs Wednesday. Friday, which is non-PowerDNS, did 200kqueries in 5 minutes in peak, at a load of 1.75 at peak.
The next Wednesday, we see a peak of 300kqueries in five minutes, with a load of 0.6 at peak.
If we combine these numbers, we see the efficiency (queries divided by cpu load) go up by a factor of 4. It should be noted that this is a dual CPU machine, which explains why the load can exceed 1 when running a single name server.
Thanks to Jan Gyselinck for these graphs.
Posted in Linux, PowerDNS, Netherlabs | no comments
Posted by bert hubert
Wed, 09 Aug 2006 21:04:00 GMT
(Update: I’ve upgraded my Ruby on Rails, thanks for warning me! See here)
Well, big news, we’ve decided PowerDNS needs a new homepage, and that it needs to tell you why you should run PowerDNS. All pretty obvious of course, but it took us some time to realise PowerDNS use is spreading purely based on word of mouth, and not because we promote it so well (which we don’t).
The main page currently projects a sort of post-dotbomb shareware image. The wiki is fine as it goes, but only suitable for hardcore developers. And finally, the documentation contains lots of gems on how to best use PowerDNS, but it is all very spread out.
So, until we have our new homepage, some promotion. A large PowerDNS deployment is set to make 120 servers redundant. In energy costs alone this should save around 100kW, continuously (Update: ok, perhaps a bit less. Allow me some artistic license here. If you include cooling, it is not that far off.) . For reference, that kind of power requires four of these to generate:
We might as well follow Sun’s lead and rename PowerDNS ‘The most Eco Friendly Nameserver’. EcoDNS has a nice ring to it.
Some more promotion. Switching to PowerDNS does not just save the environment, it also makes your mail go faster. A happy PowerDNS user sent us this graph:
This says, in Dutch, “average mail delivery time, in seconds”. Note the dramatic shift very early Thursday morning, from around 1.8 seconds to 0.8, and later around 0.65-ish.
The almost threefold speedup happened immediately after the switch to PowerDNS. This makes some kind of sense, with the massive amounts of spam these days, a mail server can spend an awful amount of time trying to resolve strange sender addresses, and traversing often very bad or weird reverse delegations. Spammers have also been known to try to make their DNS so misconfigured that DNS-based filtering attempts fail.
No matter what the exact cause is, there is a nearly threefold speedup. Made all the more spectacular by the non-zero based graph!
We’ll try to move the hype from here to serious white papers on the new homepage. But it feels good to share some of the improvements people are achieving by switching to PowerDNS.
Posted in Linux, PowerDNS, Netherlabs | no comments
Posted by bert hubert
Wed, 02 Aug 2006 21:54:00 GMT
Quick note. If you’ve sensed a disturbance in the force, it is because tens of millions of internet connections moved to PowerDNS just now.
I still have goosebumps.
Posted in Linux, PowerDNS, Netherlabs | 1 comment
Posted by bert hubert
Tue, 11 Jul 2006 06:41:00 GMT
Ok, I no longer think spammers should be shot. I now think they should be force fed their own intestines, and THEN be shot.
I’m usually a friendly kind of person, but I now estimate that I lose around 5% of my productivity on cleaning out spam from Wikis, bug trackers or mailing list queues, not even mentioning deleting unfilterable spam from my own mailbox.
The latest thing has been spammers who not just tack on advertising to the fine PowerDNS wiki, but actively delete existing content and replace it by their own advertising.
Perhaps Guantanamo does have a purpose.
Mortgage
Well, it turns out banks aren’t crazy (see previous post). If you negotiate a cut rate mortgage, they’ll start to eliminate risk left and right, because the deal is not worth it for them otherwise.
I’ve now had to provide documentation on my father, brother, all real estate agents involved, shoe sizes, and loads of other crap. Next I think they’ll ask for my swimming certificate. But they are now committed, in that they’ve provided the all important guarantee on the 10% downpayment. Which is good as it was due today :-)
Posted in PowerDNS, Life | no comments
Posted by bert hubert
Wed, 28 Jun 2006 20:51:00 GMT
PowerDNS
It’s been a period of contrasts. One the one hand I’ve had to deal with the fact that banks currently take the narrow minded view that I’m an unemployed bum (see previous post), while on the other hand the PowerDNS recursor appears to be taking the world by storm, and does bring in money.
A really really rough guestimate would be that around 40 million internet connections are now powered, or will soon be powered by PowerDNS. This includes deployments (or upcoming deployments) I know about, and a healthy 50% bonus for those people running the recursor without telling me, or having to report bugs.
The bonus is pretty conservative, I only hear from most PowerDNS users in case they have problems, and it appears many PowerDNS installations are mostly trouble free.
This also highlights a problem with generating PowerDNS, or more in general, open source, income. Being free software, we don’t make any direct money from PowerDNS sales.
Furthermore, if people really trust software, and experience nor expect any problems with it, they won’t feel the need to buy support. Nor do we really want to become a company mostly consisting of support personnel.
The upshot is that we are being punished for writing software people feel good about. Rather perverse.
The good news is that some big PowerDNS users have found it in their hearts to send money our way anyhow, either in the form of paid enhancement requests, or by simply taking out support which they don’t really expect to need.
Sun Niagara T2000
I’ve previously written about the ‘Try and Buy’ Sun ‘Coolthreads’ T2000, which was made available for us to test. More about this later, but Sun has decided to make such a server available permanently for PowerDNS development, which is good news. This is a big server with lots of processors and memory, with some special features to boot. We’ll make sure PowerDNS performs really well on this architecture, which should be good for everybody.
Mortgage
Wow, did I almost fall for it. I tend to want to understand things around me, which is why I studied physics. I’m good with numbers and I deal with complex systems all day.
Then I went to get a mortgage and decided to just trust my bank. I’ve rarely been shafted that badly. Thanks to my good friends Remco, Remco and Tsjoi, I barely escaped from a hideously complex and expensive mortgage.
It is like this (at least in The Netherlands). The bank has lots of savings from other people, and they need to make money to pay the interest those other people expect. So they decide to rent out that money to people who need it, and are willing to pay for it. A mortgage generally has a house to back it up, so their risks can be near-zero. If the mortgage is worth less than your house, perhaps because you are not an unemployed bum after all but have savings, the bank is in a happy position.
They can rent you the money for a nice fee, and if you can’t pay, they sell the house, take the money you owe them, and give you the rest. So almost no risk at all. They also demand that you insure your house, thus eliminating even the risk of it burning down.
Renting out money like this is not a high-margin product for a bank. Everybody with money to spare can offer such a deal, and competition is fierce.
So if you barge in with zero knowledge of mortgages, what happens? They sell you a monstrosity consisting of:
- an expensive life insurance (possibly including a thorough medical evaluation)
- an investment plan
- a loan you only pay interest on, and never pay back
They invest your savings for 30 years, make you pay for the life insurance, and after thirty years, in theory the life insurance pays out and your savings will have increased to an amount around the original loan, which is then paid back in one go.
Sounds fine. Except that you’ve just committed to a 30 year life insurance you could not select yourself, there is but one choice. If you die, the insurance pays the money to the bank, not to you.
Furthermore, you’ve bought a full 30 years of managed investment, so delivered by your bank. You can’t get anybody else to invest for you.
And most interestingly, in the end, if their investments didn’t work out you are screwed, because you’ll have a residual debt.
So, after my friends told me I was being screwed, I decided to get to the bottom of things. I now have a Gnumeric (Open Source equivalent of Excel) spreadsheet that is more spiffy than any of the mortgage software I’ve seen at banks, and in fact has caught a number of errors in mortgages that have been calculated for me.
Then I went back to the banks and saved a stunning 40% in total costs.
So it is really true that knowledge is power.
Posted in Linux, PowerDNS, Life | 5 comments
