ISP Kart Competition, PowerDNS CPU utilization coolness
Posted by bert hubert Sun, 02 Apr 2006 19:06:00 GMT
Ok, the Dataman ISP Kart Competition was cool. I met many old friends there and also a lot of PowerDNS users, like XS4ALL, True, ISP Services and WideXS (didn’t speak to them, they were too busy karting it appears). Dave Aaldering appears to have done another fine job this year!
I was a bit hammered from a party the previous day and left a bit early. I’ll be there again next year though!
This week is supposedly light on activities, but I will be at the monthly gathering of a number of Delft internet companies.
PowerDNS
One big PowerDNS user had some performance complaints and I outfitted them with the previously mentioned ‘cache cache’ to help improve things. A nameserver is a very dynamic system and it is a good idea to monitor its CPU usage very closely. That is easier said than done though, but then I had a very cool idea. Operating systems in general keep a count of how much CPU has been spent in total on a process. This is an ever increasing number, which in itself is sort of interesting but doesn’t tell us a lot.
However, the first derivative of that number is the “instantaneous” CPU load. And taking derivatives is what graphing tools like rrd or mrtg do. So all I had to do was export the number of milli-seconds the OS has already accounted PowerDNS has occupied the CPU, and graph that.
And it works just fine. This should be a fine tool to determine the cause of PowerDNS problems.
On another note, one such cause has been that PowerDNS needs to be able to limit its memory consumption for it to be useful on smaller machines.
The Sun T2000 (Niagara)
Did initial runs of the PowerDNS recursor on the big Sun and it appears some work is needed to make things perform - as expected. At first I wanted to revamp most of the recursor but it now appears I can (almost) get away with calling pthread_create() 24 times and adding some locking.
This will probably speed things up massively as DNS recursing is mostly memory and network bound, which means there are large opportunities to get all the ‘strands’ operating in tandem. I think it might even speed up PowerDNS on SMP Opteron systems.
Stefan Arentz is working on making boost::asio benefit from Solaris ‘completion ports’, I’m pondering if it would be possible to move the PowerDNS recursor to asio. This would in theory be faster as it would need a few less system calls per packet (which we now spend on MTasker).
