Three Monkeys, Three Typewriters, Two Days

January 11, 2010

More on distcc

A few days ago I mentioned that I've been experimenting with distcc. I finally took the time to do some timings. All times below, unless stated otherwise, are for a completely clean debug non-libxul build on my Mac. In particular, none of the files are in ccache, but ccache _is_ being used (so the compile time includes the cost of caching the new object files). distcc is being invoked via CCACHE_PREFIX. The distcc builds were done with -j10 (I also tried higher -j numbers and saw things between no effect and a slowdown as I got up close to -j20) and allowing 10 distcc connections to the server. The non-distcc builds were done with -j3, which is my normal build setup (since the mac in question only has 2 cores). distcc over ssh did not use ssh connection sharing, and was doing X forwarding on each ssh connection; I should retest with those two things changed at some point, if I decide I don't like doing things over tcp. The network is a wireless 802.11g on the Mac laptop end and gigabit ethernet to the wireless router on the Linux desktop. I found that ,lzo for the distcc builds didn't materially affect things (may have made the build 1-2% faster when used).

Build typeTime
local build on the Mac38 minutes
distcc over ssh50 minutes
distcc over tcp32 minutes

This was obviously distressing. I didn't test distcc over ssh after this, but particularly distressing was the fact that the compilation seemed largely diskbound and therefore distcc didn't help much. This got me started looking into hard drive performance as a function of utilization, since I could swear that the Mac's hard drive has been getting slower as it fills up. Turns out, this is actually the case. So I moved my rarely-used Linux VM to an external hard drive, which took free space on my internal disk from 4GB to 65GB or so. With that change done:

Build typeTime
local build on the Mac32 minutes
distcc over tcp22.5 minutes

Progress! Next, I tried dropping ccache, which let me measure the impact of pump mode. Since I do generally see a benefit from ccache, and wanted to measure it, I also tried a build without distcc but with a primed ccache (so build a tree, delete the objdir, rebuild that tree). Results:

Build typeTime
no-ccache distcc in pump mode16 minutes
no-ccache distcc without pump mode17 minutes
Local build coming entirely from ccache15 minutes

So if I were pretty sure I'd almost always be near this Linux box, just not using ccache and using pump mode might be the way to go. As it is, using distcc + ccache seems like a good approach for now.

Now all I need is a way to figure out a way to get this sort of data without it taking several hours, a way to pick good values for -j, and a faster internal hard drive in this laptop.

Posted by bzbarsky at January 11, 2010 3:48 PM