I noticed something unexpected when building GCC today. My triple core AMD desktop machine would build the compiler in 55 s but a new, decent-ish dual core laptop was taking about 200 s. The problem turned out to be the on-demand CPU governor which, due to me ‘nicing’ the build, was idling the CPU along at half speed. De-nicing it brought the build down to 53 s.
I had a play with distcc and ran into similar things. My first try at spreading the build across five cores surprisingly didn’t speed up the build at all. I noticed that Ubuntu 9.04 had niced distcc causing the desktop machine to also idle along at 1 GHz. Fixing this brought the build down to 41 s.