Three Monkeys, Three Typewriters, Two Days

January 8, 2010

Toolwhip, distcc, and faster compiles

Ted recently pointed me to the toolwhip project, which is a way to run Apple's gcc on Linux, cross-compiling to produce Mac binaries. Clearly desirable, since my Linux box is a lot faster than my Mac one. Here are the steps to get it working, as far as I can tell, at least if you Mac has OS X 10.5:

  1. Grab yourself a toolwhip tree following their directions.
  2. Follow the instructions in cctools.README to compile and install cctools. Note that the toolwhip tree comes with an existing usr_include so you can ignore the REQUIREMENTS section in this readme.
  3. Follow the instructions in ld64.README to compile and install ld64. Once again, you can ignore the REQUIREMENTS section.
  4. Download the Apple gcc source for the exact compiler version you want from the Apple site and untar it into a directory called gcc_42-5574 or gcc_42-5566 in your toolwhip source tree. If you plan to use this compiler with distcc, make sure it's the same exact version as that installed on your Mac. If you just plan to cross-compile, I'm not sure that it matters quite as much.
  5. Apply the relevant patch (gcc_42-5574.patch or gcc_42-5566.patch) to the gcc source tree.
  6. Follow the instructions in gcc_42-5574.README or gcc_42-5566.README to compile and install gcc. This time, follow the REQUIREMENTS section as needed; in particular you will need to copy whatever SDKs you want to your Linux box.
  7. Create the following symlinks:
      ln -s Developer/usr/bin/gcc-4.2 /usr/bin/
      ln -s /Developer/usr/bin/g++-4.2 /usr/bin/
    for all those makefiles that call the compiler with an absolute path.

At this point you should have a working cross-compiler. The remaining instructions are for setting up distcc with it.

  1. Grab a toolwhip tree on your Mac a well and follow the directions in distcc.README to build distcc.
  2. Follow the directions in distcc.README to build distcc on your Linux box.

What you do after this point probably depends on exactly how you plan to run distcc and distccd, whether you plan to use ccache, whether you want pump mode, etc. There are lots of directions in distcc.README. So far I have a setup that does distcc over ssh to my Linux box. I'm hoping to sort out the rest of the distcc stuff (not using ssh, and maybe using pump mode) sometime in the next week or so. To get what I have working so far going:

  1. Set PermitUserEnvironment to "yes" in /etc/ssh/sshd_config on the Linux box.
  2. Put DISTCCD_PATH=/Developer/usr/bin in your ~/.ssh/environment on the Linux box.

Now you run a test. In my case, I use ccache, so this worked:

  export DISTCC_HOSTS=bzbarsky@linux-box:/path/to/toolwhip/distccd/on/linux,lzo
  export CCACHE_PREFIX=/path/to/toolwhip/distcc/on/mac
  export DISTCC_VERBOSE=1

and then compiling Firefox normally, using make -f client.mk. If you don't use ccache, you would need to skip the CCACHE_PREFIX and make sure that symlinks called gcc-4.2 and g++-4.2 came in your PATH on the Mac before the real compilers and pointed to the toolwhip distcc. I think that should do the trick, but haven't tried it. The ",lzo" in the DISTCC_HOSTS above just says to gzip-compress the data; it can be left out if desired.

Question for those with some distcc experience: is pump mode worth forgoing ccache?

One last note: I have not yet verified that this gives me faster compiles! Linking has to happen on the distcc client, and that tends to be a pretty slow part... Perhaps I would be better off just compiling entirely on the Linux box then copying the resulting .dmg to the Mac.

UPDATE: While the above distcc setup works for compiling an already-configured Mozilla tree, it seems that trying to configure from scratch fails because it thinks it failed to find Core Text, or more precisely because it fails to find ANSI C headers. But only if you use DISTCC_VERBOSE=1, so don't do that. It also seems that actually cross-compiling from Linux doesn't work because libffi insists on passing gnu ld options to an ld it detected as not being gnu ld. That's very sad.

Posted by bzbarsky at January 8, 2010 4:19 PM