Three Monkeys, Three Typewriters, Two Days

January 29, 2010

Cross-compiling success

I took another stab at directly cross-compiling (not via distcc) using toolwhip. My previous attempt failed, but not due to libffi at all. The problem there was that I had the Mac ld in my path before the host ld. After fixing that and a few mozilla build system issues and making sure the build system can find the right SDK and the right ar and such I now have a cross-compile setup working. No make package yet, since that requires the dmg-creation utility which I don't have on the Linux machine. No --enable-breakpad support yet due to bug 543111. --enable-shark needs headers I don't have on that machine yet, so I'd need to sort that out. But for basic bisecting, this works.

Just so I can find it later, the working mozconfig file I'm using is something along these lines:

mk_add_options MOZ_MAKE_FLAGS="-s -j10"
export MOZ_MAKE_FLAGS="-s -j10"

mk_add_options CC="ccache /Developer/usr/bin/gcc-4.2"
export CC="ccache /Developer/usr/bin/gcc-4.2"

mk_add_options CXX="ccache /Developer/usr/bin/g++-4.2"
export CXX="ccache /Developer/usr/bin/g++-4.2"

mk_add_options CROSS_LIB_PATH="/Developer/usr/lib:/Developer/SDKs/MacOSX10.5.sdk/usr/lib"
export CROSS_LIB_PATH="/Developer/usr/lib:/Developer/SDKs/MacOSX10.5.sdk/usr/lib"

mk_add_options CROSS_COMPILE="1"
export CROSS_COMPILE="1"

export LD=/Developer/usr/bin/ld
export AR=/Developer/usr/bin/ar

mk_add_options PATH="/home/bzbarsky/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/Developer/usr/bin:/Developer/SDKs/MacOSX10.5.sdk/usr/bin"
export PATH="/home/bzbarsky/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/Developer/usr/bin:/Developer/SDKs/MacOSX10.5.sdk/usr/bin"

ac_add_options --target=i686-apple-darwin9
ac_add_options --enable-application=browser
ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
ac_add_options --disable-crashreporter
ac_add_options --disable-optimize
ac_add_options --enable-debug
ac_add_options --disable-tests

Various other --enable-whatever are hanging out in there too, but the above are sort of the minimal set one needs at the moment.

Posted by bzbarsky at 5:25 PM

January 25, 2010

H.264 patents outside the US

I've seen a number of comments recently along the lines of "I'm in Europe, so the H.264 patents don't apply to me; why are you not letting me have a browser that plays my Youtube videos?" So I took a look at the list of patents on H.264. Or rather, the first 6 pages of the 43 page list. Excluding the US, there are relevant patents granted in at least the following countries:

  • Europe: Germany, France, UK, Finland, Italy, Sweden, Belgium, Bulgaria, Liechtenstein, Austria, Czech Republic, Denmark, Spain, Hungary, Ireland, The Netherlands, Poland, Romania, Portugal, Slovenia
  • Asia: Japan, China, South Korea, Hong Kong, Singapore, Taiwan, India
  • Americas: Canada, Mexico
  • Australia

I wasn't reading very carefully; I'd be suprised if I didn't miss a few, or if a few more don't come later in the list somewhere.

Posted by bzbarsky at 10:47 PM

January 20, 2010

From the mouths of babes

Today Arlan happened to look at my screen when part of the desktop was actually visible. That led to approximately the following conversation:

Папа, там лиса!
Где? Да, там лиса.
Еще лиса! Две лисы.
Один, два, три.... Три лисы!

Little did he know that the fourth one (3.5 to be exact) was hiding under the iTunes window....

Posted by bzbarsky at 12:19 AM

January 11, 2010

More on distcc

A few days ago I mentioned that I've been experimenting with distcc. I finally took the time to do some timings. All times below, unless stated otherwise, are for a completely clean debug non-libxul build on my Mac. In particular, none of the files are in ccache, but ccache _is_ being used (so the compile time includes the cost of caching the new object files). distcc is being invoked via CCACHE_PREFIX. The distcc builds were done with -j10 (I also tried higher -j numbers and saw things between no effect and a slowdown as I got up close to -j20) and allowing 10 distcc connections to the server. The non-distcc builds were done with -j3, which is my normal build setup (since the mac in question only has 2 cores). distcc over ssh did not use ssh connection sharing, and was doing X forwarding on each ssh connection; I should retest with those two things changed at some point, if I decide I don't like doing things over tcp. The network is a wireless 802.11g on the Mac laptop end and gigabit ethernet to the wireless router on the Linux desktop. I found that ,lzo for the distcc builds didn't materially affect things (may have made the build 1-2% faster when used).

Build typeTime
local build on the Mac38 minutes
distcc over ssh50 minutes
distcc over tcp32 minutes

This was obviously distressing. I didn't test distcc over ssh after this, but particularly distressing was the fact that the compilation seemed largely diskbound and therefore distcc didn't help much. This got me started looking into hard drive performance as a function of utilization, since I could swear that the Mac's hard drive has been getting slower as it fills up. Turns out, this is actually the case. So I moved my rarely-used Linux VM to an external hard drive, which took free space on my internal disk from 4GB to 65GB or so. With that change done:

Build typeTime
local build on the Mac32 minutes
distcc over tcp22.5 minutes

Progress! Next, I tried dropping ccache, which let me measure the impact of pump mode. Since I do generally see a benefit from ccache, and wanted to measure it, I also tried a build without distcc but with a primed ccache (so build a tree, delete the objdir, rebuild that tree). Results:

Build typeTime
no-ccache distcc in pump mode16 minutes
no-ccache distcc without pump mode17 minutes
Local build coming entirely from ccache15 minutes

So if I were pretty sure I'd almost always be near this Linux box, just not using ccache and using pump mode might be the way to go. As it is, using distcc + ccache seems like a good approach for now.

Now all I need is a way to figure out a way to get this sort of data without it taking several hours, a way to pick good values for -j, and a faster internal hard drive in this laptop.

Posted by bzbarsky at 3:48 PM

January 8, 2010

Toolwhip, distcc, and faster compiles

Ted recently pointed me to the toolwhip project, which is a way to run Apple's gcc on Linux, cross-compiling to produce Mac binaries. Clearly desirable, since my Linux box is a lot faster than my Mac one. Here are the steps to get it working, as far as I can tell, at least if you Mac has OS X 10.5:

  1. Grab yourself a toolwhip tree following their directions.
  2. Follow the instructions in cctools.README to compile and install cctools. Note that the toolwhip tree comes with an existing usr_include so you can ignore the REQUIREMENTS section in this readme.
  3. Follow the instructions in ld64.README to compile and install ld64. Once again, you can ignore the REQUIREMENTS section.
  4. Download the Apple gcc source for the exact compiler version you want from the Apple site and untar it into a directory called gcc_42-5574 or gcc_42-5566 in your toolwhip source tree. If you plan to use this compiler with distcc, make sure it's the same exact version as that installed on your Mac. If you just plan to cross-compile, I'm not sure that it matters quite as much.
  5. Apply the relevant patch (gcc_42-5574.patch or gcc_42-5566.patch) to the gcc source tree.
  6. Follow the instructions in gcc_42-5574.README or gcc_42-5566.README to compile and install gcc. This time, follow the REQUIREMENTS section as needed; in particular you will need to copy whatever SDKs you want to your Linux box.
  7. Create the following symlinks:
      ln -s Developer/usr/bin/gcc-4.2 /usr/bin/
      ln -s /Developer/usr/bin/g++-4.2 /usr/bin/
    for all those makefiles that call the compiler with an absolute path.

At this point you should have a working cross-compiler. The remaining instructions are for setting up distcc with it.

  1. Grab a toolwhip tree on your Mac a well and follow the directions in distcc.README to build distcc.
  2. Follow the directions in distcc.README to build distcc on your Linux box.

What you do after this point probably depends on exactly how you plan to run distcc and distccd, whether you plan to use ccache, whether you want pump mode, etc. There are lots of directions in distcc.README. So far I have a setup that does distcc over ssh to my Linux box. I'm hoping to sort out the rest of the distcc stuff (not using ssh, and maybe using pump mode) sometime in the next week or so. To get what I have working so far going:

  1. Set PermitUserEnvironment to "yes" in /etc/ssh/sshd_config on the Linux box.
  2. Put DISTCCD_PATH=/Developer/usr/bin in your ~/.ssh/environment on the Linux box.

Now you run a test. In my case, I use ccache, so this worked:

  export DISTCC_HOSTS=bzbarsky@linux-box:/path/to/toolwhip/distccd/on/linux,lzo
  export CCACHE_PREFIX=/path/to/toolwhip/distcc/on/mac

and then compiling Firefox normally, using make -f If you don't use ccache, you would need to skip the CCACHE_PREFIX and make sure that symlinks called gcc-4.2 and g++-4.2 came in your PATH on the Mac before the real compilers and pointed to the toolwhip distcc. I think that should do the trick, but haven't tried it. The ",lzo" in the DISTCC_HOSTS above just says to gzip-compress the data; it can be left out if desired.

Question for those with some distcc experience: is pump mode worth forgoing ccache?

One last note: I have not yet verified that this gives me faster compiles! Linking has to happen on the distcc client, and that tends to be a pretty slow part... Perhaps I would be better off just compiling entirely on the Linux box then copying the resulting .dmg to the Mac.

UPDATE: While the above distcc setup works for compiling an already-configured Mozilla tree, it seems that trying to configure from scratch fails because it thinks it failed to find Core Text, or more precisely because it fails to find ANSI C headers. But only if you use DISTCC_VERBOSE=1, so don't do that. It also seems that actually cross-compiling from Linux doesn't work because libffi insists on passing gnu ld options to an ld it detected as not being gnu ld. That's very sad.

Posted by bzbarsky at 4:19 PM