« Authentication | Main | Chronicle Update »
May 20, 2007
Speeding Up Mkdepend
I did a little bit of research into the problem of mkdepend.exe being slow under VMWare with sources in VMWare's shared file system.
The first thing to do was to measure the problem. This was pretty easy: running mkdepend on nsPresShell.cpp consistently takes 32 seconds. Actually compiling the file takes only a few seconds. So clearly something is very wrong.
Poor-man's profiling in the debugger shows that mkdepend is spending all its time in stat() calls. These stat calls are just checking to see whether a file exists; on a hunch I wondered if there's a faster way to check for file existence than stat(). Some Web searches suggested that GetFileAttributesEx is a good fast way. So I refactored the code to use a does_file_exist function which calls GetFileAttributesEx on Windows. Ta da! Now mkdepend on nsPresShell.cpp takes 1.5 seconds. GAME OVER. Patch submitted.
I conjecture that stat() is so slow because it fills in a bunch of information about devices, inode numbers, and access permissions, and on Windows that may require several filesystem operations to get that information. Perhaps VMWare's file system is particularly slow at some of those extra operations.
Posted by roc at May 20, 2007 9:45 AM
Comments
I'm going to ask a naive question (given I don't know the material at all) for a bit - did you check if similar timesaving stuff exists on Mac or Linux? This sounds like a boon which will be good to have on any platform, really!
Posted by: Gijs at May 20, 2007 10:20 AM
what bug was that?
Posted by: Shawn Wilsher at May 20, 2007 11:43 AM
Unsurprisingly nsLocalFileWin::Exists already uses GetFileAttributesEx().
Posted by: Neil at May 20, 2007 11:48 AM
Yay! But the same argument applies to Unix: access(2) is faster than stat(2). Care to test?
/be
Posted by: Brendan Eich at May 20, 2007 4:48 PM
Shawn: 381247
Brendan, Gijs: on Mac and Linux I believe we don't use mkdepend normally, we generate the information with gcc -MD as a byproduct of normal compilation.
Posted by: Robert O'Callahan at May 20, 2007 7:48 PM
Oh, I was misremembering http://benjamin.smedbergs.us/blog/2006-11-22/depths-of-the-mozilla-build-system/ -- amazing the lengths we go to for always-fresh dependencies.
/be
Posted by: Brendan Eich at May 20, 2007 9:39 PM
Amazing. How much it speeds up the whole compilation of firefox?
Posted by: Diego Calleja at May 21, 2007 5:16 AM
Can't wait to see what this does for windows build times for the release team since we use VMWare tehre as well...
Posted by: schrep at May 21, 2007 5:23 AM
schrep: might help a bit but I think those builds have the sources on a local filesystem, so we won't see the huge penalty.
Posted by: Robert O'Callahan at May 21, 2007 10:29 AM