« June 2005 | Main | February 2007 »
November 30, 2006
bzr/hg/git performance
This posting is related to Paul Reed's recent posting about our investigation into switching away from CVS for our Mozilla 2.0 work. As you can read in Paul's post, the two alternatives on the plate at the moment are Mercurial and Bazaar, both of which are primarily written in Python (Bazaar is completely written in Python, Mercurial has some bottlenecks written in C from what I understand).
When we started talking about this I decided to do some performance tests to see how well these systems keep up with GIT, which I've been using for quite some time now and really like. Unfortunately GIT doesn't work anywhere near well enough on Windows, so using GIT is out of the question.
As Brett found out, Bazaar seems to be on the order of 2 to 3 times slower than Mercurial, which sounds bad, but depending on the actual performance might not really matter. So I decided to do some more unscientific tests to see how the performance of these two systems would compare to GIT, which seems really snappy with a repository containing all of Firefox and Thunderbird's source, including tests etc, and CVS/ directories.
Brett already tested commit speed and compared it. My first test (and only so far) was diff performance, as I tend to look at intermediate changes as I work fairly frequently. Here's what I did with Mercurial (hg), Bazaar (bzr). I already had a GIT (git) repository set up, so I used that.
- Initialize a fresh repository
- Add the whole Mozilla tree
- Commit
Once that was done, I made a one-line change to the file dom/src/base/nsDOMClassInfo.cpp, and did a set of diff tests and got the following results (all numbers are best of 3 runs, back to back on the same mostly idle computer):
| Operation | bzr (0.12.0c1) | hg (0.9) | git (1.4.2.4) |
| diff (top level) | 16.957 | 5.600 | 1.572 |
| diff dom/ | 10.596 | 2.240 | 0.140 |
| diff dom/src/ | 10.504 | 2.212 | 0.124 |
| diff dom/src/base/ | 10.468 | 2.212 | 0.124 |
| diff dom/src/base/nsDOMClassInfo.cpp | 10.472 | 2.084 | 0.116 |
| diff dom/src/base/nsGlobalWindow.cpp | 10.012 | 2.024 | 0.088 |
| diff in dom/ | 16.833 | 5.548 | 0.136 |
| diff in dom/src/base/ | 16.881 | 5.504 | 0.112 |
What's interesting in this data is that bzr takes a huge amount of time to do a diff operation even if you explicitly tell it to only check part of the source tree (either by changing into a subdirectory or giving it a directory or file name on the command line) as it does if you do a top-level diff (~2/3 of the time). hg appears to have partially solved this, but not in all cases. Another interesting note is that explicitly diffing a file that has no changes takes essentially as long as diffing a file with changes.
So what does this all mean? Well, to me personally it means Bazaar is not yet ready for a repository the size of Mozilla. Mercurial I can live with, even if it's not snappy. Git is fast (and yeah, I kind of wish we could use it).
Posted by jst at 1:57 PM | Comments (13)