May 9, 2007

Testcase redux: Mozilla tools?

One of my "favorite" phrases in working on bugs is "reduced testcase". It's not always easy to do that, though, and I'm starting to think about ways to do that. I'm posting this now to get some feedback.

Please read the extended entry for details (it's a bit technical).

In Verbosio right now there's a strange cross-platform layout bug. I've got an iframe with a yellow background on top of another iframe. Windows shows the yellow background; Linux and Mac don't. This sounds like a good need for a layout reftest. Unfortunately, to get to the strange bustage, you have to go through about five steps. Reducing that to a simple reftest does not sound easy.

I'm personally thinking we could do something about that:

  1. Serialize the document, taking anonymous content into the real world (and replacing the bound nodes with XUL boxes).
  2. Load the transformed document source into a new window (Test Window A)
  3. Use DOM Inspector on Test Window A to remove DOM nodes which don't affect the bustage. (If you remove something that does affect the bustage, DOM-I has a nice undo feature.)
  4. Serialize the document in Test Window A again and save it as the first part of the reftest.
  5. Build another XUL document with only <xul:box/> elements, appropriate styling and text by script, which appears identical to Test Window A. Load the new document into Test Window B.
  6. Verify Test Window B renders identically to Test Window A.
  7. Serialize Test Window B's document as the second half of the reftest.

That's a long-winded way of saying "Record, reduce, generate a comparison document, save."

For building a xpcshell-based testcase and reducing it, there might be other possibilities. Shane Caraveo's Casper record/replay harness could be a starting point, along with Venkman for tracing:

  1. Use Casper to record your actions.
  2. Replay those actions step by step, with Venkman running this time.
  3. Record calls from the user interface into component code, including what the arguments were. (Where an argument isn't a literal value like a string or a number, retrace the recording to find out how the object was created, and record the object's creation too.) Each call is a "step".
  4. Store the sequence of recorded steps as the testcase.

In short, this means "Record, reduce, save." Not too different from the procedure I suggest for a layout reftest generator tool.

I tried pitching the above xpcshell test idea to Silver several weeks ago, but I don't think I was very clear then. I'm not even sure I'm clear now. Also, for Verbosio, I can't use xpcshell on my custom components because they're not native to XULRunner (see bug 359830 for details). But I have a test engine component which does basically the same thing.

Please let me know what you think. Are tools like this feasible? Do you have better ideas for implementing them? Can you think of similar tool designs for reducing testcases? Are you willing to write tools like these?

Posted by WeirdAl at May 9, 2007 1:47 PM