Mochitest By The Numbers
I spent the afternoon grabbing a new tree and running through a Mochitest run and looking at the output in Numbers. What follows is my simple analysis of the contents of Mochitest on Mac – the numbers are slightly different for other platforms but comparable.
- External sources 4555
- Mozilla contributors 43943
I want to be clear that these numbers may be slightly misleading. I can’t point at a single one of these and say with utter certainty that “that is a single unittest”. When we asked David Baron about the 20-odd-thousand tests included in Layout/Style, he told us:
There are a bunch of tests that test things combinatorically. In other words, for a long list of potential values for CSS properties, testing a large number of things related to that value’s parsing, serialization, and computation through different APIs.
Rob Sayre also had a note of caution regarding these figures:
The definition of what constitutes a single test is pretty fluid. I would just say “tens of thousands”
What we can say then, with some certainty is that Mozilla contributors have written, constructed or otherwise created tens of thousands of unit tests in just over 1 year. We’ve also outnumbered the external tests included in our test code by nearly 10 to 1. Areas we’re testing include, but are by no means limited to: content, canvas, layout, chrome, docshell, HTML parsing, XUL, … really too many areas to list like this. See the analysis doc for details.
I think this is a phenomenal achievement and would like to thank everyone who’s contributed. Clearly Mozilla has become a very test-focused project and this has enabled us to create better builds with demonstrably fewer regressions on nightlies and betas.
There’s always more to do. The external test-suites we’ve included could stand to be updated and augmented. Below are a set of bugs John Resig recently filed. If you’re a developer looking to hone your mochitest skills or want to contribute to our growing population of tests, pick one of these up and take a crack at it.