~robcee/ more than just sandwiches

27 March 2008 @ 1pm

Mozilla, Testing

Mochitest By The Numbers

In an email discussion yesterday, Schrep posed the perfectly cromulent question, how many unittests are we running and where did they come from? A good chunk of our testing comes from a few popular JavaScript programming libraries that some of our more intrepid test coders translated to run inside Mochitest. I knew we’d translated jQuery, Scriptaculous and Mochikit‘s own test frameworks, to name a few, but what else was in there? And what portion of the 50,000 odd mochitests were from large external frameworks?

I spent the afternoon grabbing a new tree and running through a Mochitest run and looking at the output in Numbers. What follows is my simple analysis of the contents of Mochitest on Mac – the numbers are slightly different for other platforms but comparable.

  • External sources 4555

  • Mozilla contributors 43943

I want to be clear that these numbers may be slightly misleading. I can’t point at a single one of these and say with utter certainty that “that is a single unittest”. When we asked David Baron about the 20-odd-thousand tests included in Layout/Style, he told us:

There are a bunch of tests that test things combinatorically. In other words, for a long list of potential values for CSS properties, testing a large number of things related to that value’s parsing, serialization, and computation through different APIs.

Rob Sayre also had a note of caution regarding these figures:

The definition of what constitutes a single test is pretty fluid. I would just say “tens of thousands”

What we can say then, with some certainty is that Mozilla contributors have written, constructed or otherwise created tens of thousands of unit tests in just over 1 year. We’ve also outnumbered the external tests included in our test code by nearly 10 to 1. Areas we’re testing include, but are by no means limited to: content, canvas, layout, chrome, docshell, HTML parsing, XUL, … really too many areas to list like this. See the analysis doc for details.

I think this is a phenomenal achievement and would like to thank everyone who’s contributed. Clearly Mozilla has become a very test-focused project and this has enabled us to create better builds with demonstrably fewer regressions on nightlies and betas.

There’s always more to do. The external test-suites we’ve included could stand to be updated and augmented. Below are a set of bugs John Resig recently filed. If you’re a developer looking to hone your mochitest skills or want to contribute to our growing population of tests, pick one of these up and take a crack at it.

1 Comment

[...] think a lot of the uncertainty about testing comes from the fact that, until recently, there was almost no automated testing. And so we’re going through a period of growing pains [...]