it’s the little things February 7th, 2007
That little page was my first view of the new testing buildbots living in the Mozilla colocation facility. It may not look like much yet, but it’s overflowing with potential. What does it do, you wonder? It runs test-enabled builds, then runs make check, reftest and mochitest. From the three columns, you might infer that it does this for the three major platforms, Windows XP, Linux and Mac OS X — or at least it will when the Xserve arrives at the colo.
You can see the results reporting to the MozillaTest tree on tinderbox.mozilla.org. Currently, the two operational buildbots are “linref test” and “winxp test” (don’t worry, I have an exciting naming scheme all picked-out and ready for action – stay tuned!). Check out the logs for examples of what the make check, reftest and mochitest reports look like. If winxp is orange or burning, it’s because we still have a couple of issues to work out. Some of the reftests are failing on that platform and dbaron is writing a modification to the manifest language so we can mark individual tests as failures on specific platforms. The other, more troubling bug is a debugger prompt I can’t seem to get rid of. Neither setting XPCOM_DEBUG_BREAK to “warn” nor adding a registry entry for windbgdlg (thanks timeless!) seemed to work. I’m hoping the problem will just go away with the addition of more RAM, but that seems unlikely.
On my todo list, I have some work to do to get the different tests reporting rolled-up stats to the tinderbox. We’ll be able to see why a tree is orange instead of looking through the full logs. There are a couple of little profile writing things I can make better. At some point, soon, I’m going to work with Bob Clary on getting the JS tests automated and reporting under this system. Then there’s going to be some work getting Alice Nodelman’s performance testing framework running under it and reporting to the tinderboxes. I’m also trying to setup a leak-testing build to replace the Tinderbox that just stopped working and weaseled its way to the front of my to-do list. Ultimately, I’m hoping we can have these on a more visible tinderbox, but they need to be stabilized first. Oh, and I need to check these configuration changes in and flesh out the very skimpy docs I wrote while putting this together.
Whew, this blog post would’ve been much shorter if I’d just written it last week when I had that one little picture.
Hopefully, having regularly running and reporting tests will encourage developers to write more of them. Individually, unit tests are tiny and kind of uninteresting. When you’ve got thousands of them running constantly though, then you’ve got something.
PS, special thanks to everybody who helped out along the way to get this stuff running. To the IT crew (mrz, justdave, aravind, justin, …) for putting up with the constant bug-filing back-and-forth for VMs I needed or VMs I’d busted, the tireless efforts of the build team (Rob Helmer and J. Paul Reed) taking time away from their real jobs to setup the linux reference VM, Rob Sayre and Dave Baron for their excellent test frameworks, Jeff Walden for the js http server, Ben Hearsum for his buildbot work and Dave L. for getting the ball rolling many months ago. All the developers who wrote tests… I’m being played off the stage… thanks Mom!
Technorati Tags: Build, Computing, Firefox, Software
Posted in Build, Code, Firefox, Quality, Testing | Comments (1)
May 28th, 2007 at 12:57 am
[...] The tool will give us this data in a tab-separated text file. Once we figure out how that works, it should be pretty easy for someone to put robcee’s buildbot work to use with Purify, and output a gigantic set of static HTML files after the runs complete. We’ll need a few extra switches for the harnesses, but nothing too complicated. [...]