Ok, that’s laying it on pretty thick, but I have a fond affection for a few of these machines. For instance, qm-xserve01 was the original Mac unittest Xserve and nothing else has been able to touch it for compile and test time. We recently dropped qm-xserve06 onto the farm and at best, it’s about 10 minutes slower end-to-end than xserve01. I don’t know why this is and it shouldn’t be – xserve06 is newer and sporting more recent processors and I believe a faster bus. See also qm-win2k3-01, our first physical box running Windows unittests. It’s been there since April 26th of last year, as Shaver once quipped, “qm-win2k3-01 taught me to love”. We’re still waiting for the t-shirts.

Why are there three of everything? We borrowed a trick from Talos. One of the difficulties of running a testing box is that you sometimes get funny results. There are a lot of moving pieces on these machines and we try to keep them as tight as possible, running as few extra components as we can on identical operating systems (across a single platform), but sometimes things just go a little funny. The thinking was having sets of three (triadic) machines would introduce some error-checking into the runs. One orange run out of three can be discarded. Two out of three and you might have something a little harder to reproduce. Three out of three and you close the tree and backout. I think that system works pretty well in most cases.

The one sad point we’ve really struggled with during this period is the linux platform. We’ve been running linux unittests on our Centos5 ReferencePlatform which is solid and well-maintained by Ben Hearsum and Nick Thomas. Running it on VMs introduces some variability that has been really hard to shake down. Tests involving timers are almost guaranteed to fail at some point. Sometimes, if disk access gets a little busy, we see errors in reftests loading some resources, usually pngs of even the smallest size. Then there are the random mochi* failures. These problems are hard to reproduce and even harder to diagnose, usually resulting in a shrug and someone shakily pushing the “force build” button on the waterfall page in the hopes that the problem will go away on the next build. It often does.

If you can shed any light on these on-going issues, we could use some help. Please see bugs:

• https://bugzilla.mozilla.org/show_bug.cgi?id=431745
• https://bugzilla.mozilla.org/show_bug.cgi?id=435064
• https://bugzilla.mozilla.org/show_bug.cgi?id=428865

Moving forward to mozilla-central, we’ve got a few machines chugging along there and I hear qm-win2k3-03 picked up its first real regression yesterday so things seem to be progressing nicely. Over the next few weeks, we’re going to give the Firefox3/Gecko1.9 machines a spit-polish, upgrading all of them to buildbot 0.7.7 and matching their installations to what we’re using on the new machinery. As for what the new machinery will look like, I have a sneaky suspicion it’s going to look a little bit like Talos.

These are still a work in progress and I’m still sorting out how clobbers are going to work (clobber files will live in Mercurial under http://hg.mozilla.org/build/buildbot-configs/mozilla2-unittest) as well as a few other hg-related end-pieces.

Lastly, a couple of other notes about failures, the Mac (OSX Darwin 9.2.2… qm-moz2mini01) and Windows (qm-win2k3-moz2-01) machines landed in a state of orange. Both are reporting reftest failures and will need some looking into.

Please file bugs machinery-related bugs against mozilla.org::Release Engineering.

See the details in bugs 420073 and 419759.

Many thanks to Mikeal Rogers for setting up the PGO box (twice!) and to Coop for slaving over the hot Windows ovens.

It was a fun work week in Mountain View. Despite the large number of toys available in the main area of Building K, there was a tremendous amount of work getting done on Firefox 3. Beta 3 freeze occurred (and continues) through stalwart sheriffing and sheer force of will, pushing a steady stream of patches being into the tree in preparation for its release. Mostly, I tried to stay out of everyone’s way.

On the test development and automation end of things, we had some productive meetings to set our direction for the next few months. We’re going to put some fast-cycling talos machines up to take over from the aging, test-only tinderboxes that have been keeping Rob Helmer up at night. These should come into being in the next couple of weeks and we’ll run them in parallel during the bake-in period to compare the numbers.

I also had the opportunity to sit down with Bob Clary and bring up a couple more machines for the JavaScript Testing project, aka “Sisyphus“. You can visit them on the MozillaTest tree on the tinderbox server — they’re the ones with “js test” in the title and are currently all orange due to tinderbox error parsing funniness. Over the next little while we’ll bring up the remaining platform (Mac OS X Intel) and fix up the displays so they’re only orange when they’re supposed to be orange.

But wait! There’s more! We’re also going to be beefing up the unittest farm with some redundant machinery. Each platform will become part of a triadic set similar to the triples of machines we have running talos tests. This is the first phase in building in some parallelization and error checking into these machines in an attempt to cut down on cycle time and improve reliability.

Finally, Chris Cooper improved the clobber support across all three platforms and applied a little special sauce on the Windows box that should dramatically reduce the number of stuck processes under Mozilla-build.

First, a picture. This is the way stuff happens now:

(edit – X axis is time, think of these diagrams as gantt charts with some extra lines showing what happens, i.e., where things come from and where they go)

A few of notes about this group of largely disconnected systems:

• Builds happen all the time.
• Nightlies are produced regardless of what the other machines are reporting, i.e., there is no feedback from any of the testing machines to the build system that produces nightlies.
• There is no direct route from the Try server to the build system. Worse, try builds are not being put through unittests leaving the burden of testing on the developer who writes the patch.

In a short while, this picture will morph (slightly) into this:

In this slightly better world:

• Builds happen on checkin
• Nightlies will still be produced from the last build at a certain hour
• Try server builds will be unit tested and run through Talos for performance testing

In this view, the build machines and unittest machines work on checked-in patches, in parallel. Talos picks up the builds once they’ve landed on staging from the build machines. This picture is somewhat simplified as we may have parallel build machines and we already have parallel Talos machines taking in builds as they become available.

Another benefit we should see is Talos results lining up more closely with the build they’re associated with. Currently, because the Tinderboxes keep churning builds, Talos runs at a lag, sometimes with a build or two in queue as they struggle to catch up. This should go away with some gaps during the day allowing the Talos boxes to maintain parity with the main build machines.

Of notable difference here, when a patch gets submitted to the Try Server, it runs through the full gamut of unittests and at the end of that, runs a “make package” on the objdir producing a light(ish) build and putting it into the staging area for consumption. At that point, the talos try server can pick it up and run it through the full performance tests for analysis. While that’s going on, people can download the try build and play with it to see if they’ve broken anything. That’s step one in the testing lifecycle of a patch. This should be reality very soon, thanks in part to some help from the good people at Seneca College.

But we can still do better.

In this future, utopian landscape, the unittest and build machines can pick up a patch and start churning and testing builds before handing them over to Talos. At a specified time of day, the nightly machines which have been waiting on the results of the test boxes can now pick up and run the day’s patches and produce a proper build devoid of any extraneous testing code and optimized for userland. This build then gets picked up by the Talos servers and run through the performance tests again. If and only if all of these testing stages complete, the build can be pushed to the update servers for wider dispersion. This should severely limit the number of bad builds we’re able to ship to the world.

Ok, I admit that these pictures aren’t wildly-different from one another. The differences are in the connections and how we use them. By adding some dependency on the testing machines, we can ensure that we’re building the most robust nightlies possible. There are a few caveats to get to this awesome future. The Windows test machines must become more solid. We’ve been struggling with these for almost a year now and the time has come to do something about it. Better displays for gathering at-a-glance tree information are becoming increasingly important as we cram more information onto the main Firefox Tinderbox page. This is going to increase over the year as the JS testing machines come online, possibly requiring a separate tests only page. As more tests are added to Talos, it will become harder to keep on top of the results as builds come in, so this will require some rapidly-accessible view onto that data.

It’s 2008. Do you believe in the future?

The other problem of note is that builds and testruns take a lot longer on this machine. I don’t know if it’s due to slow disk, slow hardware (should be the same as qm-xserve01? I think?) or some other reason (no, Time Machine is not enabled), but it’s going to cause problems if we have to wait on it to turn green.

So, I’m looking for help with these issues. Feel free to comment here or give me a poke on irc.

I’ve been picking at Bob Clary’s JavaScript testing frameworks, called Sisyphus from here-on out, for many months now. He’s been patiently explaining it to me and making it better and easier to use. So, at long last we have a linux machine reporting some results to the oft-burning MozillaTest tinderbox tree (as “Linux Sisyphus js”, “Linux Sisyphus opt browser” and “Linux Sisyphus debug browser”). Currently it’s running the JS test portion against trunk on checkin. At midnight and 6am, a nightly build happens (opt and then debug) building the browser and running the Browser JS tests using the Spider extension. These test suites have grown over the last ten years and the full set takes quite awhile to run against the browser. Also, the logs are LONG. Once we figure out what data we’re interested in, we’ll try to pare down the output and possibly setup a logserver to contain the logs, sparing tinderbox and our networks some of the agony. Expect the other platforms to follow along shortly.

Also, while I’m posting update-type things, the Windows Talos boxes on the main Firefox tree will soon be replaced by the Windows XP Minis. You have been warned…

currently listening to: Symbolistic White Walls from the album “Last of the Ghetto Astronauts” by Matthew Good Band

If this causes any serious problems, please let us know.