Orange ftw. Blog post to follow. http://twitpic.com/gcizx

Well, here’s the blog post.

That little orange box is the first successful run of Firebug’s FBTest test automation tool through buildbot. It’s currently posting to the MozillaTest Tinderbox.

This is phase 1. We’re currently running Firebug’s 1.5 branch out of SVN against the latest linux build available in the Firefox 3.6 tinderbox builds directory. These are kicked off by SVN checkins to Firebug and FBTest. To give better coverage, and be somewhat more useful to Firefox devs, we’re going to run the latest built version of Firebug 1.5 against Firefox 3.6 branch and 3.7 trunk after a successful Tinderbox build. This will give us a known quantity (a Firebug that works in 3.5) testing against the different Firefox versions for incompatibilities. This should allow us to spot regressions as they happen in Firefox and report them much more quickly.

Once these are setup, we’ll create a separate Firebug tree, or if the sheriffs and drivers are amenable to it, drop a column on the appropriate Firefox tree on Tinderbox. I also want to experiment with different notification systems.

Please be advised that Firebug 1.4a26 has been declared “Dead on Arrival”. Please don’t update to this version.

Our goal is to test through some basic features through litmus.mozilla.org (TBD), run Jan Odvarko’s test suite and/or a couple of online tutorials. The purpose of these tests is going to be to determine Firebug compatibility with Firefox 3.0 and Firefox 3.5b4.

Issues discovered will be logged on the Firebug issues tracker.

See the write-up on QMO on their events page. We’re hoping for a good turn out, so I hope to see you there.

Enter FireUnit! An extension for Firebug that aims to be able to test Firebug itself, and later on, perhaps be a useful framework for developing unittests for the web. It’s still very early on, but there’s a core emerging.

Installing FireUnit is currently a bit of a manual affair. Download the source code from the github repository and extract the downloaded zip or tar file to your local filesystem.

Open your profile/extensions directory for the Firefox profile that you use with Firebug. Create a new empty text file there with the name “fireunit@mozilla.com” (no quotes, of course). In this text file, add the full path to the FireUnit directory you unpacked in the previous step. E.g., “/Volumes/Data/Projects/FIREBUG/fireunit” in my case.

Test panel of FireUnit after running commandline.html

When you open/restart Firefox with this profile and open Firebug, you should now see a new “Test” panel in the list of tabs. Go to the location bar and type in / paste chrome://fireunit/content/test/commandline.html. You should see some flickering as FireUnit runs through the tests in that commandline.html. When done, you should see a bunch of passes listed in the Test panel (hopefully no failures!).

While I’m rambling about testing, I should mention that there will soon be a new alpha version of Firebug 1.3 (a4) available for download at getfirebug.com/releases/firebug1.3. This would be an ideal version for you to try out FireUnit on! For the more adventurous in the crowd, Chromebug 0.3a8 is R261 on https://fireclipse.svn.sourceforge.net/svnroot/fireclipse/trunk/FireclipseExtensions/chromebug.

What became pretty clear was that there was only a very negligible performance hit for having Firebug installed. Better yet, it’s quite performant even with the Firebug console enabled, meaning you can keep Firebug on your default profile without worrying about impacting your regular browsing.

Memory performance was similar and even with the script debugger enabled, memory usage peaked and levelled off indicating that there were no real leaks visible, even though pages took noticeably longer to load.

This isn’t to say that we’re sitting idly on our hands congratulating everyone on a job well-done. John Barton noticed a particular JavaScript test that was noticeably slower than it’s neighbors and it jogged his memory about some testing he’d done earlier in the 1.2 development cycle. This should translate into improved eval script performance in 1.3 and future versions of Firebug.

I’m looking forward to revisiting these benchmarks down the line to track our improvements.

Ok, that’s laying it on pretty thick, but I have a fond affection for a few of these machines. For instance, qm-xserve01 was the original Mac unittest Xserve and nothing else has been able to touch it for compile and test time. We recently dropped qm-xserve06 onto the farm and at best, it’s about 10 minutes slower end-to-end than xserve01. I don’t know why this is and it shouldn’t be – xserve06 is newer and sporting more recent processors and I believe a faster bus. See also qm-win2k3-01, our first physical box running Windows unittests. It’s been there since April 26th of last year, as Shaver once quipped, “qm-win2k3-01 taught me to love”. We’re still waiting for the t-shirts.

Why are there three of everything? We borrowed a trick from Talos. One of the difficulties of running a testing box is that you sometimes get funny results. There are a lot of moving pieces on these machines and we try to keep them as tight as possible, running as few extra components as we can on identical operating systems (across a single platform), but sometimes things just go a little funny. The thinking was having sets of three (triadic) machines would introduce some error-checking into the runs. One orange run out of three can be discarded. Two out of three and you might have something a little harder to reproduce. Three out of three and you close the tree and backout. I think that system works pretty well in most cases.

The one sad point we’ve really struggled with during this period is the linux platform. We’ve been running linux unittests on our Centos5 ReferencePlatform which is solid and well-maintained by Ben Hearsum and Nick Thomas. Running it on VMs introduces some variability that has been really hard to shake down. Tests involving timers are almost guaranteed to fail at some point. Sometimes, if disk access gets a little busy, we see errors in reftests loading some resources, usually pngs of even the smallest size. Then there are the random mochi* failures. These problems are hard to reproduce and even harder to diagnose, usually resulting in a shrug and someone shakily pushing the “force build” button on the waterfall page in the hopes that the problem will go away on the next build. It often does.

If you can shed any light on these on-going issues, we could use some help. Please see bugs:

• https://bugzilla.mozilla.org/show_bug.cgi?id=431745
• https://bugzilla.mozilla.org/show_bug.cgi?id=435064
• https://bugzilla.mozilla.org/show_bug.cgi?id=428865

Moving forward to mozilla-central, we’ve got a few machines chugging along there and I hear qm-win2k3-03 picked up its first real regression yesterday so things seem to be progressing nicely. Over the next few weeks, we’re going to give the Firefox3/Gecko1.9 machines a spit-polish, upgrading all of them to buildbot 0.7.7 and matching their installations to what we’re using on the new machinery. As for what the new machinery will look like, I have a sneaky suspicion it’s going to look a little bit like Talos.

I’m writing this in the hopes that we can make the process a little easier for those of you responsible for packaging Firefox (and hopefully other Mozilla apps down the road) on Linux.

First, the background: We do automated unit testing. We do it on every checkin and you can see the results posted to http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox. The machines responsible for producing those results are running buildbot. We’re currently using, a slightly customized version 0.7.5 (available from Mozilla’s cvs repo at cvs-mirror.mozilla.org in mozilla/tools/buildbot, branch BUILDBOT_BRANCH_0_7_5) on top of Twisted 2.4.0. Soon, we’ll be moving forward to version 0.7.7 and Twisted 8.x on all our machines as part of a Grand Unified Vision. More on that later.

We have some scattered documentation around the mozilla wikisphere that we’ll be consolidating and cleaning up over the summer. The important bits are: the main Mozilla Buildbot Wiki page at http://wiki.mozilla.org/Buildbot. Of some but decreasing importance is the BuildbotTestFarm setup page. It has links to the important buildbot master configuration and setup files, namely master.cfg and mozbuild.py in the unittest directory. Also, http://wiki.mozilla.org/BuildbotTestFarmUse has a really awesome diagram that I created (and recently updated!) that you’ll want to show to all your friends and possibly print out and hang on your cubicle wall.

Ok, let’s use all this to setup an example buildbot master and slave on your linux environment of choice. First, we’ll assume you’re capable of actually building the browser on your slave machine according to Linux_Build_Prerequisites. On your buildbot master (it can be the same machine as the slave if you’re just doing a single builder), create the master directory by typing “buildbot create-master unittest_master” (where unittest_master is your directory name, it can be anything). Copy master.cfg, mozbuild.py and the default mozconfig file into this newly-created directory and modify the .cfg file to taste. If you’re doing a straight copy and paste from our version, you can delete all those extra machine names and will have something that will look more like this: master.cfg. Rename URLs and links as appropriate. If you don’t use Tinderbox server for hosting your builds, you can remove the “c['status'].append(tinderbox.TinderboxMailNotifier(…” section entirely and just use the buildbot waterfall.

Once you’ve got that setup, start it up by running “buildbot start unittest_master” in the parent directory and you should see some log chatter as it starts up. You can verify by visiting localhost:2005 on the master machine and seeing a waterfall with an empty table in it. It’s waiting for a slave to connect. If you don’t see this, you’ve done something horribly wrong and will need to start from scratch, probably after thinking about it for a few minutes and maybe getting a cup of your favorite caffeinated warm beverage. This thoughtful reflection is important. I can’t stress this enough!

Now that you’re properly caffeinated, let’s create the slave. I’m going to assume you’re running it on the same machine as the one you’ve just created the master on. In a nearby directory, type “buildbot create-slave slave-dir localhost:9989 linux 1h34rtL1nuX”. It should create a directory named “slave-dir” and add a few files to it. Now the fun part: type “buildbot start slave-dir” and you should be up and running. Verify this on the buildbot waterfall page where you should see the slave connected and a build starting up.

At this point, I’m going to finish up with a few caveats. These instructions assume you’re using the same buildbot version we’ve used for these setups (version 0.7.5 running on twisted 2.4.0 with python 2.4 or 2.5). You’ll need to modify your master.cfg file slightly if you’re using a newer version. For excruciating details on how we setup these machines, see http://wiki.mozilla.org/ReferencePlatforms/Linux-CentOS-5.0. All machines running Reftests are required to have 24 or 32bit displays for image comparison. Lastly, the test machine will have to have run any version of Firefox before for the default profile to have been initialized.

The file “mozbuild.py” mentioned above contains the interesting bits. If you’re curious about how to run any of the individual unittests from a run, be sure to take a look in there at the “command=” parameters to the various testing classes. Good luck, and have fun!

I spent the afternoon grabbing a new tree and running through a Mochitest run and looking at the output in Numbers. What follows is my simple analysis of the contents of Mochitest on Mac – the numbers are slightly different for other platforms but comparable.

• External sources 4555

• Mozilla contributors 43943

I want to be clear that these numbers may be slightly misleading. I can’t point at a single one of these and say with utter certainty that “that is a single unittest”. When we asked David Baron about the 20-odd-thousand tests included in Layout/Style, he told us:

There are a bunch of tests that test things combinatorically. In other words, for a long list of potential values for CSS properties, testing a large number of things related to that value’s parsing, serialization, and computation through different APIs.

Rob Sayre also had a note of caution regarding these figures:

The definition of what constitutes a single test is pretty fluid. I would just say “tens of thousands”

What we can say then, with some certainty is that Mozilla contributors have written, constructed or otherwise created tens of thousands of unit tests in just over 1 year. We’ve also outnumbered the external tests included in our test code by nearly 10 to 1. Areas we’re testing include, but are by no means limited to: content, canvas, layout, chrome, docshell, HTML parsing, XUL, … really too many areas to list like this. See the analysis doc for details.

I think this is a phenomenal achievement and would like to thank everyone who’s contributed. Clearly Mozilla has become a very test-focused project and this has enabled us to create better builds with demonstrably fewer regressions on nightlies and betas.

There’s always more to do. The external test-suites we’ve included could stand to be updated and augmented. Below are a set of bugs John Resig recently filed. If you’re a developer looking to hone your mochitest skills or want to contribute to our growing population of tests, pick one of these up and take a crack at it.

• https://bugzilla.mozilla.org/show_bug.cgi?id=424813

• https://bugzilla.mozilla.org/show_bug.cgi?id=424814
• https://bugzilla.mozilla.org/show_bug.cgi?id=424816
• https://bugzilla.mozilla.org/show_bug.cgi?id=424818
• https://bugzilla.mozilla.org/show_bug.cgi?id=424819
• https://bugzilla.mozilla.org/show_bug.cgi?id=424820

It was a fun work week in Mountain View. Despite the large number of toys available in the main area of Building K, there was a tremendous amount of work getting done on Firefox 3. Beta 3 freeze occurred (and continues) through stalwart sheriffing and sheer force of will, pushing a steady stream of patches being into the tree in preparation for its release. Mostly, I tried to stay out of everyone’s way.

On the test development and automation end of things, we had some productive meetings to set our direction for the next few months. We’re going to put some fast-cycling talos machines up to take over from the aging, test-only tinderboxes that have been keeping Rob Helmer up at night. These should come into being in the next couple of weeks and we’ll run them in parallel during the bake-in period to compare the numbers.

I also had the opportunity to sit down with Bob Clary and bring up a couple more machines for the JavaScript Testing project, aka “Sisyphus“. You can visit them on the MozillaTest tree on the tinderbox server — they’re the ones with “js test” in the title and are currently all orange due to tinderbox error parsing funniness. Over the next little while we’ll bring up the remaining platform (Mac OS X Intel) and fix up the displays so they’re only orange when they’re supposed to be orange.

But wait! There’s more! We’re also going to be beefing up the unittest farm with some redundant machinery. Each platform will become part of a triadic set similar to the triples of machines we have running talos tests. This is the first phase in building in some parallelization and error checking into these machines in an attempt to cut down on cycle time and improve reliability.

Finally, Chris Cooper improved the clobber support across all three platforms and applied a little special sauce on the Windows box that should dramatically reduce the number of stuck processes under Mozilla-build.