Orange ftw. Blog post to follow. http://twitpic.com/gcizx

Well, here’s the blog post.

That little orange box is the first successful run of Firebug’s FBTest test automation tool through buildbot. It’s currently posting to the MozillaTest Tinderbox.

This is phase 1. We’re currently running Firebug’s 1.5 branch out of SVN against the latest linux build available in the Firefox 3.6 tinderbox builds directory. These are kicked off by SVN checkins to Firebug and FBTest. To give better coverage, and be somewhat more useful to Firefox devs, we’re going to run the latest built version of Firebug 1.5 against Firefox 3.6 branch and 3.7 trunk after a successful Tinderbox build. This will give us a known quantity (a Firebug that works in 3.5) testing against the different Firefox versions for incompatibilities. This should allow us to spot regressions as they happen in Firefox and report them much more quickly.

Once these are setup, we’ll create a separate Firebug tree, or if the sheriffs and drivers are amenable to it, drop a column on the appropriate Firefox tree on Tinderbox. I also want to experiment with different notification systems.

Over the course of the day, we filed around 16 bugs, 1 of which was deemed invalid (Other Component, Weave), and 8 of those remaining were fixed since then.

But we’re not done yet. As of this writing, we have 1.4a27 available for you to try out. It features an even more simplified activation model allowing for more code cleanup under the covers. I also predict there will be an a28 including the fixes from this weekend real soon now. Keep an eye on http://getfirebug.com/releases/firebug for updates.

So, thanks again for helping out. Special props to jcm, tmyoung and jhillacre for their hard work. And a special thank you to aakashd and the rest of Moz QA for their assistance in making this happen.

Update: For now, I recommend users stick with 1.4a25.

Please be advised that Firebug 1.4a26 has been declared “Dead on Arrival”. Please don’t update to this version.

Our goal is to test through some basic features through litmus.mozilla.org (TBD), run Jan Odvarko’s test suite and/or a couple of online tutorials. The purpose of these tests is going to be to determine Firebug compatibility with Firefox 3.0 and Firefox 3.5b4.

Issues discovered will be logged on the Firebug issues tracker.

See the write-up on QMO on their events page. We’re hoping for a good turn out, so I hope to see you there.

Enter FireUnit! An extension for Firebug that aims to be able to test Firebug itself, and later on, perhaps be a useful framework for developing unittests for the web. It’s still very early on, but there’s a core emerging.

Installing FireUnit is currently a bit of a manual affair. Download the source code from the github repository and extract the downloaded zip or tar file to your local filesystem.

Open your profile/extensions directory for the Firefox profile that you use with Firebug. Create a new empty text file there with the name “fireunit@mozilla.com” (no quotes, of course). In this text file, add the full path to the FireUnit directory you unpacked in the previous step. E.g., “/Volumes/Data/Projects/FIREBUG/fireunit” in my case.

Test panel of FireUnit after running commandline.html

When you open/restart Firefox with this profile and open Firebug, you should now see a new “Test” panel in the list of tabs. Go to the location bar and type in / paste chrome://fireunit/content/test/commandline.html. You should see some flickering as FireUnit runs through the tests in that commandline.html. When done, you should see a bunch of passes listed in the Test panel (hopefully no failures!).

While I’m rambling about testing, I should mention that there will soon be a new alpha version of Firebug 1.3 (a4) available for download at getfirebug.com/releases/firebug1.3. This would be an ideal version for you to try out FireUnit on! For the more adventurous in the crowd, Chromebug 0.3a8 is R261 on https://fireclipse.svn.sourceforge.net/svnroot/fireclipse/trunk/FireclipseExtensions/chromebug.

I’m writing this in the hopes that we can make the process a little easier for those of you responsible for packaging Firefox (and hopefully other Mozilla apps down the road) on Linux.

First, the background: We do automated unit testing. We do it on every checkin and you can see the results posted to http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox. The machines responsible for producing those results are running buildbot. We’re currently using, a slightly customized version 0.7.5 (available from Mozilla’s cvs repo at cvs-mirror.mozilla.org in mozilla/tools/buildbot, branch BUILDBOT_BRANCH_0_7_5) on top of Twisted 2.4.0. Soon, we’ll be moving forward to version 0.7.7 and Twisted 8.x on all our machines as part of a Grand Unified Vision. More on that later.

We have some scattered documentation around the mozilla wikisphere that we’ll be consolidating and cleaning up over the summer. The important bits are: the main Mozilla Buildbot Wiki page at http://wiki.mozilla.org/Buildbot. Of some but decreasing importance is the BuildbotTestFarm setup page. It has links to the important buildbot master configuration and setup files, namely master.cfg and mozbuild.py in the unittest directory. Also, http://wiki.mozilla.org/BuildbotTestFarmUse has a really awesome diagram that I created (and recently updated!) that you’ll want to show to all your friends and possibly print out and hang on your cubicle wall.

Ok, let’s use all this to setup an example buildbot master and slave on your linux environment of choice. First, we’ll assume you’re capable of actually building the browser on your slave machine according to Linux_Build_Prerequisites. On your buildbot master (it can be the same machine as the slave if you’re just doing a single builder), create the master directory by typing “buildbot create-master unittest_master” (where unittest_master is your directory name, it can be anything). Copy master.cfg, mozbuild.py and the default mozconfig file into this newly-created directory and modify the .cfg file to taste. If you’re doing a straight copy and paste from our version, you can delete all those extra machine names and will have something that will look more like this: master.cfg. Rename URLs and links as appropriate. If you don’t use Tinderbox server for hosting your builds, you can remove the “c['status'].append(tinderbox.TinderboxMailNotifier(…” section entirely and just use the buildbot waterfall.

Once you’ve got that setup, start it up by running “buildbot start unittest_master” in the parent directory and you should see some log chatter as it starts up. You can verify by visiting localhost:2005 on the master machine and seeing a waterfall with an empty table in it. It’s waiting for a slave to connect. If you don’t see this, you’ve done something horribly wrong and will need to start from scratch, probably after thinking about it for a few minutes and maybe getting a cup of your favorite caffeinated warm beverage. This thoughtful reflection is important. I can’t stress this enough!

Now that you’re properly caffeinated, let’s create the slave. I’m going to assume you’re running it on the same machine as the one you’ve just created the master on. In a nearby directory, type “buildbot create-slave slave-dir localhost:9989 linux 1h34rtL1nuX”. It should create a directory named “slave-dir” and add a few files to it. Now the fun part: type “buildbot start slave-dir” and you should be up and running. Verify this on the buildbot waterfall page where you should see the slave connected and a build starting up.

At this point, I’m going to finish up with a few caveats. These instructions assume you’re using the same buildbot version we’ve used for these setups (version 0.7.5 running on twisted 2.4.0 with python 2.4 or 2.5). You’ll need to modify your master.cfg file slightly if you’re using a newer version. For excruciating details on how we setup these machines, see http://wiki.mozilla.org/ReferencePlatforms/Linux-CentOS-5.0. All machines running Reftests are required to have 24 or 32bit displays for image comparison. Lastly, the test machine will have to have run any version of Firefox before for the default profile to have been initialized.

The file “mozbuild.py” mentioned above contains the interesting bits. If you’re curious about how to run any of the individual unittests from a run, be sure to take a look in there at the “command=” parameters to the various testing classes. Good luck, and have fun!

I spent the afternoon grabbing a new tree and running through a Mochitest run and looking at the output in Numbers. What follows is my simple analysis of the contents of Mochitest on Mac – the numbers are slightly different for other platforms but comparable.

• External sources 4555

• Mozilla contributors 43943

I want to be clear that these numbers may be slightly misleading. I can’t point at a single one of these and say with utter certainty that “that is a single unittest”. When we asked David Baron about the 20-odd-thousand tests included in Layout/Style, he told us:

There are a bunch of tests that test things combinatorically. In other words, for a long list of potential values for CSS properties, testing a large number of things related to that value’s parsing, serialization, and computation through different APIs.

Rob Sayre also had a note of caution regarding these figures:

The definition of what constitutes a single test is pretty fluid. I would just say “tens of thousands”

What we can say then, with some certainty is that Mozilla contributors have written, constructed or otherwise created tens of thousands of unit tests in just over 1 year. We’ve also outnumbered the external tests included in our test code by nearly 10 to 1. Areas we’re testing include, but are by no means limited to: content, canvas, layout, chrome, docshell, HTML parsing, XUL, … really too many areas to list like this. See the analysis doc for details.

I think this is a phenomenal achievement and would like to thank everyone who’s contributed. Clearly Mozilla has become a very test-focused project and this has enabled us to create better builds with demonstrably fewer regressions on nightlies and betas.

There’s always more to do. The external test-suites we’ve included could stand to be updated and augmented. Below are a set of bugs John Resig recently filed. If you’re a developer looking to hone your mochitest skills or want to contribute to our growing population of tests, pick one of these up and take a crack at it.

• https://bugzilla.mozilla.org/show_bug.cgi?id=424813

• https://bugzilla.mozilla.org/show_bug.cgi?id=424814
• https://bugzilla.mozilla.org/show_bug.cgi?id=424816
• https://bugzilla.mozilla.org/show_bug.cgi?id=424818
• https://bugzilla.mozilla.org/show_bug.cgi?id=424819
• https://bugzilla.mozilla.org/show_bug.cgi?id=424820

First, a picture. This is the way stuff happens now:

(edit – X axis is time, think of these diagrams as gantt charts with some extra lines showing what happens, i.e., where things come from and where they go)

A few of notes about this group of largely disconnected systems:

• Builds happen all the time.
• Nightlies are produced regardless of what the other machines are reporting, i.e., there is no feedback from any of the testing machines to the build system that produces nightlies.
• There is no direct route from the Try server to the build system. Worse, try builds are not being put through unittests leaving the burden of testing on the developer who writes the patch.

In a short while, this picture will morph (slightly) into this:

In this slightly better world:

• Builds happen on checkin
• Nightlies will still be produced from the last build at a certain hour
• Try server builds will be unit tested and run through Talos for performance testing

In this view, the build machines and unittest machines work on checked-in patches, in parallel. Talos picks up the builds once they’ve landed on staging from the build machines. This picture is somewhat simplified as we may have parallel build machines and we already have parallel Talos machines taking in builds as they become available.

Another benefit we should see is Talos results lining up more closely with the build they’re associated with. Currently, because the Tinderboxes keep churning builds, Talos runs at a lag, sometimes with a build or two in queue as they struggle to catch up. This should go away with some gaps during the day allowing the Talos boxes to maintain parity with the main build machines.

Of notable difference here, when a patch gets submitted to the Try Server, it runs through the full gamut of unittests and at the end of that, runs a “make package” on the objdir producing a light(ish) build and putting it into the staging area for consumption. At that point, the talos try server can pick it up and run it through the full performance tests for analysis. While that’s going on, people can download the try build and play with it to see if they’ve broken anything. That’s step one in the testing lifecycle of a patch. This should be reality very soon, thanks in part to some help from the good people at Seneca College.

But we can still do better.

In this future, utopian landscape, the unittest and build machines can pick up a patch and start churning and testing builds before handing them over to Talos. At a specified time of day, the nightly machines which have been waiting on the results of the test boxes can now pick up and run the day’s patches and produce a proper build devoid of any extraneous testing code and optimized for userland. This build then gets picked up by the Talos servers and run through the performance tests again. If and only if all of these testing stages complete, the build can be pushed to the update servers for wider dispersion. This should severely limit the number of bad builds we’re able to ship to the world.

Ok, I admit that these pictures aren’t wildly-different from one another. The differences are in the connections and how we use them. By adding some dependency on the testing machines, we can ensure that we’re building the most robust nightlies possible. There are a few caveats to get to this awesome future. The Windows test machines must become more solid. We’ve been struggling with these for almost a year now and the time has come to do something about it. Better displays for gathering at-a-glance tree information are becoming increasingly important as we cram more information onto the main Firefox Tinderbox page. This is going to increase over the year as the JS testing machines come online, possibly requiring a separate tests only page. As more tests are added to Talos, it will become harder to keep on top of the results as builds come in, so this will require some rapidly-accessible view onto that data.

It’s 2008. Do you believe in the future?

The other problem of note is that builds and testruns take a lot longer on this machine. I don’t know if it’s due to slow disk, slow hardware (should be the same as qm-xserve01? I think?) or some other reason (no, Time Machine is not enabled), but it’s going to cause problems if we have to wait on it to turn green.

So, I’m looking for help with these issues. Feel free to comment here or give me a poke on irc.

reggie_se -S 0b1100 -b 4:1 -i a -n IA32_CLOCK_MODULATION

the important bit there is the -S option (set) with 0b1100. The first 1 is to enable clock modulation, the “100″ corresponds to the following table:

000B Reserved
001B 12.5% (Default)
010B 25.0%
011B 37.5%
100B 50.0%
101B 63.5%
110B 75%
111B 87.5%

Oh, and one final note, you might want to take a snapshot of your registers before you begin using:

reggie_se -r -i a -n IA32_CLOCK_MODULATION

You can turn off speed-stepping by disabling the first bit in the first command (-S 0b0001 to set back to default), or, if you just want to be done with this nonsense, a reboot will get you back to starting state.

happy underclocking!