First, there are a couple of caveats about Chromebug. It’s still in an experimental state. Some parts work well, like the inspection of a chrome window’s DOM tree and CSS. Other parts are a little flakey or down-right broken. As of right now, the default build instructions will attach you to a 1.3 version of Firebug which has some improvements to handling large script files but is also somewhat broken. It should improve shortly.

So, with that out of the way, I’ll run through a quick Chromebug setup and install. The instructions are on the sourceforge page for Fireclipse, a plugin for Eclipse that adds some debugging capabilities for Firefox and/or JavaScript.

Step 1. From a suitable directory on your computer, check out the chromebug sourcecode from svn. There are external svn references to include two other add-ons, chromelist and firebug. These will get checked out as well.

Step 2. Create a new profile for Firefox using the Profile Manager. I called mine “Chromebug”. Make sure you start Firefox with this new profile to finish the creation process.

Step 2a. Make note of the location of this new profile. You’ll need the full path for the following step. On Windows, this would typically be in “C:\Documents and Settings\%USERNAME%\Application Data\Mozilla\Firefox\Profiles\xxxxxxxx.Chromebug\”. Replace %USERNAME% with your username, and xxxxxxxxxx with some randomly generated string. Just look for *.Chromebug in that directory. You will need this for the “install.dir” property in step 4 below.

Step 3. In a console (or terminal window), cd to the chromebug directory you checked out earlier. E.g., cd ~/Projects/fireclipse/extensions/chromebug

Step 4. Create a local.properties file using your favorite text editor in this directory. Add the following properties to it (each property should be on a single line, regardless of how this blog breaks it up):

Step 5. Create links to the chromebug extensions. This can be most-easily accomplished using the build.xml file provided in the fireclipse/extensions/chromebug directory and the Java build utility ant. If you don’t have ant installed, you can do this manually as per the instructions in the included readme file. It’s standard extension development linkage.

And now you should be ready to fire it up using (on windows):

Replace firefox.exe with firefox-bin on unixey OSes. You should see two windows open. One, a smaller window with the title Chromebug, the other, a standard browser window. Before you get too excited and jump in, you should quit the application and restart it (again, using the above command) to make sure the chrome files get registered properly. Now you should be ready to play!

Please keep in mind that Chromebug is experimental. There’s a lot of code in there and not all of it’s working. We are interested in getting more people using it and reporting problems though so check it out, and if you encounter any problems, feel free to come to #firebug in irc.mozilla.org and ask questions. You may have better luck using Firebug 1.2 instead of 1.3 currently, but I haven’t tested that myself. It should be a matter of changing the firebug.dir property in the local.properties file from step 5 above.

I’d like to thank John Barton for patiently walking me through the above and giving me a walkthrough of the code itself. It’s incredibly cool stuff and I’d love to see it become part of the standard arsenal of extension development tools as well as part of the toolkit for developing firefox front end features. In the coming weeks, I’ll post follow-ups on the state of chromebug and what you can do with it.

Happy debugging!

Ok, that’s laying it on pretty thick, but I have a fond affection for a few of these machines. For instance, qm-xserve01 was the original Mac unittest Xserve and nothing else has been able to touch it for compile and test time. We recently dropped qm-xserve06 onto the farm and at best, it’s about 10 minutes slower end-to-end than xserve01. I don’t know why this is and it shouldn’t be – xserve06 is newer and sporting more recent processors and I believe a faster bus. See also qm-win2k3-01, our first physical box running Windows unittests. It’s been there since April 26th of last year, as Shaver once quipped, “qm-win2k3-01 taught me to love”. We’re still waiting for the t-shirts.

Why are there three of everything? We borrowed a trick from Talos. One of the difficulties of running a testing box is that you sometimes get funny results. There are a lot of moving pieces on these machines and we try to keep them as tight as possible, running as few extra components as we can on identical operating systems (across a single platform), but sometimes things just go a little funny. The thinking was having sets of three (triadic) machines would introduce some error-checking into the runs. One orange run out of three can be discarded. Two out of three and you might have something a little harder to reproduce. Three out of three and you close the tree and backout. I think that system works pretty well in most cases.

The one sad point we’ve really struggled with during this period is the linux platform. We’ve been running linux unittests on our Centos5 ReferencePlatform which is solid and well-maintained by Ben Hearsum and Nick Thomas. Running it on VMs introduces some variability that has been really hard to shake down. Tests involving timers are almost guaranteed to fail at some point. Sometimes, if disk access gets a little busy, we see errors in reftests loading some resources, usually pngs of even the smallest size. Then there are the random mochi* failures. These problems are hard to reproduce and even harder to diagnose, usually resulting in a shrug and someone shakily pushing the “force build” button on the waterfall page in the hopes that the problem will go away on the next build. It often does.

If you can shed any light on these on-going issues, we could use some help. Please see bugs:

• https://bugzilla.mozilla.org/show_bug.cgi?id=431745
• https://bugzilla.mozilla.org/show_bug.cgi?id=435064
• https://bugzilla.mozilla.org/show_bug.cgi?id=428865

Moving forward to mozilla-central, we’ve got a few machines chugging along there and I hear qm-win2k3-03 picked up its first real regression yesterday so things seem to be progressing nicely. Over the next few weeks, we’re going to give the Firefox3/Gecko1.9 machines a spit-polish, upgrading all of them to buildbot 0.7.7 and matching their installations to what we’re using on the new machinery. As for what the new machinery will look like, I have a sneaky suspicion it’s going to look a little bit like Talos.

I’m writing this in the hopes that we can make the process a little easier for those of you responsible for packaging Firefox (and hopefully other Mozilla apps down the road) on Linux.

First, the background: We do automated unit testing. We do it on every checkin and you can see the results posted to http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox. The machines responsible for producing those results are running buildbot. We’re currently using, a slightly customized version 0.7.5 (available from Mozilla’s cvs repo at cvs-mirror.mozilla.org in mozilla/tools/buildbot, branch BUILDBOT_BRANCH_0_7_5) on top of Twisted 2.4.0. Soon, we’ll be moving forward to version 0.7.7 and Twisted 8.x on all our machines as part of a Grand Unified Vision. More on that later.

We have some scattered documentation around the mozilla wikisphere that we’ll be consolidating and cleaning up over the summer. The important bits are: the main Mozilla Buildbot Wiki page at http://wiki.mozilla.org/Buildbot. Of some but decreasing importance is the BuildbotTestFarm setup page. It has links to the important buildbot master configuration and setup files, namely master.cfg and mozbuild.py in the unittest directory. Also, http://wiki.mozilla.org/BuildbotTestFarmUse has a really awesome diagram that I created (and recently updated!) that you’ll want to show to all your friends and possibly print out and hang on your cubicle wall.

Ok, let’s use all this to setup an example buildbot master and slave on your linux environment of choice. First, we’ll assume you’re capable of actually building the browser on your slave machine according to Linux_Build_Prerequisites. On your buildbot master (it can be the same machine as the slave if you’re just doing a single builder), create the master directory by typing “buildbot create-master unittest_master” (where unittest_master is your directory name, it can be anything). Copy master.cfg, mozbuild.py and the default mozconfig file into this newly-created directory and modify the .cfg file to taste. If you’re doing a straight copy and paste from our version, you can delete all those extra machine names and will have something that will look more like this: master.cfg. Rename URLs and links as appropriate. If you don’t use Tinderbox server for hosting your builds, you can remove the “c['status'].append(tinderbox.TinderboxMailNotifier(…” section entirely and just use the buildbot waterfall.

Once you’ve got that setup, start it up by running “buildbot start unittest_master” in the parent directory and you should see some log chatter as it starts up. You can verify by visiting localhost:2005 on the master machine and seeing a waterfall with an empty table in it. It’s waiting for a slave to connect. If you don’t see this, you’ve done something horribly wrong and will need to start from scratch, probably after thinking about it for a few minutes and maybe getting a cup of your favorite caffeinated warm beverage. This thoughtful reflection is important. I can’t stress this enough!

Now that you’re properly caffeinated, let’s create the slave. I’m going to assume you’re running it on the same machine as the one you’ve just created the master on. In a nearby directory, type “buildbot create-slave slave-dir localhost:9989 linux 1h34rtL1nuX”. It should create a directory named “slave-dir” and add a few files to it. Now the fun part: type “buildbot start slave-dir” and you should be up and running. Verify this on the buildbot waterfall page where you should see the slave connected and a build starting up.

At this point, I’m going to finish up with a few caveats. These instructions assume you’re using the same buildbot version we’ve used for these setups (version 0.7.5 running on twisted 2.4.0 with python 2.4 or 2.5). You’ll need to modify your master.cfg file slightly if you’re using a newer version. For excruciating details on how we setup these machines, see http://wiki.mozilla.org/ReferencePlatforms/Linux-CentOS-5.0. All machines running Reftests are required to have 24 or 32bit displays for image comparison. Lastly, the test machine will have to have run any version of Firefox before for the default profile to have been initialized.

The file “mozbuild.py” mentioned above contains the interesting bits. If you’re curious about how to run any of the individual unittests from a run, be sure to take a look in there at the “command=” parameters to the various testing classes. Good luck, and have fun!

What does this mean to you, the developer/tester/user? If you find a bug in one of the automated systems, you should now file that under “Mozilla.org:Release Engineering” in Bugzilla. I’ll still be monitoring Core:Testing but that should be used predominantly for Mozilla-related test harnesses and tools. We’re currently in the process of triaging Build & Release bugs and relevant bugs under Core:Testing and hope to have them re-assigned and re-bucketed within a week or two.

In other moving-related news, I’ve managed to reposition myself “closer to Greenland” as some have quipped. This shouldn’t really affect my availability for those of you in the West as I’ve shifted my hours up to compensate – this just means I get to sleep in a little more. What has been an issue is my ongoing struggle with Aliant.net to get my internet up to the speeds I’m actually paying for. They have a few more days to get this fixed and then I’m going to open up the bidding.

To the shamrock-wearers, happy St. Paddies day!

It was a fun work week in Mountain View. Despite the large number of toys available in the main area of Building K, there was a tremendous amount of work getting done on Firefox 3. Beta 3 freeze occurred (and continues) through stalwart sheriffing and sheer force of will, pushing a steady stream of patches being into the tree in preparation for its release. Mostly, I tried to stay out of everyone’s way.

On the test development and automation end of things, we had some productive meetings to set our direction for the next few months. We’re going to put some fast-cycling talos machines up to take over from the aging, test-only tinderboxes that have been keeping Rob Helmer up at night. These should come into being in the next couple of weeks and we’ll run them in parallel during the bake-in period to compare the numbers.

I also had the opportunity to sit down with Bob Clary and bring up a couple more machines for the JavaScript Testing project, aka “Sisyphus“. You can visit them on the MozillaTest tree on the tinderbox server — they’re the ones with “js test” in the title and are currently all orange due to tinderbox error parsing funniness. Over the next little while we’ll bring up the remaining platform (Mac OS X Intel) and fix up the displays so they’re only orange when they’re supposed to be orange.

But wait! There’s more! We’re also going to be beefing up the unittest farm with some redundant machinery. Each platform will become part of a triadic set similar to the triples of machines we have running talos tests. This is the first phase in building in some parallelization and error checking into these machines in an attempt to cut down on cycle time and improve reliability.

Finally, Chris Cooper improved the clobber support across all three platforms and applied a little special sauce on the Windows box that should dramatically reduce the number of stuck processes under Mozilla-build.