A lot’s changed since those dark days, however. To say that we spend a lot of energy doing performance analysis is a bit of an understatement. We have entire teams dedicated to improving performance. Build options are analyzed and re-analyzed in an attempt to find cheap performance improvements. Specifying global code optimization settings can actually have an adverse effect on overall system performance because certain modules react better to, say, -O2 versus -O3, for example. All code that gets checked in has to make it through our performance tracking system or it gets backed out.

In the interest of Science, I wanted to actually see if there were any differences, so I pulled a fresh copy of mozilla-central and applied the referenced .mozconfig.

The first thing I noticed, was that –enable-profile-guided-optimization didn’t seem to be doing what I thought it would. Namely, I expected to see a browser window popup and run through some automation before starting a second compilation phase. It didn’t. Even so, the resultant build did feel a little faster at opening windows and tabs*.

So, I applied the latest stand-alone version of Talos to the build and executed the Ts (startup) and Twinopen (Window open) tests and compared this with the most-recent Mozilla nightly build of Minefield. Surprisingly, the windows didn’t appear to be opening much faster, but the startup time was on average 5% faster in the new “optimized” build vs. moz-central nightlies.

By this time, Ted had commented in the bug tracking all this and pointed me to the PGO page on MDC. It turns out, PGO builds require a separate make target “profiledbuild” instead of “build”. I reran the build, saw the automation was working this time and then reran the Talos tests. No change. Best guess around these parts is that GCC just isn’t very good at doing PGO. The same option gives us an across-the-board 10% speedup to all our tests on Windows.

It turned out that the option that was giving us the 5% startup time (and likely a slight boost to tab performance) was the 10.6 SDK target. Each revision of OS X includes optimizations and Snow Leopard is no different. If you want your own optimized build, just put that target (–with-macos-sdk=/Developer/SDKs/MacOSX10.6.sdk) in there if you’re on Snow Leopard and you’ll get a little extra snappy.

I haven’t booted into the 64 bit kernel to see if there’s any difference there, or even if the build options produce a 64 bit build. I don’t believe they do and the version shipped on the above optimized builds page don’t have the tell-tale “Open in 32 bit mode” checkbox in the app’s “Get Info” dialog.

* A word about perceived performance differences: It’s very hard to tell the difference between, say, 20ms and 30ms. That’s a 50% slow-down compared to 20ms but the actual slices of time are so small that the human brain has a hard time picking up on them. Nevertheless, people used to dealing with high-velocity timings (musicians, gamers, fighter pilots) can actually detect that kind of change. But there’s also a “wishful thinking” factor. It’s very easy to be told you’re going to see a speedup, and when given an identical application, think you’re seeing something faster. “It just feels snappier” you might say when given this placebo.

What does this mean to you, the developer/tester/user? If you find a bug in one of the automated systems, you should now file that under “Mozilla.org:Release Engineering” in Bugzilla. I’ll still be monitoring Core:Testing but that should be used predominantly for Mozilla-related test harnesses and tools. We’re currently in the process of triaging Build & Release bugs and relevant bugs under Core:Testing and hope to have them re-assigned and re-bucketed within a week or two.

In other moving-related news, I’ve managed to reposition myself “closer to Greenland” as some have quipped. This shouldn’t really affect my availability for those of you in the West as I’ve shifted my hours up to compensate – this just means I get to sleep in a little more. What has been an issue is my ongoing struggle with Aliant.net to get my internet up to the speeds I’m actually paying for. They have a few more days to get this fixed and then I’m going to open up the bidding.

To the shamrock-wearers, happy St. Paddies day!

First, a picture. This is the way stuff happens now:

(edit – X axis is time, think of these diagrams as gantt charts with some extra lines showing what happens, i.e., where things come from and where they go)

A few of notes about this group of largely disconnected systems:

• Builds happen all the time.
• Nightlies are produced regardless of what the other machines are reporting, i.e., there is no feedback from any of the testing machines to the build system that produces nightlies.
• There is no direct route from the Try server to the build system. Worse, try builds are not being put through unittests leaving the burden of testing on the developer who writes the patch.

In a short while, this picture will morph (slightly) into this:

In this slightly better world:

• Builds happen on checkin
• Nightlies will still be produced from the last build at a certain hour
• Try server builds will be unit tested and run through Talos for performance testing

In this view, the build machines and unittest machines work on checked-in patches, in parallel. Talos picks up the builds once they’ve landed on staging from the build machines. This picture is somewhat simplified as we may have parallel build machines and we already have parallel Talos machines taking in builds as they become available.

Another benefit we should see is Talos results lining up more closely with the build they’re associated with. Currently, because the Tinderboxes keep churning builds, Talos runs at a lag, sometimes with a build or two in queue as they struggle to catch up. This should go away with some gaps during the day allowing the Talos boxes to maintain parity with the main build machines.

Of notable difference here, when a patch gets submitted to the Try Server, it runs through the full gamut of unittests and at the end of that, runs a “make package” on the objdir producing a light(ish) build and putting it into the staging area for consumption. At that point, the talos try server can pick it up and run it through the full performance tests for analysis. While that’s going on, people can download the try build and play with it to see if they’ve broken anything. That’s step one in the testing lifecycle of a patch. This should be reality very soon, thanks in part to some help from the good people at Seneca College.

But we can still do better.

In this future, utopian landscape, the unittest and build machines can pick up a patch and start churning and testing builds before handing them over to Talos. At a specified time of day, the nightly machines which have been waiting on the results of the test boxes can now pick up and run the day’s patches and produce a proper build devoid of any extraneous testing code and optimized for userland. This build then gets picked up by the Talos servers and run through the performance tests again. If and only if all of these testing stages complete, the build can be pushed to the update servers for wider dispersion. This should severely limit the number of bad builds we’re able to ship to the world.

Ok, I admit that these pictures aren’t wildly-different from one another. The differences are in the connections and how we use them. By adding some dependency on the testing machines, we can ensure that we’re building the most robust nightlies possible. There are a few caveats to get to this awesome future. The Windows test machines must become more solid. We’ve been struggling with these for almost a year now and the time has come to do something about it. Better displays for gathering at-a-glance tree information are becoming increasingly important as we cram more information onto the main Firefox Tinderbox page. This is going to increase over the year as the JS testing machines come online, possibly requiring a separate tests only page. As more tests are added to Talos, it will become harder to keep on top of the results as builds come in, so this will require some rapidly-accessible view onto that data.

It’s 2008. Do you believe in the future?

While I’m on the subject, we’ve gone through a number of attempts to repair qm-xserve01. The most recent and most drastic measure was a full reinstall of the OS and build tools. The machine’s still running slowly and we’re looking at getting the hardware looked at. In the meantime, please use bm-xserve07 on the firefox page. Another big thank you to Nick Thomas (cf) and John O’Duinn for donating that. I swear we’ll give it back Real Soon Now!

Last but definitely not least, Ben Hearsum has signed-on as a permanent resident and is rocking the buildbot talos stuff. He’s currently embroiled in the setup and installation of the talos linux and mac ports as well as bringing up some new Windows hardware. He’s also recently put some finishing touches on the buildbot try server which should help take some of the orangeness off of the main unittest machines. A heart-felt welcome and thank you goes out to him too.

As proof of all the awesomeness that’s going on, talos is about to begin reporting on the main Firefox tree for your amusement. Expect a few small fires along the way.

So, what am I going to do now that I have all this excellent help? Am I going to take a bunch of long vacations and begin a series of complex, time-consuming hobbies? Oh, dear me, no… I’ll be helping Bob Clary get his JavaScript testing frameworks automated under (you guessed it) buildbot. This is a long-overdue task that’s been taunting me from my to-do list since I started here.

Ted M. noted that 3 of the four failing reftests appear to be kerning differences. The fourth, “(actually the first in the list) looks like it’s off by one pixel in size or so”.

The mochitests and chrome tests I haven’t looked at closely, but they appear to be a mixture of timeout, textarea and focus bugs.

If you want to take a look, grep the full log for instances of ERROR FAIL. Your help in getting qm-centos5-01 to turn green will be greatly-appreciated.

Also, an apology to Paul Reed: Apparently twm is the fallback window manager after all other options have been exhausted. I deleted the standard build user and was left with a partially-configured user. With his help I was able to turn on gdm. IOU five bucks, Paul!

Don’t fear him. He’s here to help.

This adds an extra bit of width to the Firefox tinderbox tree. Over the next week or so, I’m going to monitor qm-centos5-01’s progress and eventually, we’ll probably pull qm-rhel02 from the page, putting it to work elsewhere.

Also, props to Paul Reed for choosing the coolest window manager 1993 had to offer!

There has been an awful lot of orange on the Windows 2003 and OS X unit test boxes lately. Mostly due to shut-down errors. If you see one of these boxes turn orange after a checkin, but no obvious failures in the suites (pass/fail/todo in mochitest and reftest), it’s likely because of one of these exit failures which you can see at the end of the log.

On Windows 2003, the failure looks like,

FAIL Exited with code 1280 during test run

It turns out that Error code 1280 is specific to Windows 2003 and happens when a thread is converted to a fiber one too many times. To help track down the cause of the error, I’m working with Ted Mielc… Ted Milcza… Milk… luser on IRC and setting up an Airbreak/Breakpad/crash reporter processor to try and capture and debug this error. Expect some downtime on qm-win2k3-01 while I’m setting this up.

PS, Apologies to the always-charming Ted Mielczarek whose name I had to look up for proper spelling.

PPS, kudos to jwatt for landing a set of SVG tests into the reftest suite this morning!

But what does this mean to you, Dear Developer? I’m glad you asked. The new machine is now cycling Windows builds and test runs in about 20 minutes — approximately 3 times faster than its Windows XP VM counterpart, WINNT 5.1 qm-winxp01 dep unit test. We’ll leave them both running for awhile before eventually moving qm-winxp01 to lighter duty elsewhere.

Also, the build environment on qm-win2k3-01 (can I just call it Robby?) is updated with the latest version of mozilla-build and contains Zero Cygwin. This should keep this machine more compatible with future changes to the build system on Windows and cut down on excess kruft.

I’ll try to keep the other machines running as much as possible during the update, but there may be a few hiccups. I’ll be monitoring #developers on irc.mozilla.org just in case something goes Horribly Horribly Wrong.