Archive for February, 2007
qm-winxp01 dep unit test going down for RAM upgrade today February 22nd, 2007
The unit test box (with the name you can tap your foot to) is going offline for a RAM and processor upgrade today. We’re hoping to decrease build times on it for even more rapid test cycling.
Posted in Build, Quality, Testing | Comments (0)
The System Works February 21st, 2007
This afternoon we closed the tree while sorting out a mochitest failure. It was hard to tell if the crash was real initially, as there had been some tweaking of the tinderbox output from the testbots, and some changes to the harness itself. After some investigation it was deemed to be a real bug and the tree was closed.
Turn-around time from identification of the bug to tree closure to fix was about 90 minutes. Everyone in #developers remained calm and there was no name-calling or crying. While it’s likely that this bug would have been discovered eventually, the fact that it was caught when it was checked in and immediately corrected is not a small point. It may have saved everyone hours of detective-work and struggling with CVS. Instead, it was a minor event.
In other news, the reporting format of the testbots has changed to a more compact layout. They’re now reporting as number of passes and fails on a single line, separated by a slash. Mochitests are reporting as passes/fails/todos.
Additionally, another harness is coming online and reporting as “chrome”. This will be used for testing XUL and browser chrome. It should be available on all platforms by the end of the week.
Posted in Quality, Testing | Comments (3)
I’m sure they were here a minute ago… February 19th, 2007
The testfarm buildbots have been moved to the main Firefox tree on Tinderbox and given spiffy new names. They are now known as “Linux qm-rhel02 dep unit test”, “MacOSX Darwin 8.8.4 qm-xserve01 dep unit test” and “WINNT 5.1 qm-winxp01 dep unit test”. Sure rolls off the tongue, doesn’t it?
We haven’t really nailed-down what it means to have these here yet, and as of this writing, the logging summaries aren’t being printed out so the information’s only semi-useful at-a-glance. We’ll get that fixed quickly though. Ultimately, I think failures on these boxes should be taken seriously and if you check in and see them turn orange, you should take a close look at the logs to see what happened.
Also, if you see any problems with these machines, please file a bug on it in core/testing. I’ll be watching that bucket with great vigilance.
Posted in Build, Firefox, Programming, Quality, Testing | Comments (2)
buildbot testfarm: 1 week later February 16th, 2007
One week later and the OS X test box turns green just in time for the long weekend. It didn’t happen by magic. Rob Sayre graciously donated some of his busy schedule to help track down an error in the configuration files. If I weren’t so happy it was fixed, I’d be feeling a little bit of shame right now. I’m sure it’ll set in by the weekend.
What does this picture mean? If you take a walk over to the MozillaTest tree, you’ll see the three boxes on the end, linref test, osx test and winxp test. Each is reporting values for TUnit (aka “make check”), reftest and mochitest.
TUnit is the least consistent reporting of the three test suites and as a result, you’ll see the funny “Suites” numbers. On linref and winxp, this is around 700 and refers to the number of directories make is descending into looking for tests. It’s only reporting 30-40 passes, but that doesn’t include the 181 tests in the feeds component. Or the correct number of “TestCookie” tests. The real number is probably closer to 230 tests passing in this section.
reftest is running and passing somewhere between 215 and 230 tests on each platform. These are testing layout and rendering and now that we have means to mark tests as random on particular platforms, this number should grow.
mochitest is now reporting a very respectable 1600 tests passing on all platforms. This is huge and I also expect to see more tests landing in that framework really soon.
So, not to put too fine a point on it, but we’re now running somewhere around 2000 tests on every checkin to the repository. If you want to see if you’ve broken something, feel free to take a take a look at these boxes after you’ve checked something in.
Posted in Build, Firefox, Quality, Testing | Comments (3)
it’s the little things February 7th, 2007
That little page was my first view of the new testing buildbots living in the Mozilla colocation facility. It may not look like much yet, but it’s overflowing with potential. What does it do, you wonder? It runs test-enabled builds, then runs make check, reftest and mochitest. From the three columns, you might infer that it does this for the three major platforms, Windows XP, Linux and Mac OS X — or at least it will when the Xserve arrives at the colo.
You can see the results reporting to the MozillaTest tree on tinderbox.mozilla.org. Currently, the two operational buildbots are “linref test” and “winxp test” (don’t worry, I have an exciting naming scheme all picked-out and ready for action – stay tuned!). Check out the logs for examples of what the make check, reftest and mochitest reports look like. If winxp is orange or burning, it’s because we still have a couple of issues to work out. Some of the reftests are failing on that platform and dbaron is writing a modification to the manifest language so we can mark individual tests as failures on specific platforms. The other, more troubling bug is a debugger prompt I can’t seem to get rid of. Neither setting XPCOM_DEBUG_BREAK to “warn” nor adding a registry entry for windbgdlg (thanks timeless!) seemed to work. I’m hoping the problem will just go away with the addition of more RAM, but that seems unlikely.
On my todo list, I have some work to do to get the different tests reporting rolled-up stats to the tinderbox. We’ll be able to see why a tree is orange instead of looking through the full logs. There are a couple of little profile writing things I can make better. At some point, soon, I’m going to work with Bob Clary on getting the JS tests automated and reporting under this system. Then there’s going to be some work getting Alice Nodelman’s performance testing framework running under it and reporting to the tinderboxes. I’m also trying to setup a leak-testing build to replace the Tinderbox that just stopped working and weaseled its way to the front of my to-do list. Ultimately, I’m hoping we can have these on a more visible tinderbox, but they need to be stabilized first. Oh, and I need to check these configuration changes in and flesh out the very skimpy docs I wrote while putting this together.
Whew, this blog post would’ve been much shorter if I’d just written it last week when I had that one little picture.
Hopefully, having regularly running and reporting tests will encourage developers to write more of them. Individually, unit tests are tiny and kind of uninteresting. When you’ve got thousands of them running constantly though, then you’ve got something.
PS, special thanks to everybody who helped out along the way to get this stuff running. To the IT crew (mrz, justdave, aravind, justin, …) for putting up with the constant bug-filing back-and-forth for VMs I needed or VMs I’d busted, the tireless efforts of the build team (Rob Helmer and J. Paul Reed) taking time away from their real jobs to setup the linux reference VM, Rob Sayre and Dave Baron for their excellent test frameworks, Jeff Walden for the js http server, Ben Hearsum for his buildbot work and Dave L. for getting the ball rolling many months ago. All the developers who wrote tests… I’m being played off the stage… thanks Mom!
Technorati Tags: Build, Computing, Firefox, Software
Posted in Build, Code, Firefox, Quality, Testing | Comments (1)