~robcee/ more than just sandwiches

buildbot testfarm: 1 week later

One week later and the OS X test box turns green just in time for the long weekend. It didn’t happen by magic. Rob Sayre graciously donated some of his busy schedule to help track down an error in the configuration files. If I weren’t so happy it was fixed, I’d be feeling a little bit of shame right now. I’m sure it’ll set in by the weekend.

What does this picture mean? If you take a walk over to the MozillaTest tree, you’ll see the three boxes on the end, linref test, osx test and winxp test. Each is reporting values for TUnit (aka “make check”), reftest and mochitest.

TUnit is the least consistent reporting of the three test suites and as a result, you’ll see the funny “Suites” numbers. On linref and winxp, this is around 700 and refers to the number of directories make is descending into looking for tests. It’s only reporting 30-40 passes, but that doesn’t include the 181 tests in the feeds component. Or the correct number of “TestCookie” tests. The real number is probably closer to 230 tests passing in this section.

reftest is running and passing somewhere between 215 and 230 tests on each platform. These are testing layout and rendering and now that we have means to mark tests as random on particular platforms, this number should grow.

mochitest is now reporting a very respectable 1600 tests passing on all platforms. This is huge and I also expect to see more tests landing in that framework really soon.

So, not to put too fine a point on it, but we’re now running somewhere around 2000 tests on every checkin to the repository. If you want to see if you’ve broken something, feel free to take a take a look at these boxes after you’ve checked something in.

Is this going to end up replacing tinderbox, or just supplement it?

“So, not to put too fine a point on it, but we’re now running somewhere around 2000 tests on every checkin to the repository.”

That is so hot.

1. I don’t think so Shawn. Not in the short-term, anyway. Right now we’re using buildbot for a few specialized functions. We’ll be adding more to the test farm in the near future and hope to be able to take some of the hard-coded tests in tinderbox and migrate them to buildbot. I’m also hoping we can have some help doing that.

2. Looks like there were some checkins over the weekend. We’re closer to 2500 tests now.