A lot’s changed since those dark days, however. To say that we spend a lot of energy doing performance analysis is a bit of an understatement. We have entire teams dedicated to improving performance. Build options are analyzed and re-analyzed in an attempt to find cheap performance improvements. Specifying global code optimization settings can actually have an adverse effect on overall system performance because certain modules react better to, say, -O2 versus -O3, for example. All code that gets checked in has to make it through our performance tracking system or it gets backed out.

In the interest of Science, I wanted to actually see if there were any differences, so I pulled a fresh copy of mozilla-central and applied the referenced .mozconfig.

The first thing I noticed, was that –enable-profile-guided-optimization didn’t seem to be doing what I thought it would. Namely, I expected to see a browser window popup and run through some automation before starting a second compilation phase. It didn’t. Even so, the resultant build did feel a little faster at opening windows and tabs*.

So, I applied the latest stand-alone version of Talos to the build and executed the Ts (startup) and Twinopen (Window open) tests and compared this with the most-recent Mozilla nightly build of Minefield. Surprisingly, the windows didn’t appear to be opening much faster, but the startup time was on average 5% faster in the new “optimized” build vs. moz-central nightlies.

By this time, Ted had commented in the bug tracking all this and pointed me to the PGO page on MDC. It turns out, PGO builds require a separate make target “profiledbuild” instead of “build”. I reran the build, saw the automation was working this time and then reran the Talos tests. No change. Best guess around these parts is that GCC just isn’t very good at doing PGO. The same option gives us an across-the-board 10% speedup to all our tests on Windows.

It turned out that the option that was giving us the 5% startup time (and likely a slight boost to tab performance) was the 10.6 SDK target. Each revision of OS X includes optimizations and Snow Leopard is no different. If you want your own optimized build, just put that target (–with-macos-sdk=/Developer/SDKs/MacOSX10.6.sdk) in there if you’re on Snow Leopard and you’ll get a little extra snappy.

I haven’t booted into the 64 bit kernel to see if there’s any difference there, or even if the build options produce a 64 bit build. I don’t believe they do and the version shipped on the above optimized builds page don’t have the tell-tale “Open in 32 bit mode” checkbox in the app’s “Get Info” dialog.

* A word about perceived performance differences: It’s very hard to tell the difference between, say, 20ms and 30ms. That’s a 50% slow-down compared to 20ms but the actual slices of time are so small that the human brain has a hard time picking up on them. Nevertheless, people used to dealing with high-velocity timings (musicians, gamers, fighter pilots) can actually detect that kind of change. But there’s also a “wishful thinking” factor. It’s very easy to be told you’re going to see a speedup, and when given an identical application, think you’re seeing something faster. “It just feels snappier” you might say when given this placebo.

First, a little terminology. Feel free to skip this paragraph if you consider yourself a master of Javascript internals or compiler run-time optimization makes your skin crawl. Tracing is the mechanism Firefox’ Javascript engine (aka, “SpiderMonkey”) uses to improve code execution performance. It provides a major speedup for code running in Firefox 3.5 and up, often an order of magnitude for certain types of operations. It is the basis of the JIT or Just-in-Time compiler. Without tracing, the JS engine can’t optimize code as well, leading to significantly slower execution.

I need to be clear here: If you have Firebug installed you are probably not getting fast Javascript. Firebug doesn’t have to be active on your current page. If you have the grey icon on your status bar, you have probably disabled the JIT. This is true if you have ever enabled the Console and consequently the Script panels and left them on. This is likely true for most recent versions of Firebug. The quick fix is to disable the Script and Console panels via the mini menu on their respective tabs.

Boris Zbarsky has been doing a lot of poking around in the belly of the JS debugger lately. Working in conjunction with John Barton, they have concocted fixes for the soon-to-be-released Firebug 1.5 and Firefox 3.6. We’re working to get testable versions out as quickly as possible, but I can say that I’ve tested development builds using the pages mentioned in the bug and can verify that these fix the problem. As a side-benefit, I got to watch Boris dissect the problem through gdb in an Xterm (yes, an Xterm) and Emacs and it was pretty impressive and fun, to boot.

When installing Firebug one of the first things most people do is enable the Console. It’s such a useful debugging tool. I myself have been using Firebug for over a year without JITted code. I probably never noticed the slowdown simply because I’m so used to running my browser this way and have pretty fast machines. After disabling the Console and Script panels there are a few pages that just load much much quicker. It’s kind of shocking.

Since the patch has landed on Firefox 3.6, I’m expecting either a beta or release candidate of that to come out shortly. I’m also hoping for another Firebug beta (or release candidate) later this week. In the meantime, if you’re not using them, I recommend disabling the Console and Script panels and find out what you’ve been missing and turn them on only as-needed. This will likely be the work-around for Firefox 3.5 users unless we decide to push this back to that branch but I have my doubts that this is something we would do.

I can say without hyperbole that this was the best conference I’ve attended. The presentations were from people doing real large-scale web development and included a fair amount of real data and some of their solutions to hard problems.

One of the highlights was John Adams of Twitter talking about the engineering challenges they’ve encountered keeping your tweets flowing. They’ve rewritten intensive chunks of code in C. They’ve implemented their own queue system (called Kestrel which has an API very similar to memcached). They use Google Analytics to monitor the frequency of the Fail Whale. You can see the whole presentation here, on video.

Other speakers I enjoyed seeing included Marissa Mayer of Google, Nicholas Zakas of Yahoo on JavaScript performance tuning and the mighty Steve Souders (pictured above) talking about Even Faster Websites. I liked his presentation so much, I bought the book!

Some of the take-aways from the conference were: Instrument Everything. Analyze your performance constantly. Employ phased roll-outs and measure user reactions to changes.

I was surprised to learn that changes in website responsiveness sometimes as little as 50ms could incur changes to pageviews or searches, directly impacting revenue. Anyone whose ever played Rockband or, gosh, a real guitar or set of drums can tell you that 50ms is very noticeable and at the outer-limit of responsiveness (not to mention can make your music sound imprecise). Humans are capable of detecting latency well below that number, but the surprising thing is that we humans are now expecting this level of performance from our websites.

You can see a simple visual demonstration of this with Ben Galbraiths’ feedback demo, presented by Ben and Dion Almaer in their On Responsiveness session.

I don’t think it’s something we would consciously think about when using a site, though it would likely form some sort of nebulous impression upon us. Using a search engine that didn’t post any response for 50ms or worse would probably make us think, “this thing’s slow” or “I’m not getting any results”. Those delays add up over time. As users know there’s going to be a delay, they might start a search and flip to another application or browser tab. As soon as that happens, the user’s attention is lost and may not return.

What I find really interesting about this, is that people are now treating websites with the same levels of performance expectations that they have with video games and desktop applications. Website performance is more important than ever, so if you’re creating or maintaining a website, keep that in mind.

Too much content, not enough blog. I’ll try to talk about some of the other cool things I saw later this week.

What became pretty clear was that there was only a very negligible performance hit for having Firebug installed. Better yet, it’s quite performant even with the Firebug console enabled, meaning you can keep Firebug on your default profile without worrying about impacting your regular browsing.

Memory performance was similar and even with the script debugger enabled, memory usage peaked and levelled off indicating that there were no real leaks visible, even though pages took noticeably longer to load.

This isn’t to say that we’re sitting idly on our hands congratulating everyone on a job well-done. John Barton noticed a particular JavaScript test that was noticeably slower than it’s neighbors and it jogged his memory about some testing he’d done earlier in the 1.2 development cycle. This should translate into improved eval script performance in 1.3 and future versions of Firebug.

I’m looking forward to revisiting these benchmarks down the line to track our improvements.