A Bug Story: Debugger Breakpoints and Page Reloads in Firefox
What you need to know: If you set a breakpoint in the Debugger in Firefox 15 and then hit “Reload”, the Debugger is going to shutdown. This has been fixed in Firefox 16.
And now the story…
A couple of weeks ago, Dave Herman (aka @littlecalculist) CC’d me on a tweet. That led to this tweet:
@littlecalculist That was an earnest question, not snark. I couldn’t get debugger statements or breakpoints to work.
— Tom Dale (@tomdale) August 16, 2012
Initially, I didn’t know what was going on. I’ve been using the debugger daily but not in any deep way. I’ve seen breakpoints and debug statements work regularly. After a couple more exchanges, Tom pointed us to a video of his situation which was setting a breakpoint early in his script and then hitting reload, expecting the breakpoint to hit during the parsing phase.
This too I know I’d seen work, so there was definitely something wrong so I filed bug 783393. Panos wasn’t around so I asked Dave (Camp) what he thought and he poked around at it for a bit. He determined that we had a nasty race condition in our Debugger reattach mechanics. Dave still knows the Debug Protocol code as well as anyone (despite his protests) and he agreed to hack up a patch for us.
By now it’s late Thursday, August 16th. Twelve days before the merge.
What’s a merge you ask? That’s where we take our Nightly browser and push it up to Aurora. Aurora becomes Beta and Beta becomes the new Firefox. Yes, that’s right, we have 4 branches we maintain and shuttle through our release processes. Who takes care of each of these channels? Our release managers!
By Monday, dcamp had a decent patch to deal with the race condition. It required rewriting the way the Debugger detaches and reattaches to content on navigation. It had gotten a little complex, but it was something. Panos was back from the beach and we brought him up to speed on the story and handed the patch over to him. He’d just gotten back from the beach and seemed very relaxed.
Dave and I decided it would be best if we had a patch to disable the debugger on reloads and navigation in Beta. The likelihood of the full fix making it in was minimal, so I wrote the tiniest patch I could think of to do it. A total hack job. Dave had switched back to getting his new markup panel for the inspector into a shippable state. I could have sworn I heard the sepulchral undertones of Norwegian metal when I spoke to him in IRC.
By Tuesday, Panos was still grinding on the patch, fixing failing tests, rewiring parts of it and dealing with the fallout. The patch was mutating a little but Panos was improving it. The beard he’d grown the previous week on the beach was starting to itch but he clung to it. I think it helped him code. Kept him in a beachy state of mind. I pung Lukas Blakk to tell her what was going on and warn her that we were going to have to try to get something into Beta. We didn’t know what at that point because we weren’t sure we’d be able to port Dave and Panos’ fix that far forward.
I gave an update in the engineering meeting that went something like, “yeah. breakpoints in reload. not so much. working on it…” Afterwards, Lukas, Dave and I went through our freshly-cooked plan to ship the shutdown patch on Beta and the real fix on Nightly and Aurora which should be ready Any Minute Now. Lukas agreed this was a sensible approach. The alternatives were waiting for a patch that might not work on Beta or kill-switching the feature. We felt the Debugger was still useful enough without reload breakpoints working that it was worth shipping and I still believe that’s true. It just might require some additional debugging strategy or a jump to the new Beta or Aurora (where most devs should probably be anyway).
Wednesday. The patch was not ready yet. I started wondering if we should kill reloads on Aurora as well. I was getting a little scared, but Panos’ beard gave me hope. Then he shaved it off.
Thursday and we had our tests passing. The patch was ready, but not quite perfect. I encouraged us to move ahead with an imperfect patch. I reviewed a couple of things Victor had been working on. Little things like full script search and some collapsible panels in the debugger. Nice things he’d been patiently waiting to land.
Friday the patch was ready. Panos landed it amidst a tree of fire and watched the tests pass. It was glorious. I conferred with Lukas one last time to get preapproval for Aurora. I’d be landing it on Saturday, but that was cool. Stuff was landing. The devtools team was pushing features into the fx-team repository at a shocking pace. It was like that scene from Mission Control when the rover touched down and everybody including mohawk guy were awkwardly high-fiving and cheering. Yes it was that good.
On Saturday, Victor and I hung out and rebased Panos’ patch for Aurora. Fixed some tests. Made it work. Victor did most of the work. He’s good that way.
What’s really impressive to me is how this whole thing unfolded from start to finish. An innocuous tweet about one feature from someone outside the project could turn into a flurry of activity and have such an excellent resolution in a very short time-scale. Everybody worked really hard on this thing and if we’d shipped without those breakpoints working on reload, it would’ve undermined people’s confidence in it. I think this extra effort makes the Debugger that much more usable. I’m also really impressed with how Dave, Panos, Victor, Mihai, Lukas and Nick dealt with all of this. It’s a huge privilege to work with such amazing and talented people.
Also, thanks to Tom Dale for reaching out to us and asking a question. This couldn’t have happened without him and Dave Herman pointing us to it.