The Release Process at FreshBooks: A Virtuous Circle
January 6, 2010
Every second Wednesday here at FreshBooks is Release Day, the day we release our latest efforts into production for our beloved FreshBooks users. We like to make frequent releases so that our users feel the momentum of the changes we make. But there’s another important reason for a regular release schedule: process improvement — the pressure to keep deploying compels us to improve our deploy process.
Thanks in part to a “virtuous circle” (note: opposite of a “vicious circle”) we’ve developed, new releases at FreshBooks are usually uneventful. Behind the scenes we work hard to ensure the lessons learned from the last release are passed along to future deploys. The key to this circle is a pair of meetings that work together to keep us organized, I want to take the rest of this post to tell you more these meetings and how they impact our process.
Every second Monday – two days before each release – we hold a Release Planning meeting. This meeting never takes more than half an hour (and is usually quite a bit less). Basically, it’s just a run through of an existing checklist. But checklists are whoa, man, boring. I remember working at a place where EVERY PROBLEM was handled by adding a new line item to THE CHECKLIST. As you can imagine, that checklist got awfully long (it was three pages!). If all you’re doing is running through items in a checklist, you’re going to sleepwalk through the meeting.
So we broke the planning meeting into three parts:
Done Yet?: There are a number of things that ought to be done prior to the Release Planning meeting — making sure the Support team has a liaison to manage communications during the release, writing up a set of deployment instructions for the IT group, and so on. We breeze through these at the meeting because the work is usually done. It’s really just status check in.
Key Assignments: We assign people to the process. Which developer will be responsible for cutting the release branch and staging it for deploy? Who has to be present during the release (in case of catastrophe)? At the meeting we agree on these items, confirm availability of resources and write down the commitments.
Reminders: In addition, there’s always a bunch of critical things that we run through to make sure whoever is responsible knows they’re responsible. This is the “checklist” part of the meeting and it’s brief. It’s also not entirely serious — the last few items on the list belong to me and include “Savour SUPREME EXECUTIVE POWER” and “Acquire cake FOR ALL”.
[NOTE: take a moment and appreciate those last all-caps. They got added to the check-list after a release day on which I managed to scrounge up a slice of delicious cake for myself, but there wasn’t any for the people actually doing the work. Since my item said only “Acquire cake”, I was able to argue (tongue in cheek) that I’d accomplished my assigned task, when really it had just slipped through the cracks. Folks took exception and the issue came up at the Retrospective (more on THAT directly below)].
Which reminds me that a few bits of silliness here and there go a LONG way to keeping any meeting or process less painful for everyone involved. Everyone perks up when they get a chance to chuckle.
That’s the current state our Release Planning process — although it’s always evolving, based on what we learn. The other side of our virtuous circle is the another meeting: the Release Retrospective meeting.
On the day after each release (Thursday), everyone involved comes to a Release Retrospective. This meeting is run with sticky notes — in two colours. One colour is for Things That Were Good, and another colour is for Things That Were Not Really So Good. We spend ten or fifteen minutes just writing stuff down on these sticky notes and putting them up on the wall. No discussion, everyone with sticky notes just jotting down observations and posting them up for everyone to see.
This style of idea generation accomplishes two things. First, it takes away any sense of blame or finger-pointing. We don’t write, “Corey forgot to order the celebratory cake,” – we write, “There was no cake.” That makes it easier for me to voluntarily take ownership of the failure, since I can’t deny that there was no cake. No energy is wasted trying to defend oneself because oneself is never under attack. Second, it keeps us from dwelling on any single item just yet – we don’t argue about whether or not someone’s suggestion is worthy or silly (indeed, a good number of silly suggestions get put up on the wall each time), we just take the written note and put it up on the wall for everyone to see.
Once the flurry of suggestions and ideas has settled down, we go through each and every one and assign it to someone present.
We never just shrug and say, “Well, that’s just how things are.” If something’s not working right, we have to take action to make it better — which means somebody has to be responsible for making it happen. If a good thing happened by accident, we need to find a way to make that becomes part of our process. People volunteer to take responsibility for things until there aren’t any more. Often, of course, multiple notes really point to a single underlying cause, and so the task of assigning notes to team members isn’t always as daunting as it looks.
Things only change when someone is made responsible for creating that change. We make sure every change item has an owner, so that we can follow up and make sure it gets done.
The Virtuous Circle
Each meeting is about assigning owners to actions. At the Release Planning meeting we assign the deploy work that needs doing, and at the Release Retrospective we assign the “meta-work” that determines the deploy work that needs doing. Each Release Planning meeting incorporates new tactics that emerged from the previous Release Retrospective, and each Release Retrospective we go over the impact of the new Release Planning process. We’re not quite at the point where releases NEVER get a little exciting at one point or another, but it’s very rare for us to get surprised on a Wednesday morning.
And it’s now my duty to acquire cake for everyone on the team, not just myself. Sometimes my processes don’t always work to my advantage, but it’s a sacrifice I’m willing to make. I’m a team player.