Doing More, Doing Better: QA at FreshBooks
November 25, 2010
This chart makes me very happy and thankful for one of our teams. At FreshBooks, we track all sorts of statistics, release by release, and this chart compares two important numbers — Velocity and Bugs. Velocity is the amount of work that gets done in each release (every two weeks). Bugs is the number of issues introduced into production, per release. These issues are dealt with as a quick “hotfix” right after the release. To measure our success over time, we keep a record of every mistake we make so we know how well we’re doing (the goal is zero Bugs).
What’s the trend?
You can see that there is a pretty clear correlation between the two — every time Velocity goes up, so does Bugs. That’s normal. But the great thing about tracking numbers like this over time is that it allows you to observe trends in your own process. What we see here is fantastic — while we’ve been steadily increasing Velocity over the past couple of years, you can see that the average number of Bugs is steadily DECREASING.
We’re doing more in each release, but we’re making fewer and fewer mistakes.
What’s the secret?
Meet the FreshBooks Quality Assurance team (QA). These folks work closely with the development team. They test new features, run regressions every release, build and maintain a massive automated test suite that exercises the FreshBooks web UI, and even copy-edit all the text that appears in the product. Their job is to make sure we can all be confident in FreshBooks.
What about developer testing?
Now, our developers are not a careless bunch. They write their own tests for every function they create (“unit tests“), to make sure it does exactly what they expect it to do (and nothing else!). But the biggest problems in software quality are not “unit testing”, they are “integration testing” — what happens when you connect one piece of software to another? A modern web app includes dozens of different bits of software all talking to each other behind the scenes, so integration testing is a critical problem.
How do you solve this critical problem?
This is where our QA Team comes in. They run an environment that exactly duplicates our production environment (actually, they have more than one), along with all the little installed bits and pieces that make a web app like ours work. They investigate every change to see if it does what it is supposed to, and ensure that the change has no unintended impact. They use various automated test suites to affirm that everything which used to work in the past still does. The process it topped off by all the instinct-driven “this is probably going to break” activities that are the hallmark of a great QA team.
Having a team dedicated to finding your mistakes is a necessary part of any serious software group. When you’re building something, you want it to succeed, and so it is very difficult to overcome the mindset of “it probably works”. QA exists to perpetuate the mindset of “it probably doesn’t work.” To that end, our QA team has been steadily building up its capacity to test all the work of the FreshBooks development team. We’ve been hiring and growing the team, which has certainly helped, but the development team has been growing at the same rate (or faster), so the ongoing improvement shown in the chart above is due more to better processes and better automated testing. It’s why we’re so thankful for our QA team.
For more posts like this one, check out the FreshBooks Developers Blog.