Unexpected outage this morning

At 9:32 AM EDT this morning we encountered an issue on a database server in our Dallas datacentre which made the FreshBooks application unavailable. We switched over to a standby database server at 9:37 AM, but the root cause of the issue was in a storage node used by both database servers so the problem continued.

At 9:40 AM we chose to switch over to our secondary datacentre in Virginia. That move was completed at 9:45 AM at which point service was restored. In total FreshBooks was unavailable for 13 minutes.

We’ll continue to use our Virginia datacentre until we’re confident that the storage problem in Dallas is resolved, at which point we’ll schedule the switch back outside of FreshBooks’ busiest times. We already have plans underway to provide dedicated direct-attached storage for each database server to prevent this sort of failure.

We put a high value on always being there for you, and are truly sorry for the inconvenience. If this outage affected you, please contact us and we will try to make it up to you.

(For those keeping score: in the last few weeks the Toronto area we’ve had an earthquake, a tornado, the G20, a record-setting heatwave, office air-conditioning failing during said heatwave, a citywide power outage, a two-alarm fire in the subway that connects FreshBooks’ neighbourhood with downtown, and today’s FreshBooks outage. How’s everyone’s July going?)