Issues with this morning’s scheduled maintenance

by Rich Lafferty - July 19/2010

Earlier tonight we performed scheduled maintenance on FreshBooks, during which we moved from our secondary datacentre in Virginia back to our primary datacentre in Dallas. This followed an emergency switch to Virginia earlier this month when our Dallas database servers encountered problems with their storage hardware.

Unfortunately, the move back to Dallas tonight was unsuccessful, and we’ve had to switch back to Virginia and restore our databases to the last snapshot prior to tonight’s maintenance. As a result, any changes made in FreshBooks between 12:10 AM Eastern time (4:10 AM UTC) and 2:00 AM Eastern (6:00 AM UTC) have been lost, and you will need to re-enter that data.

We’re aghast that this situation has occurred and that the affected time period is as long as it is, and will be making changes to our processes immediately to prevent the problems that led to tonight’s failed cutover. If you are affected by this situation and need assistance in re-entering your changes, please don’t hesitate to give us a call.

Jul 19/10
7:49 am

Erik says:

I’m sorry you guys had trouble–these things happen. Were you planning on pushing a notification out to customers via email? Or just hoping that folks saw this blog post or the Tweet?

Jul 19/10
10:31 am

Mike McDerment says:

Hey Erik – we’re still wrestling some things down.

Things are stable now and we are just getting in touch with the account holders who were directly impacted by the outage (email, phone, etc), before communicating more broadly.

Stay tuned for more and thanks for your comment.

Jul 19/10
11:51 am

Sometimes being a pro is all you’ve got says:

[...] move was planned in advance. It was communicated to our customers. It was run on schedule. And it still went wrong. The best part? The retreat to our secondary data [...]

Jul 23/10
1:27 pm

Scheduled maintenance window this Sunday night says:

[...] Issues with this morning’s scheduled maintenance [...]