Back to Blog Home

Update: Monday mornings are no fun

by Mike McDerment  |  July 19/2010  | 

This post outlines two systems issues we encountered last night.

Introduction

At 12:01 AM EDT this morning we performed scheduled maintenance. The work involved returning to our primary data centre, after having switched to our secondary data centre earlier in the month. We did the work on time, but it did not go well. There were complications and we elected to retreat to our secondary data centre.

Problem #1

Retreating to our secondary data centre was the right decision, but there was a cost: any data entered over that one hour and forty five minute period not available in your account now.

Technically the data is not lost, but we can’t automatically restore it either.  Our recommendation is to re-enter the data if you can.  If you need help recollecting what the data is, you can call us and we can assist you on a case by case basis.

We are contacting everyone who logged in over this time to let them know what happened.  Fortunately, this period (Sunday at 12 AM EDT- 2 AM EDT) is the about lowest period of system usage and that is why we chose to do this work then. But clearly, we failed on many levels while attempting to move back to our primary data centre.

We are incredibly sorry for this. If you were affected, please get in touch with us and we will do our best to make it up to you.  If you do not receive an email, and you were not working last night from 12:01 AM EDT- 3 AM EDT, you and your account will not have been impacted by Problem #1.

Problem #2

Before we retreated to our secondary data centre in Virginia, we had managed to bring up our primary data centre in Dallas. When we retreated, some systems continued to run and replicate (this is a good thing) in Dallas. One of these systems was a cron that manages our recurring billing process. So at 5 AM EDT this morning, we discovered that for some FreshBooks accounts (see below), we ran recurring billing twice meaning a handful of our customers charged their customers twice, which was entirely our fault. Here are the specifics on the duplicate transactions:

73 FreshBooks accounts
214 transactions
$14,940 charged

On behalf of our customers, we are in the process of voiding all duplicate transactions. We have already been in touch with the payment gateways responsible for running these transactions, so by the end of today, our customers and their customers’ credit cards should not be impacted. That said, issues and customer confusion throughout today is a very real possibility and we are very sorry to have created it.

There is some good news in all this. The good news is, the number of accounts impacted was relatively low. There are two reasons why the number of accounts was low: first, not all the many people charge cards on the 19th of the month. Far more do on the 15th and the 1st and doing the maintenance on the 19th was intentional.

Second, we have re-architected our recurring billing process in recent months, but we have been gradually rolling it out to ensure we do not create errors – obviously this is critical and complicated system that is sensitive to errors. Ironically, our new system is more intelligent than the one that has worked dutifully for the last 7 years and it does not permit repeat transactions. So, if everyone were running on the new architecture, this would not have been an issue at all.

Where to from here

You will receive an email restating this information if you were affected. That said, our mission is to help you get paid painlessly, so you can focus on what you love to do. Losing work you may have done during this period does not help you focus, or get paid painlessly. It is likely to drain love out of the work you do, and certainly your enjoyment of using FreshBooks.

We are gravely sorry for any inconvenience this may have caused and welcome the opportunity to make it up to you. Please contact us if we can provide any further assistance.


  • http://www.hrlocker.com John Dennehy

    Every software company in the world has infrastructure or hosting problems from time to time. It’s great to see one that’s actually honest and informative about it. Fair play for your transparency.

  • http://www.oirinteractive.com Justin

    Thanks for you honesty and transparency, Mike. Many business owners could learn a lot from you.

  • http://www.homestars.com Brian

    Considering $1B in transactions that number is amazingly small. We got hit and had to go talk to some customers, who weren’t too happy. But I think we’ve smoothed things over. Guess the recurring profiles aren’t used as much as we do among a larger audience.
    Thanks for keeping us in the loop. The second email detailing the transaction helped us immensly in sorting out our issues in figuring out which customers to talk to.

  • majoreno

    Thanks for you honesty and transparency,


Rodney's 404 Handler Plugin plugged in.