Update on Systems Issues: Monday Mornings Are No Fun

July 19, 2010

This post outlines two systems issues we encountered last night.


At 12:01 AM EDT this morning we performed scheduled maintenance. The work involved returning to our primary data centre, after having switched to our secondary data centre earlier in the month. We did the work on time, but it did not go well. There were complications and we elected to retreat to our secondary data centre.

Problem #1

Retreating to our secondary data centre was the right decision, but there was a cost: any data entered over that one hour and forty five minute period not available in your account now.

Technically the data is not lost, but we can’t automatically restore it either.  Our recommendation is to re-enter the data if you can.  If you need help recollecting what the data is, you can call us and we can assist you on a case by case basis.

We are contacting everyone who logged in over this time to let them know what happened.  Fortunately, this period (Sunday at 12 AM EDT- 2 AM EDT) is the about lowest period of system usage and that is why we chose to do this work then. But clearly, we failed on many levels while attempting to move back to our primary data centre.

We are incredibly sorry for this. If you were affected, please get in touch with us and we will do our best to make it up to you.  If you do not receive an email, and you were not working last night from 12:01 AM EDT- 3 AM EDT, you and your account will not have been impacted by Problem #1.

Problem #2

Before we retreated to our secondary data centre in Virginia, we had managed to bring up our primary data centre in Dallas. When we retreated, some systems continued to run and replicate (this is a good thing) in Dallas. One of these systems was a cron that manages our recurring billing process. So at 5 AM EDT this morning, we discovered that for some FreshBooks accounts (see below), we ran recurring billing twice meaning a handful of our customers charged their customers twice, which was entirely our fault. Here are the specifics on the duplicate transactions:

73 FreshBooks accounts
214 transactions
$14,940 charged

On behalf of our customers, we are in the process of voiding all duplicate transactions. We have already been in touch with the payment gateways responsible for running these transactions, so by the end of today, our customers and their customers’ credit cards should not be impacted. That said, issues and customer confusion throughout today is a very real possibility and we are very sorry to have created it.

There is some good news in all this. The good news is, the number of accounts impacted was relatively low. There are two reasons why the number of accounts was low: first, not all the many people charge cards on the 19th of the month. Far more do on the 15th and the 1st and doing the maintenance on the 19th was intentional.

Second, we have re-architected our recurring billing process in recent months, but we have been gradually rolling it out to ensure we do not create errors – obviously this is critical and complicated system that is sensitive to errors. Ironically, our new system is more intelligent than the one that has worked dutifully for the last 7 years and it does not permit repeat transactions. So, if everyone were running on the new architecture, this would not have been an issue at all.

Where to from here

You will receive an email restating this information if you were affected. That said, our mission is to help you get paid painlessly, so you can focus on what you love to do. Losing work you may have done during this period does not help you focus, or get paid painlessly. It is likely to drain love out of the work you do, and certainly your enjoyment of using FreshBooks.

We are gravely sorry for any inconvenience this may have caused and welcome the opportunity to make it up to you. Please contact us if we can provide any further assistance.

about the author

Co-Founder & CEO, FreshBooks Mike is the co-founder and CEO of FreshBooks, the world’s #1 cloud accounting software for self-employed professionals. Built in 2003 after he accidentally saved over an invoice, Mike spent 3.5 years growing FreshBooks from his parents’ basement. Since then, over 10 million people have used FreshBooks to save time billing, and collect billions of dollars. A lover of the outdoors, Mike has been bitten so many times it’s rumored he’s the first human to have developed immunity to mosquitoes.