Earlier today the FreshBooks service experienced approximately half an hour of downtime.
At approximately 11:37 AM Eastern the FreshBooks website and application were not available. We are not happy to report this and before I explain what happened, I would like to apologize to anyone that was affected.
Today’s downtime was the result of human error configuring routers at Rackspace while they were moving our old database servers to the new cabinet, compounded by miscommunication between our Rackspace account manager and their network technicians and problems with Rackspace’s phone system.
Timeline of events (EDT):
Rackspace started move: 11:26
Website unavailable: 11:37
Workaround in place: 12:05
Root cause resolved: 13:00 approx
(plus about a minute at around 12:45 to redo the workaround).
The workaround was an SSH tunnel from our database servers to our web servers on one of the vhost addresses that was still working. (No traffic could reach our database servers, but some traffic could reach some addresses on our webservers from our database servers).
In future, we’ll be scheduling changes in which Rackspace has the ability to cause collateral damage after hours, even when those changes don’t affect production systems.
Again, we are sorry for the inconveniences this caused, and if you were adversely affected by the downtime, we invite you to please send us a note or give us a call so that we can hear from you and look into making it up to you.
Thank you for your understanding.
For future reference, we have a status blog at http://status.freshbooks.com which gives real-time updates of the FreshBooks service.
Copyright © 2000-2007
7:55 pm
thanks so much for the transparency and refreshing openness.
that’s a big reason as to why i stick with freshbooks. keep up the good work.
11:58 am
agreed! Thanks for explaining what why how and and where it went wrong. Definitely a fresh approach to accountability towards your customers. That’s why you guys’ services kick ass, keep up the good work.
7:19 pm
Yes, Thank you. I hate dealing with providers who just give the run around and are NOT accountable.
Thanks again for the openness!!
3:55 pm
Thanks for the info and honesty! I was planning to move all of my commercial websites to Rack Space due to their “100% Network Up Time Guarantee.” Please let me know if you recommend them and how often you have had such issues.
4:27 pm
Hi Jan,
We definitely would recommend Rackspace. We have been using them since 2004 and have been very impressed with the level of service they provide. We have had exactly three issues with Rackspace worth mentioning over the last four years: the first was more hardware related, the second was their datacenter in Dallas having power problems, and the third was this post above from last week.
If you want more details, then send me a note at levi at freshbooks and I will be happy to share a story or two.
- Levi
4:23 pm
Definitely a fresh approach to accountability
4:25 pm
sohbet ve sohbet odalarına bakınca definitely oluyor herşey sohbet