Comments on: Dallas data centre power restored

By: Ali

Ali — Sun, 25 Nov 2007 02:58:04 +0000

I think you should use hostmysite.com for your data center instead if a data center goes down because a truck ran into their lines that is just not acceptable.

By: Aaron Adams

Aaron Adams — Thu, 15 Nov 2007 20:25:05 +0000

CB: luckily, I have full editorial control of the comments.

By: Eric Marden

Eric Marden — Thu, 15 Nov 2007 19:44:43 +0000

I think you did too, Mike.

@Tony – Remember any relationship can be salvaged with a conversation! Point them to this blog post if they need proof that it was practically an act of god that 404′d your estimate.

By: Mike McDerment

Mike McDerment — Wed, 14 Nov 2007 14:27:50 +0000

Jamen – great question. You may have noticed I mentioned our Virginia Data centre a couple times in the post. That data centre runs on a different network and is available to us to over things off in events like these. The reason we did not switch over in this instance is that we were working on some things there at the time (which is really unfortunate) and it would have taken longer than 1-2 hours to switch data centres on Monday night. Were the outage going to be longer, we would have switched over, all of which is to say, not unlike the team at Rackspace, we were forced to make a decision, and given the circumstances, I think we made the right one. So you know, we also use off site backups to a third location to ensure we aren’t down for a few days.

I hope that makes things clearer.

By: Jamen

Jamen — Wed, 14 Nov 2007 07:57:30 +0000

All thumbs up for coming clean on the down time and what now and as you can see everyone but Tony really was affected by the incident. Coming from a healthcare company we were required as with most financial and healthcare transactional based sites to have disaster recover. What would have happened if the truck had hit the building and the noc burned down? Would we all be posting thanks for letting us know that we will be down for a few days? I am still small so I am sure I would not be too affected, mind you know I will be sure to do be seeing what kind of exporting I can do on my pending/recurring invoices in the event something worse does happen. I plan to setup redundancy as soon as possible, what does freshbooks plan to do in preparation for disaster recovery in the near future? Again, what would we be doing right now if the truck hit the noc? Thanks and I too am a happy customer but would I be if it had been worse?

By: cb tells tales

cb tells tales — Wed, 14 Nov 2007 04:17:30 +0000

I should have known that you were the one who deleted my exposé. I suppose I should have known better than to post the name of one of your competitors, but those dastardly fiends should be exposed for the saboteurs that they are.

You’re lucky that all of us here in CB world think you’re A OK, Aaron, otherwise I might have been forced to post a negative comment about you.

By: Mike McDerment

Mike McDerment — Wed, 14 Nov 2007 01:47:24 +0000

Tony – we are all disappointed with what happened and I want to thank you for sounding off and representing an important constituency (FreshBooks users in who were affected in the heart of their business hours). I have sent you an email.

If there is anyone else who was adversely affected by the outage, please contact me directly at mike@youknowwhat or call my cell at 647.204.3808 so that we can go over things and I can look into crediting your account.

By: Tony

Tony — Tue, 13 Nov 2007 23:56:07 +0000

Bad timing guys. I was pitching for new work yesterday and when the client went looking for an estimate they got a PAGE CANNOT BE FOUND. As a web developer they are hardly going to hire somone who cannot even manage their own internal processes. It may have been night time for you guys but it was peek work hours here in Australia.

By: Holly

Holly — Tue, 13 Nov 2007 19:06:20 +0000

Thank’s for the info guys. Just wanted you to know I really appreciate the work you do! People now-a-days don’t take enough time to thank the people that help them do their job at the proffesional level that you allow us to do ours!
Well done! Thanks for making me look good!
Cheers!

By: Ian Rae

Ian Rae — Tue, 13 Nov 2007 17:31:05 +0000

Thanks for the details Mike, we’ve had to deal with many such incidents over the past 10 years and the lesson you learn over time is that every physical instance of your app will experience unavoidable downtime. The good news is that it is remarkably cost effective nowadays (especially given that you’re working on a LAMP stack) to deploy to multiple pops and route requests based on application availability, proximity or other factors and provide you with much more control, and remove dependence on a single provider. In Montreal. it costs about 2G/mo for a multi site infrastructure (2 racks) with fiber connecting the sites directly for database replication.

We use Basecamp and Freshbooks, so obviously we have a vested interest in you guys staying up!