The past 48 hours have been intense. Last night we got some rest and now we are beginning to reflect on our trials and tribulations.

If you read this post, you know that as an acknowledgement of our recent troubles, we decided to upgrade every FreshBooks account – even those unaffected by the issues.

Now, I’m a numbers guy. When I can, I like to make decisions based on numbers, carefully considering how things will affect the bottom line. When we decided to upgrade everyone’s account. I didn’t run the numbers. We just did it, because we felt it was the right thing to do.

Consider another scenario. One where we did not own the company and outsiders did. How would our decision to upgrade everyone’s account have played out in that context? We made the decision at approximately 2:00 AM. The process was simple: I asked Joe and Daniel if we should do it. They said yes. We did it.

In a larger company, or at a company owned by outsiders, I suspect this decision might have been much more challenging. At the very least, I suspect that more people would have had to have been consulted. That would have taken time. In times of trouble I believe that being able to communicate quickly and clearly is essential. The notice on our home page and the corresponding blog post were huge assets yesterday.

There is a long history of businesses responding to challenging times in much the same way we did yesterday. The classic example is Johnsons and Johnsons Tylenol recall. In that case the problem with the tampered Tylenol bottles was isolated. Yet despite that, Chairman James Burke swiftly decided to recall all the bottles across the nation. And there are other examples, countless in fact.

So in trying to learn from this episode - a process I am sure will go on for weeks to come – so far I can tell you this:

Communicate early.
Communicate often.
Communicate honestly.

I heard Yung Wu speak last year and he said, “In tough times, over communicate.” (tip #11). To be honest I think this whole episode may have been driven by that advice.

Sadly, after almost two and a half years of uninterrupted service, our streak of 100% uptime has come to an end in a rather painful way.

What Happened Exactly
Yesterday morning we were experiencing sub-optimal performance with a piece of hardware and we decided to replace the hardware and nip the problem in the bud. Addressing the problem immediately was the right decision. At the time we were counselled that either:

a) no downtime would be required to remedy the problem

b) a half hour window would be required

Ironically, part of our infrastructure (we are Raid 5) was designed so that this kind of maintenance would be seamless. So with the above counsel from our managed hosting provider RackSpace, we elected to proceed with the maintenance. Then trouble and pain set in…

In a nutshell, yesterday August 29th we were forced to migrate our entire infrastructure. I assure you this is not what was planned, but it became the best solution given the circumstances. Thanks to our managed back-ups, mirrored storage and previous infrastructure upgrade, this process was mostly smooth, but many stages of the migration of the process (copying databases is one example) simply cannot be accelerated.

The Problem That Arose
Besides downtime, one problem did arise for some FreshBooks systems however. As a result of one drive’s mirrored storage hardware failure – the catalyst that started this series of events - approximately 12 hours of account activity which was logged from 12:00 AM to 12:00 PM EDT on August 29th is currently irretrievable for a small percentage of FreshBooks users. Only those systems supported by the affected hardware AND who were active (i.e. sent invoices, created invoices, updated timesheets…) between 12:00 AM to 12:00 PM EDT on August 29th were affected. The vast majority of FreshBooks users would not have had any account activity during this period. Only a small percentage were sustained by the affected hardware.

What Caused the Problem
The cause of the loss and the downtime is not yet clear. Again, our infrastructure was designed to sustain 100% uptime in this exact scenario. The problem affected a mirrored drive that sustained real-time storage for a subset of FreshBooks accounts – the ones with the irretrievable account activity. We will be working with RackSpace (our managed hosting provider) to ascertain the root cause. However, up to this point we have been focusing our efforts on making sure all our users are taken care of first, before we spend our resources finding out the specifics of the cause. So until all the facts are in, I am going to withhold further comment as to the direct cause. When we do get to the bottom of things, we will share the details.

What We Are Doing About It
In the meantime, we want to clearly acknowledge the loss. We are upgrading EVERY FreshBooks account – even those not effected – as follows:

0-3 Clients (free) – 3 extra clients
3-25 Clients (free) – 25 extra clients
26-100 Clients (free) – 50 extra clients
100+ Clients (free) – 100 extra clients

How Do You Know If You Were Affected?
1. Log into your FreshBooks account. In the news section on the home page we will tell you IF your system COULD HAVE BEEN affected. Again, if your account was not active (i.e. sent invoices, created invoices, updated timesheets…) between 12:00 AM to 12:00 PM EDT on August 29th, you would have no missing data.

2. Check your email. We are sending two different emails to all FreshBooks users: one for those accounts affected, and one for those that were not affected. Be sure to check your spam filter to be sure our email did not wind up there.

What to Do if Your Account was Affected
If your account was active during the window outlined above AND your system was affected, then here is a list of considerations:

1. Any invoices, support tickets and clients you created and/or edited, along with uploaded documents, timesheet hours entered, etc. will have to be re-entered.

2. Any emails to clients and/or staff for a newly created invoices and/or support tickets may contain links that will no longer function, so be sure to resend your emails when you recreate your invoices, tickets, etc.

3. Any successful auto-billed transactions or online payments made by your clients during that period may not be “marked as paid” any longer. You should reconcile your invoices with any payments received on Aug 29th – check your payment gateway or your PayPal account for transaction details.

Update: this just in…if you subscribe to your RSS feed of “recent activity” and you had your feed reader on yesterday, you can use your feed to see what recent activity might have been lost, and then use that activity to recreate your missing activities. Thanks to Frank P. for sending this tip along.

We Are Truly Sorry
For anyone who was inconvenienced by the interruption of service and/or irretrievable data, myself and the entire FreshBooks teams are deeply sorry. I want to extend our thanks to those of you who called and emailed to enquire about the problem. To a person, everyone was polite and understanding, which under the circumstances, was greatly appreciated by myself and the other FreshBooks staff who were hard at work bringing the service back online.

A Final Word On Timing
We spoke with many of our users by telephone and email yesterday. I personally spoke with close to one hundred. I want to apologize to you those of you to whom we reported ANY kind of status update with regards to when we EXPECTED the hardware upgrade to be complete. Throughout yesterday we were given misleading information and we were reassured the service would be back up, “in about one hour”. FreshBooks team members passed this information on – myself included – and it proved to be woefully wrong. We are incredibly sorry for sharing information that proved to be misleading and will learn from this experience.

Here is an excerpt from Small Giants - a fantastic book about exceptional small companies. Here is a quote from the founder of Inc Magazine:

“I kept thinking the entrepreneur is like an artist, only business is the means of his expression…He creates [a business] from nothing, just like a blank canvas. It’s amazing. Someone goes into a garage, has nothing but an idea, and out of the garage comes a company, a living company. It’s so special what they do. They are a treasure”.

At the time Bernie Goldhirsh (Inc’s founder) came to the realization above, entrepreneurs were looked down upon - like business outcasts in a time when everything in business revolved around big corporations. Thanks to resources like Inc, the tide has changed and now many entrepreneurs - even those running small companies - are revered.

To me entreprenuers are a constant source of inspiration - just like artists. Reading business magazines and company profiles are as enjoyable as reading the biographies of my favourite musican’s, like B.B. King for example. So if you are running a business - no matter what size - don’t forget that you too are an artist, and your business is an expression of your talent, and everyday is a new opportunity to hone your craft.

I’m about to coin a phrase, or make a fool of myself by describing a concept that has been around for ages. Hold on to your hats, here come my thoughts on “transitional services”.

Transitional Services are services that facilitate a user’s transition from one platform to the next – or at the least, ease their pain.

Whenever there is a platform shift, there is transition, and straddling. For example, for the past ten years the photography industry has been shifting from celluloid to digital. The industry and its consumers are undergoing a transition from one platform to another. This transition has consequences. Many users are reluctant to transition because they are invested in the first platform (i.e. “I have cameras and film, slide projectors and photo albums”). Once the decision to transition has been made, users may want to bring their old platform content (think printed photos) with them to the new platform format (think scanning photos) and they find themselves at a point where they are straddling the new platform and the old. Both the transition and the straddling phases create pain and opportunity in the marketplace.

With me so far?

I wrote Paul Kedrosky a note saying I think there is a huge and growing market for transitional services in the Web 2.0. I pointed out how helping people get from offline processes to online processes – while helping to ease the pain of the straddling phase - will be a strategy that start-ups and established players can leverage and that I foresee an increasing number doing so in the coming years.

This whole conversation was sparked by FreshBooks recent release of its transitional ground mail service. The solution FreshBooks is selling is to help business transition their invoicing/receivables process online where significant benefits can be realized (streamlined processes, reduce costs, and improved customer relations). Businesses want to get online, but there is a world of pain awaiting them in the transition phase (“How do we build the service we need?”) and straddling phases (“How do we manage our cash flow when half our clients pay us online and half pay us offline?”)

That ability to painlessly transition customers from ground mail invoices to online invoices and recurring billing is what FreshBooks offers, but there are other examples of businesses that facilitate traditional office activities. You can create and send photo albums as gifts via Flickr. This is an example of a reverse transitional service where Flickr is facilitating a transition from the new platform (digital images) to the old (printing and mailing images).

What’s magical about all of this, and a hallmark of a transitional service in the Web 2.0, is how the line between the online world and the offline world blurs. The slicker the service, the more seamless the delivery, the more the offline world gets pulled online.

In terms of opportunities, I foresee more and more services leveraging transitional strategies and delivering transitional services as value added backend services and incremental revenue generators.

So, while none of these concepts is new, and the act of delivering such services has been around for some time, I have seen no overt attempts to define the phenomenon, so I have done it here. If it has been done elsewhere, please let me know - rain on the parade, it’s okay. As I have not had as much time as I would like to consider all the implications of transitional strategies, I encourage you to sound off with your own thoughts. Can you think or other examples? Better yet, can you think of industries in need of transitional services, where ripe opportunities exist? Please comment below.