Archive for August, 2006
The past 48 hours have been intense. Last night we got some rest and now we are beginning to reflect on our trials and tribulations.
If you read this post, you know that as an acknowledgement of our recent troubles, we decided to upgrade every FreshBooks account – even those unaffected by the issues.
Now, I’m a numbers guy. When I can, I like to make decisions based on numbers, carefully considering how things will affect the bottom line. When we decided to upgrade everyone’s account. I didn’t run the numbers. We just did it, because we felt it was the right thing to do.
Consider another scenario. One where we did not own the company and outsiders did. How would our decision to upgrade everyone’s account have played out in that context? We made the decision at approximately 2:00 AM. The process was simple: I asked Joe and Daniel if we should do it. They said yes. We did it.
In a larger company, or at a company owned by outsiders, I suspect this decision might have been much more challenging. At the very least, I suspect that more people would have had to have been consulted. That would have taken time. In times of trouble I believe that being able to communicate quickly and clearly is essential. The notice on our home page and the corresponding blog post were huge assets yesterday.
There is a long history of businesses responding to challenging times in much the same way we did yesterday. The classic example is Johnsons and Johnsons Tylenol recall. In that case the problem with the tampered Tylenol bottles was isolated. Yet despite that, Chairman James Burke swiftly decided to recall all the bottles across the nation. And there are other examples, countless in fact.
So in trying to learn from this episode – a process I am sure will go on for weeks to come – so far I can tell you this:
Communicate early.
Communicate often.
Communicate honestly.
I heard Yung Wu speak last year and he said, “In tough times, over communicate.” (tip #11). To be honest I think this whole episode may have been driven by that advice.
Sadly, after almost two and a half years of uninterrupted service, our streak of 100% uptime has come to an end in a rather painful way.
What Happened Exactly
Yesterday morning we were experiencing sub-optimal performance with a piece of hardware and we decided to replace the hardware and nip the problem in the bud. Addressing the problem immediately was the right decision. At the time we were counselled that either:
a) no downtime would be required to remedy the problem
b) a half hour window would be required
Ironically, part of our infrastructure (we are Raid 5) was designed so that this kind of maintenance would be seamless. So with the above counsel from our managed hosting provider RackSpace, we elected to proceed with the maintenance. Then trouble and pain set in…
In a nutshell, yesterday August 29th we were forced to migrate our entire infrastructure. I assure you this is not what was planned, but it became the best solution given the circumstances. Thanks to our managed back-ups, mirrored storage and previous infrastructure upgrade, this process was mostly smooth, but many stages of the migration of the process (copying databases is one example) simply cannot be accelerated.
The Problem That Arose
Besides downtime, one problem did arise for some FreshBooks systems however. As a result of one drive’s mirrored storage hardware failure – the catalyst that started this series of events – approximately 12 hours of account activity which was logged from 12:00 AM to 12:00 PM EDT on August 29th is currently irretrievable for a small percentage of FreshBooks users. Only those systems supported by the affected hardware AND who were active (i.e. sent invoices, created invoices, updated timesheets…) between 12:00 AM to 12:00 PM EDT on August 29th were affected. The vast majority of FreshBooks users would not have had any account activity during this period. Only a small percentage were sustained by the affected hardware.
What Caused the Problem
The cause of the loss and the downtime is not yet clear. Again, our infrastructure was designed to sustain 100% uptime in this exact scenario. The problem affected a mirrored drive that sustained real-time storage for a subset of FreshBooks accounts – the ones with the irretrievable account activity. We will be working with RackSpace (our managed hosting provider) to ascertain the root cause. However, up to this point we have been focusing our efforts on making sure all our users are taken care of first, before we spend our resources finding out the specifics of the cause. So until all the facts are in, I am going to withhold further comment as to the direct cause. When we do get to the bottom of things, we will share the details.
What We Are Doing About It
In the meantime, we want to clearly acknowledge the loss. We are upgrading EVERY FreshBooks account – even those not effected – as follows:
0-3 Clients (free) – 3 extra clients
3-25 Clients (free) – 25 extra clients
26-100 Clients (free) – 50 extra clients
100+ Clients (free) – 100 extra clients
How Do You Know If You Were Affected?
1. Log into your FreshBooks account. In the news section on the home page we will tell you IF your system COULD HAVE BEEN affected. Again, if your account was not active (i.e. sent invoices, created invoices, updated timesheets…) between 12:00 AM to 12:00 PM EDT on August 29th, you would have no missing data.
2. Check your email. We are sending two different emails to all FreshBooks users: one for those accounts affected, and one for those that were not affected. Be sure to check your spam filter to be sure our email did not wind up there.
What to Do if Your Account was Affected
If your account was active during the window outlined above AND your system was affected, then here is a list of considerations:
1. Any invoices, support tickets and clients you created and/or edited, along with uploaded documents, timesheet hours entered, etc. will have to be re-entered.
2. Any emails to clients and/or staff for a newly created invoices and/or support tickets may contain links that will no longer function, so be sure to resend your emails when you recreate your invoices, tickets, etc.
3. Any successful auto-billed transactions or online payments made by your clients during that period may not be “marked as paid” any longer. You should reconcile your invoices with any payments received on Aug 29th – check your payment gateway or your PayPal account for transaction details.
Update: this just in…if you subscribe to your RSS feed of “recent activity” and you had your feed reader on yesterday, you can use your feed to see what recent activity might have been lost, and then use that activity to recreate your missing activities. Thanks to Frank P. for sending this tip along.
We Are Truly Sorry
For anyone who was inconvenienced by the interruption of service and/or irretrievable data, myself and the entire FreshBooks teams are deeply sorry. I want to extend our thanks to those of you who called and emailed to enquire about the problem. To a person, everyone was polite and understanding, which under the circumstances, was greatly appreciated by myself and the other FreshBooks staff who were hard at work bringing the service back online.
A Final Word On Timing
We spoke with many of our users by telephone and email yesterday. I personally spoke with close to one hundred. I want to apologize to you those of you to whom we reported ANY kind of status update with regards to when we EXPECTED the hardware upgrade to be complete. Throughout yesterday we were given misleading information and we were reassured the service would be back up, “in about one hour”. FreshBooks team members passed this information on – myself included – and it proved to be woefully wrong. We are incredibly sorry for sharing information that proved to be misleading and will learn from this experience.
Here is an excerpt from Small Giants – a fantastic book about exceptional small companies. Here is a quote from the founder of Inc Magazine:
“I kept thinking the entrepreneur is like an artist, only business is the means of his expression…He creates [a business] from nothing, just like a blank canvas. It’s amazing. Someone goes into a garage, has nothing but an idea, and out of the garage comes a company, a living company. It’s so special what they do. They are a treasure”.
At the time Bernie Goldhirsh (Inc’s founder) came to the realization above, entrepreneurs were looked down upon – like business outcasts in a time when everything in business revolved around big corporations. Thanks to resources like Inc, the tide has changed and now many entrepreneurs – even those running small companies – are revered.
To me entreprenuers are a constant source of inspiration – just like artists. Reading business magazines and company profiles are as enjoyable as reading the biographies of my favourite musican’s, like B.B. King for example. So if you are running a business – no matter what size – don’t forget that you too are an artist, and your business is an expression of your talent, and everyday is a new opportunity to hone your craft.
I’m about to coin a phrase, or make a fool of myself by describing a concept that has been around for ages. Hold on to your hats, here come my thoughts on “transitional services”.
Transitional Services are services that facilitate a user’s transition from one platform to the next – or at the least, ease their pain.
Whenever there is a platform shift, there is transition, and straddling. For example, for the past ten years the photography industry has been shifting from celluloid to digital. The industry and its consumers are undergoing a transition from one platform to another. This transition has consequences. Many users are reluctant to transition because they are invested in the first platform (i.e. “I have cameras and film, slide projectors and photo albums”). Once the decision to transition has been made, users may want to bring their old platform content (think printed photos) with them to the new platform format (think scanning photos) and they find themselves at a point where they are straddling the new platform and the old. Both the transition and the straddling phases create pain and opportunity in the marketplace.
With me so far?
I wrote Paul Kedrosky a note saying I think there is a huge and growing market for transitional services in the Web 2.0. I pointed out how helping people get from offline processes to online processes – while helping to ease the pain of the straddling phase – will be a strategy that start-ups and established players can leverage and that I foresee an increasing number doing so in the coming years.
This whole conversation was sparked by FreshBooks recent release of its transitional ground mail service. The solution FreshBooks is selling is to help business transition their invoicing/receivables process online where significant benefits can be realized (streamlined processes, reduce costs, and improved customer relations). Businesses want to get online, but there is a world of pain awaiting them in the transition phase (“How do we build the service we need?”) and straddling phases (“How do we manage our cash flow when half our clients pay us online and half pay us offline?”)
That ability to painlessly transition customers from ground mail invoices to online invoices and recurring billing is what FreshBooks offers, but there are other examples of businesses that facilitate traditional office activities. You can create and send photo albums as gifts via Flickr. This is an example of a reverse transitional service where Flickr is facilitating a transition from the new platform (digital images) to the old (printing and mailing images).
What’s magical about all of this, and a hallmark of a transitional service in the Web 2.0, is how the line between the online world and the offline world blurs. The slicker the service, the more seamless the delivery, the more the offline world gets pulled online.
In terms of opportunities, I foresee more and more services leveraging transitional strategies and delivering transitional services as value added backend services and incremental revenue generators.
So, while none of these concepts is new, and the act of delivering such services has been around for some time, I have seen no overt attempts to define the phenomenon, so I have done it here. If it has been done elsewhere, please let me know – rain on the parade, it’s okay. As I have not had as much time as I would like to consider all the implications of transitional strategies, I encourage you to sound off with your own thoughts. Can you think or other examples? Better yet, can you think of industries in need of transitional services, where ripe opportunities exist? Please comment below.
Earlier this week FreshBooks exceeded the 80,000 user count. Along with the much anticipated ground mail release yesterday, we’ve recently received positve mentions from the likes Rafe Needleman of CNET and Marshall Kirkpatrick of Techcrunch.
With a user count milestone, a successful ground mail release, some honourable mentions plus the end of summer around the corner, I sense some after work premium lager out on the ol’ FreshBooks patio sometime soon.
Cheers.
We released our ground mail service today. Someone bought 100 stamps within the first 90 seconds. That’s a nice validation.
The real story with our release today is this: countless businesses that are STUCK in the old/offline world and trying to get to the web so they can leverage the benefits of the internet (cost savings, streamlined processes, improved client relations). Not many services are really assisting them with the transition, nor are many Web 2.0 services helping to facilitate traditional offline activities (Flickr offering to print and send photo albums is a nice example of someone who is doing this). I think there is a huge market and a huge need for services that help bridge this gap and do not simply turn their back on those users that need some aspect of their experience to be an offline experience.
Our ground mail offering is a nice example of where things are heading, and how companies like FreshBooks are empowering individuals and small businesses with tools that those individuals and small businesses could not otherwise afford to possess…more services will do similar things in the coming months and years, and I am delighted about this fact.
Marshall TechCrunched us here:
http://www.techcrunch.com/2006/08/23/freshbooks-pushes-the-envelope-in-online-billing/
New in version 3.4 is the ability for you to receive your recent activity updates in an RSS feed. For those new to the world of RSS, here is a little guide to adding your RSS feed to a feed reader. In the example below I will be using a feed reader recommended by FreshBooks called Feedreader.
1. From within your FreshBooks account, browse to the “Home” tab and click on the feed icon link.
2. Right click on the your RSS feed URL and select “Copy Shortcut”.
3. In your RSS reader select a button or link that allows you to add a new feed. When asked for the URL to your feed, paste in your FreshBooks feed URL. In Feedreader, simply click on the “New” feed button and paste the URL in the address field.
4. When adding a secure feed you will need to specify a username and password or login credentials. Enter your FreshBooks login information. In Feedreader you will receive a popup like the one below.
Simple as that! Now you should receive recent activity updates in your RSS reader. Please note that only an RSS reader that supports RSS authentication is supported.
I am beginning to see more Doom and Gloom posts lately. So I figured I’d take a moment to write a post about the single biggest obstacle standing between anyone who is thinking about running a business and their success: FEAR of FAILURE.
That’s right. It’s not money, or strategy, or talent, that stops entrepreneurs. It’s fear of failure that stops them – in many cases, it stops great businesses before they even begin.
Many people dream of starting their own business, but they shy away citing one reason or another (”the kids, the house, the pension…”). It’s very very easy to rationalize why you should not do something risky. Very easy. But there is nothing like running your own business, and the beauty is, anyone can.
If you have ever thought of running a business, read the following excerpt. If you are running a business and you are afraid of your competition, your impending deadlines, or of making a change in the way you do things, read this quote. It’s by Yann Martel from the Life of Pi:
I must say a word about fear. It is Life’s only true opponent. Only fear can defeat life. It is a clever, treacherous adversary, how well I know. It has no decency, respects no law or convention, shows no mercy. It goes for your weakest spot, which it finds with unerring ease. It begins in your mind, always. One moment you are feeling calm, self-possessed, happy. Then fear, disguised in the garb of mild-mannered doubt, slips into your mind like a spy. Doubt meets disbelief and disbelief tries to push it out. But disbelief is a poorly armed foot soldier. Doubt does away with it with little trouble. You become anxious. Reason comes to do battle for you. You are reassured. Reason is fully equipped with the latest weapons technology. But, to your amazement, despite superior tactics and a number of undeniable victories, reason is laid low. You feel yourself weakening, wavering. Your anxiety becomes dread.
Fear next turns fully to your body, which is already aware that something terribly wrong is going on. Already your lungs have flown away like a bird and your guts have slithered away like a snake. Now your tongue drops dead like an possum, while your jaw begins to gallop on the spot. Your ears go deaf. Your muscles begin to shiver as if they had malaria and your knees to shake as though they were dancing. Your heart strains too hard, while your sphincter relaxes too much. And so with the rest of your body. Every part of you, in the manner most suited to it, falls apart. Only your eyes work well. They always pay proper attention to fear.
Quickly you make rash decisions. You dismiss your last allies: hope and trust. There, you’ve defeated yourself. Fear, which is but an impression, has triumphed over you.
The matter is difficult to put into words. For fear, real fear, such as shakes you to your foundation, such as you feel when you are brought face to face with your mortal end, nestles in your memory like a gangrene: it seeks to rot everything, even the words with which to speak of it. So you must fight hard to express it. You must fight hard to shine the light of words upon it. Because if you don’t, if your fear becomes a wordless darkness that you avoid, perhaps even manage to forget, you open yourself to further attacks of fear because you never truly fought the opponent who defeated you.
If your fear is fear of failure, you need to embrace your fear, TRY TO START YOUR BUSINESS and see where you wind up. My favourite discussions with entrepreneurs, are discussions of failure…where and how. You will make mistakes, but that’s okay because you will learn, not just how to run a business successfully, but you will learn the rewards that reside on the other side of your fear. Good luck.
Now FreshBooks is truly a billing platform. Why are we a billing platform? With the release of our API, businesses can leverage the benefits of FreshBooks and integrate FreshBooks into that existing process in much the same way as businesses use Salesforce’s App Force to leverage SalesForce.
A major benefit of FreshBooks is it helps business transition their receivables process from the offline to the online world. If you are an alarm service with 5,000 clients, it likely costs you $3,000-$4,000/month to send your invoices in the mail. If those same clients received their invoices electronically, that would save you $3,000/month or $40,000-50,000/year and speed up collections. The problem is it’s hard to smoothly transition your clients from receiving invoices by post to receiving them online. That is where FreshBooks comes in. By leveraging the FreshBooks API businesses can leverage the FreshBooks platform and transition your customers smoothly. Don’t build your own solution – it’s risky. Instead, build on a proven platform.
We are about to release Ground Mail and API support for it. We will soon be releasing version II of our API. It will make developers more powerful. We hope you will build on this platform – go ahead and get started.
Levi wrote this about three years ago. I’m republishing it today because, much of it is as true today as it ever was. Sometimes people tell us they were going to build their own online billing solution until they came across FreshBooks and realized there was no point. Leveraging a proven solution makes a lot of sense – it’s one of the major benefits of Software as a Service (SAAS). Here’s why:
Quality of Application
No organization today builds their own word processing application, they purchase Microsoft Word or Corel WordPerfect, which are far cheaper and superior to anything that could be developed in house. Publicly licensed software designed for mass consumption is generally superior to applications developed in house primarily because of economies of scale. Why would an online invoicing solution be any different? If you do decide to develop your own solution it is clear that the security, features and reporting capabilities will not be as robust, stable or as user friendly as a professionally developed application built by specialists.
Interface Design and Customization
Perhaps the number one reason many SMB’s choose to develop an in-house solution is so that they can design and customize the user interface to fall in line with their corporate identity. You may have noticed that some (roughly 10%) of shopping cart applications will redirect users to external pay pages during the checkout procedure. The other 90% consciously decided to incur other additional costs to ensure their shoppers are not forced to leave the familiar environment of their shopping cart during the payment process. To ensure this, each one has born the additional cost to purchase an SSL certificate ($200 to $700 US annually) and to develop a payment plug-in, which necessitates a proper software development cycle (including specification design, build, test and installation) which costs in the range of $2,000.00 to $5,000.00. These costs alone can finance years of service from an online invoicing application service provider.
Direct Costs
There are high costs associated with developing your own online invoicing system. Looking at the tasks involved, it is easy to see why. A typical online application development project would likely have at least some of the following tasks, if not more:
- Needs analysis assessment
- Business process fit-gap analysis
- Application interface design
- Database design
- Technology assessment, costing and procurement
- Server and security setup
- Application build
- Application Testing
- Pilot Rollout
- BETA Testing
- Production Rollout
- Ongoing Support and Maintenance
If outsourced, projects of this nature will take at least four to eight months to complete and cost from $10,000.00 to $250,000.00 depending on the complexity of the application and the security level required.
Indirect Costs
Furthermore, many application development project costs are hidden. Redirection of resources is one example of a hidden cost that often goes unaccounted for. Every hour your team members spend developing an invoicing system, they could be focusing on your core business activities. Also, if your team members are not experienced in online invoice application development, there will be a high learning cost added to the project, and without question, increased chance your project will fail.
Risks
Author Julia King stated that for companies that are not among the top 25% of technology users, thirty percent of all technology projects fail.(i) The good news is that 70% are successful; however Julia did not research how many of those successful projects went over budget. Some research indicates as many as half of all IT projects go over budget. If your company is not among the top quarter of technology users, chances are your online application development project is in jeopardy of failing and/or going over budget. Worse still, it is most likely your final product will be of lower quality and security than one developed by a specialist.
Maintenance
Online applications demand ongoing maintenance, and in rapidly evolving markets like e-commerce, constant change. This maintenance – which can include bug fixes, transitions to superior technologies, server patches to name a few – is a source of organizational stress and in most cases best outsourced to specialists.
Security
Application security is defined by the application design, the technology selection, the safeguards implemented and by the team who vigilantly monitors the application 24 hours x 7 days a week. Most SMB’s do not have the resources to adequately invest in these areas of a project. They only have the resources to achieve the necessary functionality the applications demand and have to sacrifice security. Proper investment in expert developers, testing, dedicated firewalls and round the clock monitoring are all costly components of a secure online application.
Services and Storage
An online application hosted in-house will have costs that can easily be overlooked when initial specifications are done. There is the cost of the space and setup required for a secure and stable server room. Some may say that they have already spent this on their current server room, but adding business critical servers may tip the limits of your current server room configuration. Adding more online servers inherently increases your risk of security threats your entire network from viruses and hackers. In addition, a brief power failure could cost your IT staff – and the rest of your business – dearly when you are hosting your own solution. Another necessary cost when hosting your own solution is the time and money required to do proper backups.
Conclusion
Considering all the reasons mentioned here, and all the reasons not mentioned, it is clear that developing your own online invoicing and payment system is a high cost, high risk undertaking. Will your business be able to afford development and hosting costs when they hit your bottom line? Do you even have the capital and the know how to consider developing your own solution? You owe it to your business to do the research, consider every direct and hidden cost of developing your own system, and then to take a good look at the alternatives. We’re betting it’s not worth the effort, expense or risk – especially when you can leverage an API (like FreshBooks’ API) – and achieve virtually any level of customization
(i) King, Julia. “Survey shows common IT woes“, Computerworld, June 23, 2003
Copyright © 2000-2010