But there’s good news: after a long day of checking every corner, we managed to track down the problem to an unexpected bottleneck in MySQL. Clearing up that bottleneck has let us tap the full speed of our new
hardware.

Among other measurements, we monitor site performance by measuring the amount of time it takes to log in and send an invoice from various locations around the world. In the graph below you can see how eliminating that bottleneck has helped performance significantly:

FreshBooks is now running faster than ever. Thanks to those of you who struggled through yesterday with us.  We are now pleased to introduce you to our new hardware and the fastest running FreshBooks experience to date – enjoy!

For those curious about what happened, there’s some details under the cut.

The problem was caused by our new configuration of MySQL.  The usual approach to setting MySQL’s table cache size is to make it large enough that it can hold all of your tables open, but we have too many tables for that to be practical. Instead, we decided to make it big, to improve the odds that tables we need were already in the cache. And since we had new, bigger hardware with more RAM and faster disks, we decided to cache even more tables.

Unfortunately, while opening tables is fast in MySQL, the speed at which it can close tables gets slower and slower as the table cache grows. Linearly, in fact. And since the cache will always end up full, the speed that MySQL can close tables becomes the speed that MySQL can open tables. And to exacerbate the problem, opening and closing tables in MySQL uses a global lock, so only one thread can be opening or closing tables at a time, regardless of how many threads are executing in parallel.

Once we understood this, we realized that if the cache is going to be full, it needs to be kept as small as possible. We reduced it to the minimum we could get away with without having to reopen tables in the middle of one session interacting with FreshBooks. And now that closing tables is faster, everything’s faster.

It’s possible other patrons raised their eyebrows at FreshBooks business cards sticking out of picture frames with encouraging messages like “This Way!”, but we didn’t receive any complaints.

A bunch of FreshBooks customers and a few of us FreshBookers gathered Thursday night to get together and talk about the ups and downs of running a business these days. We had Connie, the librarian-turned-social-networks-consultant. We had Emmanuel, the Flash programmer running his own studio. There were computer consultants, artist representatives, overseas traders and web developers. A diverse bunch, to say the least.

Josh and I shared stories about our respective times in Tokyo, and Kimberly confirmed for me the story about the flaming Taipei monorail I’d only ever heard from an insane painter (it was good to get it from a slightly more reliable source). I talked a great deal with Jerry about how the study of literature relates to software development, and James told me about a crazy spreadsheet wizard who can predict the future with spooky accuracy.

It was great getting together with all these folks and hearing their stories, understanding how FreshBooks can provide value to so many different types of users. Running your own business is a lot of work, and it’s pretty amazing to feel like we’re helping these folks realise their dreams. I was actually touched when I asked Emmanuel when he started up his studio and he said, “The same time I signed up for FreshBooks.”

I didn’t hear back from anyone who got lost in the place looking for us, so I assume the business card breadcrumbs were 100% successful. We’d love to show you pictures of the event, but to tell you the truth, we were all having so much fun we completely forgot to break out the cameras!

Not different just to be different but different because different is interesting (I tried to add one more “different” but couldn’t make it fit…oh I guess I just did).

A great example of this took place last week at the How Design Conference.

You see we decided to exhibit at this conference and after we registered we realized that we didn’t have one of those super slick and fancy Trade Show Booths that make lasting impressions to all that see them.

This was a problem since having a poor Trade Show booth is probably worse then not having one at all.

So putting our “Fresh” glasses on we came up with a cool solution…we basically did the one thing no one expected to see at a design conference!

Instead of having a fancy booth made for us we decided to get two 8 Foot x 8 Foot Canvases. Jazz Martin – an artist who is also a FreshBooks user – took some glue and paint and created a living breathing (always changing) art installation that started at the first moment that the conference began and was worked on until the last moment of the event.

This accomplished a bunch of cool things for us:

• People liked it
• People talked about it
• People kept coming back to see the progress

We were so pleased with how this turned out and proved to ourselves once again that when you replace ordinary and boring with extraordinary, you can never really go wrong!

Update! Check out the time-lapsed music video we made of the fun we had in the trade show booth!

If you are interested in the history surrounding this May holiday, Daniel wrote about it here.

Mitch is a marketing veteran. He started working on his marketing chops at Proctor and Gamble — arguably the best outfit in the world to learn management and traditional marketing acumen. But I’d say Mitch is a little restless at heart, and he needed to be somewhere more nimble and faster growing, so he left for Lavalife where he was employee number one in the marketing department. He stayed for nine years. He left as their VP marketing, managing a team of 27 with 6 direct reports, and in the process turned Lavalife into Canada’s household name for online dating — like Match.com or eHarmony in the USA.

What a business needs to grow are pillars to build on, and those pillars are people — more specifically great managers. After doing some consulting last year with us, we knew Mitch would be a great and timely fit for FreshBooks. He’s an ace demand generation marketer, a great people manager, and he’d done it before. Sadly he was committed to other projects, but by the end of the year we were able to convince him that FreshBooks was the place he could make a real impact, and as you can tell, we feel he has and he will continue to for a long time to come.

So… with that, please join me in welcoming Mitch to the team.

You will work together with other business groups to drive product enhancements and new products to help grow the FreshBooks service and online community. You will be unafraid to ask the tough questions of both other managers as well as product developers. You know how to get the most out of people and have a keen sense of “what counts” when it comes to product and business decisions.

If you’re interested, or know somebody who might be, please check out our careers page for more information on this great new opening!

Yes, this post is even later than last year’s, but we had to wait for the statutes of limitation to run out.

We were fortunate enough to have thrice as much moustache as ‘07. Take a look at the contributions that made this year such a smashing success:

Left to right: Taavi, sporting a classic mouthbrow; Alistair, from his live showercam; and me, straight from the cab of my 18-wheeler.

Our new support rotation program

With over 300,000 users and growing, and with new team members on board, our support processes needed some attention to make sure we were always on top of providing great service. We wanted to find a way to scale our support efforts as demand continued to grow.

Quick turnaround time in support is a top priority here at FreshBooks, so a support rotation program was devised. Each employee does email, phone and forum replies once a week to stay in touch with our users; every one of us becomes what I call a “support lead,” to make sure all support queries are taken care of. Two or three support leads collaborate to form a support team for the day.

What we’ve seen so far

While some support calls take a bit longer as our “new recruits” get familiar with every nook and cranny about FreshBooks, everyone understands the reason we’re doing it and our customers are being super supportive of their obvious efforts. We expect the program to last as we double our user base, and by that time we’ll have weaned a few staff members off support, with the aim of building a dedicated support team with the help of the experience we gather from this program.

Since starting over a month ago, we have benefited from the rotation in a few ways:

• The whole team is more aware of the current set of feature requests and necessary enhancements.
• Exposure to support means everyone has a better understanding of our users’ most common issues, and a deeper knowledge of the functionality of the product.
• Better understanding of the most common support questions builds up our collection of workarounds and solutions.
• Support time allows us to exercise team effectiveness in communication, coordination, and cooperation.
• The stigma of doing support has vanished because everyone in the company is doing it.
• The rotation brings structure to our weekly schedule, allowing us to focus on both support and our individual superpowers.

Support builds a humble person

In addition to being current and up-to-date with the rotation program, the team also benefits from this humbling support role. As one of our customers, Zane Safrit, says, “putting others’ needs first builds a humble person.”

It takes quite a bit of patience from every single one of us, and creates a staff with rock star skills.

It’s official: FreshBooks RoadBurn 2008 has begun!

Yesterday our intrepid travellers attended BarCamp Miami, and today they’re enjoying the excellent Future of Web Apps conference.

According to the FOWA Web site, starting this evening the whole thing apparently just turns into one big party! A beach party this evening, a breakfast tomorrow morning, and another beach party after that. I always knew my Dad was lying when he said business trips were no fun.

Keep tabs on RoadBurn 2008

Be sure to follow our crew’s journey on the RoadBurn blog, and keep up with new posts by subscribing to the RSS feed or signing up for daily e-mail updates.

Oh, and if you’re somewhere between Miami and Austin, get in touch with us! Judging by the first video update from RoadBurn, Sunir has already kicked off the “most horrific Hawaiian shirt” contest with a real doozy. I’d like to see somebody top that thing.

Toronto Marlies hockey game

On Sunday, in conjunction with Yahoo! Search Marketing Canada and PodCamp Toronto, we held a contest to take folks out to the good ol’ hockey game. We also brought along a few of our customers. It was a sell out game, and although the Marlies lost 1-2 to the Hamilton Bulldogs, it was a hoot. Thank you, Yahoo! for providing the box and the munchies. And go Marlies! We’re rooting for you to win the Calder Cup!

Left to right: Ahmed El-Daly (Rails ninja), Dmitrii Boldyrev (Makmalauto), Val Zamulin (Seologist), Sunir Shah (me), Tommy Vallier (Talk shoe), Greg Majster (Stro1), Natalie Bomberry (pilot PMR), Himy Syed (Torontopedia), Allison Greenbaum (The Little Green Book), Alex Lahaie (Yahoo!), and crouching, Jp Dhanoa (Techsos).

Toronto customer dinner

To kick off our RoadBurn road trip from Miami to Austin, we took a few customers out to dinner in our home town yesterday.

Back row: Aaron, Mitch, and Sunir from FreshBooks. Front row: Paul Vlaovic (Sequoia Legal), Alfredo Arias (Flash Fog Security), and Stephen Megitt (Filament Creative).

Thanks for the great time, folks!