On Implementing WebHooks
A few months ago, we announced support for WebHooks in FreshBooks. It’s been said that WebHooks are so simple, you can implement them in one line of code, but in reality, this is rarely the case. When we started thinking about the design of such a system, we had serious reservations about performance and scalability. Specifically, we identified the following challenges / requirements:
- Responsiveness: Outgoing HTTP requests have to be asynchronous. We can’t make application users wait while we send requests to potentially slow endpoint scripts.
- Integrations: Most people using WebHooks would be doing so on behalf of FreshBooks users, therefore we have to provide access to WebHooks over the API as well as from the user interface.
- Security: Because we have to provide API access to WebHooks, we must also ensure that the person creating a WebHook is the owner of the endpoint URI they specify (to prevent people from using FreshBooks to generate tons of traffic to poor, unwitting servers).
- Reliability: WebHooks must be reliable. If an endpoint URI is poorly configured, temporarily innaccessible, or otherwise not ready to receive HTTP requests from FreshBooks, we must be able to provide other chances for the user to receive the payload.
- Privacy: We allow people to create WebHooks with endpoint URIs that use insecure transport layers (i.e. No SSL). As such, we had to make sure that we do not release potentially sensitive information about an account.
- Complexity: It is imperative that we try to reuse existing semantics as much as possible and do not add layers of complexity to the FreshBooks platform.
For a small application, it’s probably sufficient to have the web application code handle outgoing HTTP requests when certain events take place. This approach doesn’t scale very well though, and in any considerably sized application it is undoubtably preferable to have the sending of HTTP requests handled by a separate process.
We use a message queue, RabbitMQ specifically, for a number of mission critical tasks at FreshBooks, so using it for WebHooks was a pretty obvious choice. FreshBooks sends a message to RabbitMQ whenever something important happens in a FreshBooks account (incidentally, this is what makes the magic “ticker” and our FreshMap possible). In order to use the message queue for WebHooks, we developed a small Python application, called repeater, that listens to a specific queue and processes events. When an event is received, repeater checks for verified WebHooks that subscribe to that event and asynchronously dispatches an HTTP request. This means no waiting for application users, and we can always have multiple instances of repeater chewing through tasks in the message queue.
In order for third party applications to use our WebHooks implementation, we needed to make it possible for users to create, update and delete WebHooks through the API as well as the FreshBooks user interface. FreshBooks already has a robust API, so we extended this to support callbacks. Users can also access WebHooks from their account by clicking on “My Account”, “FreshBooks API” and then “Customize” under the WebHooks heading.
Additionally, the payload that repeater delivers includes a system field that identifies the FreshBooks account from which the event originated.
When a WebHook is first created, a unique verification code is sent to the specified URI. Somebody with control over the script residing at the URI must receive this verification code and verify the WebHook through the user interface or using the callback.verify API method. This gives us a level of certainty that the person creating the WebHook actually does control the URI they provided. Events will not be sent to the URI until the WebHook is verified. If you miss the verification code, it is possible to have it resent, again either through the user interface or using the callback.resendToken API method.
There’s definitely more we could do in the security area. Specifically we’ve tossed around the idea of signing requests using a shared secret, as well as adding in support for HTTP Basic Authentication.
When sending out HTTP requests, repeater checks the return code sent back by the HTTP server. 2xx and 3xx response codes indicate that the request was successfully received. If a 4xx or 5xx response code is received, the event is placed into a retry queue and we will make further attempts until finally giving up. At the moment we’ll continue trying for another 12 hours, but this is not a guarantee and is subject to change.
Paying attention to the HTTP response code does give us a chance to retry requests and gives users time to fix any issues that could be causing the problem (or in the case of downtime, allows time for the server to come back online).
We debated whether or not to require endpoint to use SSL. After some thought, we decided that requiring the use of SSL would be too much of an inconvenience to our users, and instead focused on making sure that no sensitive data is sent out by our WebHooks implementation. As a result, an entity id is sent in the request payload, instead of representation of the entity itself (i.e. we don’t send your entire invoice in the HTTP request when you update it, just the id).
The downside of this is that users must follow up to events using a query API method such as invoice.get or estimate.get to find out more information about the entity state. After consideration, we decided that this was an acceptable trade off versus requiring SSL.
As I mentioned earlier, FreshBooks has a robust API. API methods are mostly named for an entity and an action (i.e. invoice.create). We decided that instead of reinventing semantics to describe events, we would try to reuse these strings as much as possible. As a result, nearly all of the event types supported by our WebHooks implementation have a corresponding API request method. Experienced API users don’t have to learn new semantics and should be comfortable with the event descriptions.
A Few Months Later…
Overall, we’re very happy with our WebHooks implementation, and although we’ve identified areas where it can be improved, it works pretty well and it’s extremely cool to see what folks are doing with it. Hopefully by sharing some of the challenges we faced when implementing this functionality, we can help others who might be considering it!
As always, if you’re doing something really cool with our WebHooks implementation, or our API in general, we’d love to hear about it!