Recently I’ve come to realise what a critical role notifications and alerts can play in running an internet startup, enabling a small and scrappy team to stay on top of many things at once. By spending a small amount of time identifying and setting up alerts for key metrics you can give yourself a powerful weapon that can be used to minimise problems and wow customers.
At Buffer we started with the very basics, using pingdom to send a text message if the website became unavailable - this works great for an MVP. Over the past year as the service has grown we have gradually built up and expanded the notifications to cover many more areas of the architecture and business to make sure we always know what is going on.
In my mind notifications are best split into two categories - active and passive. Active notifications are pushed through to email, sms or mobile apps and are used for emergencies or critical problems with the service. Passive notifications are displayed in our internal admin area, team chat or are sent in daily summary emails, these are things that we would ideally action but aren’t immediate emergencies.
The key notifications that we are using today include:
Macro business metrics
With deploying code many times a day there is alway a chance that bugs will slip through unnoticed. It’s important that if this happens they are detected as fast as possible and don’t affect the things that really matter. We track the core metrics that are most important to our business every minute to ensure they are within preset acceptable bounds. Obviously these will be different for every business, for Buffer these are:
- Updates Sent
- API requests
If any one of these metrics steps outside of pre-defined bounds then we are immediately notified via push alerts and email, for example if for two consecutive minutes no updates are sent then there is clearly a problem somewhere in the chain that needs immediate attention.
Inspired by an interview with Chuck Rossi of Facebook’s release engineering team we also track sentiment (currently only on Twitter) for people mentioning @bufferapp. If there is a spike in negative tweets we are immediately notified that something may be wrong. This means we can jump into Twitter and both fix whatever the issue may be and also respond with very fast customer support when it’s needed most.
As an internet based business this is a bit of a no-brainer, but mentioned for the sake of completeness. We use Server Density to perform minutely checks on all of our public facing services including the website, embeddable buttons and API. These are done in a round-robin fashion from multiple data centres around the world which means a more accurate average response time reading and ensures we are accessible from different regions of the internet.
Upgrades and downgrades
We display a list of the days upgrades and downgrades in the admin area which is visible to everyone on the team. We often reach out individually to users to thank them or in the case of a downgrade to try and trigger a discussion on why they left us and what we might have done differently to keep them..
Obviously we would love to send a personal email to every new signup, but with this not being possible we thought it would still be nice to get in personal contact with influencers that signup for the service. At first this was an active notification, we setup the signup system so that Joel would receive an individual email every time a user with a large number of followers signed up. This quickly became a burden and the emails we’re auto archived even after the trigger for an influencer was raised time and again.
In hindsight it is now obvious that this should have been a passive alert and in a similar fashion to upgrades we now display a list in the admin area of the days influencers and regularly get in contact with users to welcome them personally - this really helps to create the customer service wow factor.
Like many businesses and individuals we use Google Alerts to inform us in a daily summary where Buffer is being talked about on the web. This lets us find posts that mention the service both in negative and positive ways. Often Leo can jump into the comments to directly answer questions and provide support.
Commits and deployments
For the past few months we have used HipChat as the hub of day to day conversation at Buffer. In general HipChat has worked wonders at keeping the team motivated and upto date on what everyone else is working on. As part of this we push passive deployment and commit notifications into HipChat from Github, usually no action is required by anyone but this helps to keep the sense that other people are working and everything is always moving forward fast!
I hope some of these examples inspire you to think about how having more notifications and information at your disposal could give your startup an edge!
What have I missed? Is there something you monitor at your business that has proved incredibly valuable? I’d love to hear how we might improve our approach or any aspects that we may be neglecting
Image thanks to rsteup