Why Internal Dogfooding Matters So Much
Dogfooding: A slang term used to reference a scenario in which a company uses its own product to demonstrate the quality and capabilities of the product.
I guess I’ve always been a little envious of companies like GitHub and 37Signals, not for the many obvious or usual reasons but because they get the joy everyday of the software they create and the software they use to create it being one and the same thing.
For certain products I feel like this actually goes a lot further than just dogfooding, every company knows they should use their own product. But in this case It’s recursive positive feedback, an upward spiral of greatness where you improve your product and this in turn makes the team better/happier/more efficient at improving your product. I can’t think of a situation more perfect for a newly born startup - where really understanding your customer and market is the most important factor.
At Sqwiggle we’re lucky enough to have this possibility, we’ve been distributed from the day we started working together and have been using the app as our primary form of communication from the moment it became viable. We have morning banter, long discussions, quick questions, rolling laughter and even more on Sqwiggle. Of course, we consistently come across small and big bugs, performance issues, user experience problems and interface improvements. Being the ones discovering the majority of these things also lets us more easily prioritise the pain that they cause.
We recently got to the bottom of a memory leak in Sqwiggle that happened only in the rarest of circumstances, but when it did occur the browser crashed spectacularly leaving no debugging info behind! Had we not been using the app as much this issue may have gone unnoticed or deprioritised for a long time (in fact it had come up in support as a random ‘white screen’).
Great beta testers and users will let you know of major issues and when they can’t understand how to use or find a feature - but there are a myriad of small user interface interactions, improvements and ideas can only be found by exhaustively using your own product all day every day.
Photo by Sh4rp_i
The Future
In February I left Buffer after working on every aspect of the app for over a year and a half (and documenting a lot of the interesting bits here along the way!). I’m immensely proud of everything that we achieved in that time, growing the service to half a million users and the company to 10 awesome team members whilst travelling! Buffer isn’t going away any time soon and I’m sure the companies future is very bright with Joel and Leo steering the ship - I can’t wait to see what they have in store.
So what now? I’m excited to say that I’m now building a new company to tackle the problems with remote working and distributed teams with a couple of awesome guys, Matt and Eric.
It’s a problem I’ve been thinking about ever since I stopped working in an office environment - how can you bring back the fun, spontaneity, and speed of communication of an office into the online world? It is a huge challenge, both technically and socially - we’re tackling the problem in a unique way and think we have a pretty great solution brewing…
If you have any thoughts on distributed teams then please go ahead and leave them in the comments, I’d love to hear any experiences or problems that you might be facing.
Photo by Sam Howzit
Hacker New
Around a year ago during my time in Hong Kong I started working on a Chrome extension for Hacker News. The aim of the extension was to make the site easier to use and add some (in my mind) much needed functionality.
Today i’ve released the latest version of the extension which addresses many of the concerns from the previous versions both on HN and Github, here’s a little tour of all it has to offer:
User Following
Hover over any username provides the option to “Follow” or “Unfollow” a user, followed users are highlighted wherever the appear, allowing you to pick out articles and comments from friends and thought leaders.
Instant Inline Replies
Previously replying to a comment meant opening a new page, inline replies mean you can type within the page and then submit when you’re finished. It also means that long replies won’t cause a timeout!
Retina Support
The extension replaces the existing image assets such as vote arrows and icons with higher resolution versions suitable for Retina displays. Thanks to Christian Genco for the idea.
Improved Readability
Hacker New increases the size and colour contrast between links and surrounding elements. It also makes it clearer which articles you have already clicked on.
Quick Profiles
Hover over a username anywhere on the site to instantly see a users Karma and affiliated websites / social profiles. This is a useful indicator of how much respect they have gathered the HN community.
Filters
Filtering allows you to hide articles, users and domains from HN that you would rather not see. I have my filters setup to filter a variety of negative keywords and trashy domains!
Endless Scrolling
The frontpage just got longer, infinitely longer in fact. Endless scrolling kicks in before you reach the bottom so the news keeps flowing. The footer and search bar are always available by hovering near the bottom of the screen.
Collapsible Threads
If you just want to immediately skip past a thread, you can now collapse the entire thing on the right hand side and get directly to the next comment.
Social sharing
Social sharing buttons are built in for Twitter, Facebook, Google+, Buffer. Of course they don’t load until you request it so there is no extra bloat on page load times.
I hope the extension improves your HN experience as much as it does mine. If you think something can be done better or would you prefer to tweak the styling go ahead and fork the project over on Github!
There Are Always More Users
Something I hear often when talking to people and also find myself thinking occasionally is that there are only so many users that your startup can acquire… isn’t there a market limit? won’t they eventually stop signing up? If I release a feature too soon and some users drop off how will I ever win them back?
But the truth is, unless you have the most incredibly niche product (think time-tracking software for ostrich farmers with a penchant for gymnastics) there really are always so many potential users out there that by the time you get anywhere near saturating the market your product will likely be completely different!
Think of the biggest sites and apps you know like Facebook, Evernote and Twitter, they have millions (perhaps billions!) of users and yet still every single minute someone, somewhere that has never used the service before signs up for the first time.
Don’t be afraid to ship early, if the first impression isn’t perfect and users don’t sign up or quit your app then you can treat it as valuable learning, iterate and try again - there are always more users!
Photo by James Cridland
Decisions, Decisions
Recently I was talking to a friend about decision making, she was stuck in a paralysing situation with the knowledge that whatever option she chose it would likely become a major turning point in her life. Mulling over the possibilities for months, weighing the pros and the cons and trying out both options had only left her wracked with the thought that whichever decision she made would ultimately leave her regretting the lost opportunities of the other.
The thing is, seemingly innocuous and unrelated actions like where you sit on a bus or who you talk to at a party also have the same potential to completely change your life or the path of your business. Of course these dots can only ever be connected in hindsight and we don’t worry ourselves over the hundreds of decisions we make every day.
It’s a problem that entrepreneurs and business owners face constantly, in fact you have to make so many different decisions that the real defining ones often get lost in the noise… Could that feature choice be costing you thousands of dollars a day or your last hire invent the killer feature that rockets your company to success?! All we can do is make the best decision possible with the information we have available at that moment.
So how do you make the best decision where everything else is even? Perhaps the answer is to minimize the possibility of regret. Not by focusing on the negatives but by choosing the option with the greatest potential payoff and bet on your own ability to achieve it.
And the friend? She’s still deciding…
An Amazing Way to Get Customer Feedback and Build a Community
So you’re building a new product or feature and want to gather feedback from your users… If you are building a startup this is probably a situation you have been in many times before and there are tons of ways you could go about doing this. An email list, forums or customer support software like Get Satisfaction / UserVoice are popular ways to gather feedback
At Buffer we found ourselves wondering if there was a better way, where the whole team could have a proper conversation with users and gather feedback incredibly fast as we rolled out new features and changes every few days.
We decided to do something a little different and use Facebook groups. We created a group for each of the three parts of our product that we are most actively working on (Web, iOS, Android) and invited anyone that wanted to join and help test the next versions of the apps in return for early access weeks or months before anyone else. Hundreds of people joined after just a few tweets on the company account - It’s hard to resist the lure of a beta test!
The Benefits
Easy and personal
At this point most people that are likely to want to test your product have a Facebook account and feel very comfortable with the interface and idea of groups. This means to get involved they only have to click one button with no extra sign in or learning curve.
Notifications
When you post an update to a group this usually shows up in members notifications and stream, it doesn’t clutter up their inbox but makes sure that members are always up to date with what’s going on. If the group isn’t secret (I would recommend a public group) then when people join their friends may also see this in the stream which spreads the word even further, maybe even introducing new people to your product.
Speed
Facebook is quick, very quick - we’ve seen comments come in just minutes after a new release has been pushed to testers. Members often see the notification on the web or their mobile phone and check out changes immediately!
Community
But the best aspect has to be the sense of community that is created, group members chat, help each other out and like posts that they agree with. We’ve run polls, asked questions and given advice - the whole team jumps in to help people out and comment on bug reports and feature requests.
It’s great having everyone’s face and real name, there is an amazing feeling that we are getting to know our users better and i’m sure they feel the same way about talking directly to the team and seeing that they are directly influencing the product. I’m confident that those in the groups will become some of our most loyal and outspoken users.
The Drawbacks
Of course, after using groups for several months we have found some drawbacks. There will always be a couple of people that don’t have or want a Facebook account, even for a social startup like ours!
Deploying features
There is also some work in making sure that everyone in the group gets the new features, whether this is enabling them in our admin area, sending out builds or manually adding people to TestFlight. Perhaps in the future this is something we could automate using the Facebook API but at the moment when bug reports come in it can be difficult to know which build someone is using or what version of the website they are seeing.
Over communicate
Ordering of posts in the group is automated, meaning that old posts often get pulled up above newer posts. Posts can be pinned so they stay at the top of the group but you definitely have to over-communicate and assume that previous posts may not have been read.
Conclusion
Overall in the last few months we have found Facebook groups to be an amazing way to connect with our most passionate users no matter where they are located. We’ve also collected some amazing feedback which has helped to shape the next version of Buffer with is launching very soon.
If you don’t currently do anything like this at your startup - give it a try! Start a group, invite your most enthusiastic users to join in and leave a link in the comments :-)
Sign Up or Sign In, Does it Even Matter?
The hurdle. It’s a metaphor!
The Idea
Like any startup, at Buffer we are always looking for ways to reduce friction at every stage of using the product, from the moment you land on the homepage to the daily use of our browser extensions and web app.
One of the key points in the user lifecycle is getting people over the hurdle of signing up for your app, of course this should be as seamless as possible, particularly in the consumer internet space. In the last few years this has become easier than ever - with the advent of OAuth solutions such as Facebook Connect and Twitter Sign In users often no longer need to enter any details at all to sign up for your service.
This got us thinking could we completely remove the barrier of signing up all together? Could we make the account creation experience so easy that you might not even notice it had happened?
The Approach
While there are some intricacies to handling a combined signup / signin on the server the first step is authorising the user with their social network of choice - and of course the UI is simpler than ever. We decided on a familiar ‘stack’ of social networking buttons that can be consistently reproduced in all of the different sign in locations we have (web, browser extension, mobile).
Our sign in screen, as seen on mobile.
On the web app the stack sits on the right hand side of our homepage and acts as a bright, strong call to action to. As more networks are added in the future we will likely add a “more options” button combined with a memory of the users last sign in choice which will always be displayed at the top of the stack.
Choosing the wording ‘Sign In’ on the buttons rather than ‘connect’ or other alternative wording hopefully gives the subtle impression that even new users already have an account and they just need to sign in to access it. The key is with ‘Sign In’ you don’t expect to have to do any more work, which makes the hurdle smaller.
Key Lessons
Users are often looking for the words “sign in”, “sign up” or “log in”
Even (or perhaps especially) with a combined signup / sign in it is important to make it very clear that the user can do both actions with the same buttons. We included a friendly note on every entry point to get this message across
Let people know that they can do both actions in the same place.
Many people forget which social network they used previously
This is one of the biggest problems we had after rolling out the changes, many users were forgetting which social network they had used to login and were creating two, three and even more accounts. Obviously this is a terrible user experience and it can also completely skew your metrics.
We dealt with this situation in a couple of ways; by storing a separate long-life cookie whenever a user signs in we can check on signup if a user has been to Buffer before - we catch people that try to sign up with the cookie and display a screen like this:
Detect and give users the option to sign in again.
Inevitably some users still fall through the cracks, perhaps because they clear their cookies or simply click on ‘create account’ anyway. In these cases we need to let them merge their accounts together.
In Buffer we allow users to connect additional social media accounts on the dashboard, when the feature was first launched connecting an account already in the system would throw an error with a message to contact support - this would affect anyone who had accidentally created two accounts! As you can imagine, merging accounts quickly became one of our biggest support requests.
Thankfully we have now completely removed this friction point also, as authorising with the social network is proof of ownership connecting an account that is already in the database simply merges the two together, seamlessly in the background.
What about their email?
The disadvantage of users signing up with a social network is that you can’t always get their email address and other details that may be important for your business. At Buffer we use the email to send alerts if there are any problems or if your Buffer becomes empty as well as optional newsletters and lifecycle triggered retention emails.
The passive email banner
Rather than immediately hassle the user after signing in with a social network (that kind of defeats the point, doesn’t it?) We opted to display a more passive banner at the top the dashboard which lets the user know the reason we would like to have their email address, this works well and only displays after a welcome tour which means the user is much more invested in the product than they would have been if asked on a signup page.
Conclusion
So, does it matter? I would say that with the right measures in place to catch and automatically repair duplicate accounts it doesn’t matter at all and in fact reducing complexity of the sign up process is well worth these measures.
It is now very rare that we get any of the support issues above and because the nature of Buffer is managing and connecting many social accounts we were probably more inclined to get these types of problems than many apps would be.
Have you implemented social sign in differently or had interesting problems with it in the past? I’d love to hear your solutions and thoughts in the comments
Photo by nickuzma
How We Increased Our Conversion to Paying Users by 171%
At Buffer we are currently focused on driving growth to a new level, in the process we are trying many different channels to increase the number of visitors and signups including content marketing, web stores, mobile markets and more.
In the past week I decided to take a quick look at our homepage and how we might improve our conversion instead of driving more new visitors - after all, if we can improve the conversion whilst still increasing visitors it has a compound affect!
Previously we have not had much luck with A/B testing, it’s something we often do and have infact built an internal library called ‘Einstein’ which makes it super easy for anyone on the team to deploy a one-line split test anywhere in the product. We have continuously been amazed at how seemingly large changes can actually make very little difference to conversion and activation, particularly in the early days of the product (more on this later).
Examples of Some Failed A/B Tests
Homepage Tagline Changes
This is an area we predicted would have a big effect, the thinking being that the main headline on the landing page would be a driving factor in whether people are interested in the value (and we all know not many users read further than the headline!).
In the early days Joel tested between several variations of text including “Be awesome on Twitter”, “Get 200% more clicks on Tweets” and “A Smarter Way to Share”. Unfortunately none of these variations had a significant impact on conversion.
Colour of Call to Action
After meeting Tim Ferris, he suggested that we try a higher contrast, orange call to action. At the time we had a single signup button that was a dark shade of green. The difference was similar to below (although unfortunately we don’t still have the exact screenshots).
The two variations used, with a bright central call to action.
Positioning Changes
We also tried moving the main call to action on the homepage to see if this would have any impact, flipping the page horizontally as well as trying a stark version with no introductory video and a large central signup button! The affect? Negligible.
Interstitials vs Dropdowns
We have also done split tests on internal product features and design variations, one in particular showed 50% of new users an interstitial to install our browser extension whilst the other 50% saw a drop down bar once they began using the product. The hope here was to increase the activity of new users by ensuring they installed the extension, but after running the test for several weeks the cohort analysis showed no discernable difference.
The First Successful A/B Test
In the past week we ran a new A/B test on the homepage that shows potential users more detail about Buffer, in particular some of our most popular features run directly below the signup box (multiple accounts, analytics and team members). The hypothesis was that potential users are now discovering Buffer through more channels than just our content marketing efforts and a lot of people might not understand how much Buffer can offer when they first land on the page.
The two variations used, original on the left and the variation on the right.
We are now tracking all of the tests we perform with Kissmetrics which enables us to easily produce reports for our entire funnel, which currently looks like:
The results (and the trigger for this post!) seem to be for the first time both very positive and statistically significant although entirely unexpected. Every point in the funnel proved to be outperforming the original by multiple percentage points, but in particular upgrades were up a whopping 171%.
This outcome was totally unexpected and also shows the importance of tracking your split tests at a macro level, and not just whether there were ‘more clicks on the button’. In this case we would have seen that the increase in signups was statistically insignificant and could have even reverted back to the old layout!
Conclusions
It Takes Many Tries
A/B Testing is difficult! It could be easy to conclude from the many posts highlighting successful A/B tests that getting clear cut results is simple and you too can double your signups by changing the colour of a button but for us that wasn’t the case.
So far we’ve found that getting statistically significant results tends to take many iterations and much bigger changes than you might imagine, I don’t doubt we will need to perform many more tests to get another significant result of this type.
Where Your Visitors Come From Makes a Difference
Many of the unsuccessful tests listed above were performed in the first eight months of Buffer’s life, at this point in time our inbound traffic was much smaller and the traffic we did have often came from early adopters and detailed blog posts explaining how to use Buffer in great detail.
I am sure this will have had an affect on the results as a larger percentage of these users will have hit the homepage knowing what Buffer is and knowing they wanted to try it out. This may go someway to explaining why many of the tests in the early days displayed little variation at all.
Have you had a run away split test success or perhaps you’ve tried hundreds of variations with no results - I’d love to hear your stories in the comments!
Resources
- KISSmetrics - Particularly tailored to enable easy tracking of SAAS metrics.
- A/B Significance Test - Handy tool to measure if your results are statistically significant.
- How not to run an A/B test - Why you shouldn’t peak at your results early.
- A Practical Guide to Controlled Experiments on the Web - The techniques used for experiments at Amazon, Microsoft and NASA.
How to Use Notifications and Alerts to Give Your Startup an Edge
She’s alert, get it?
Recently I’ve come to realise what a critical role notifications and alerts can play in running an internet startup, enabling a small and scrappy team to stay on top of many things at once. By spending a small amount of time identifying and setting up alerts for key metrics you can give yourself a powerful weapon that can be used to minimise problems and wow customers.
At Buffer we started with the very basics, using pingdom to send a text message if the website became unavailable - this works great for an MVP. Over the past year as the service has grown we have gradually built up and expanded the notifications to cover many more areas of the architecture and business to make sure we always know what is going on.
In my mind notifications are best split into two categories - active and passive. Active notifications are pushed through to email, sms or mobile apps and are used for emergencies or critical problems with the service. Passive notifications are displayed in our internal admin area, team chat or are sent in daily summary emails, these are things that we would ideally action but aren’t immediate emergencies.
The key notifications that we are using today include:
Active Alerts
Macro business metrics
With deploying code many times a day there is alway a chance that bugs will slip through unnoticed. It’s important that if this happens they are detected as fast as possible and don’t affect the things that really matter. We track the core metrics that are most important to our business every minute to ensure they are within preset acceptable bounds. Obviously these will be different for every business, for Buffer these are:
- Updates Sent
- API requests
- Signups
- Upgrades
If any one of these metrics steps outside of pre-defined bounds then we are immediately notified via push alerts and email, for example if for two consecutive minutes no updates are sent then there is clearly a problem somewhere in the chain that needs immediate attention.
Brand sentiment
Inspired by an interview with Chuck Rossi of Facebook’s release engineering team we also track sentiment (currently only on Twitter) for people mentioning @bufferapp. If there is a spike in negative tweets we are immediately notified that something may be wrong. This means we can jump into Twitter and both fix whatever the issue may be and also respond with very fast customer support when it’s needed most.
Server status
As an internet based business this is a bit of a no-brainer, but mentioned for the sake of completeness. We use Server Density to perform minutely checks on all of our public facing services including the website, embeddable buttons and API. These are done in a round-robin fashion from multiple data centres around the world which means a more accurate average response time reading and ensures we are accessible from different regions of the internet.
Passive Alerts
Upgrades and downgrades
We display a list of the days upgrades and downgrades in the admin area which is visible to everyone on the team. We often reach out individually to users to thank them or in the case of a downgrade to try and trigger a discussion on why they left us and what we might have done differently to keep them..
Influencer signups
Obviously we would love to send a personal email to every new signup, but with this not being possible we thought it would still be nice to get in personal contact with influencers that signup for the service. At first this was an active notification, we setup the signup system so that Joel would receive an individual email every time a user with a large number of followers signed up. This quickly became a burden and the emails we’re auto archived even after the trigger for an influencer was raised time and again.
In hindsight it is now obvious that this should have been a passive alert and in a similar fashion to upgrades we now display a list in the admin area of the days influencers and regularly get in contact with users to welcome them personally - this really helps to create the customer service wow factor.
Brand mentions
Like many businesses and individuals we use Google Alerts to inform us in a daily summary where Buffer is being talked about on the web. This lets us find posts that mention the service both in negative and positive ways. Often Leo can jump into the comments to directly answer questions and provide support.
Commits and deployments
For the past few months we have used HipChat as the hub of day to day conversation at Buffer. In general HipChat has worked wonders at keeping the team motivated and upto date on what everyone else is working on. As part of this we push passive deployment and commit notifications into HipChat from Github, usually no action is required by anyone but this helps to keep the sense that other people are working and everything is always moving forward fast!
I hope some of these examples inspire you to think about how having more notifications and information at your disposal could give your startup an edge!
What have I missed? Is there something you monitor at your business that has proved incredibly valuable? I’d love to hear how we might improve our approach or any aspects that we may be neglecting
Image thanks to rsteup
Scaling a Lean Startup, The Story So Far.
Since I joined Buffer in September last year we have seen awesome growth that has pushed both our hardware and mine and Joel’s sys-admin skills to their limits. In this time we have grown from a single Linode VPS shared with Joel’s previous startup to a heady total of 10 VPS’s and in the last week we undertook the biggest jump so far, moving our infrastructure from this cluster to Amazon Web Services.
Our approach to scaling has always been based on necessity - with the Lean Startup methodology ingrained into the company culture we push code fast and try to avoid prematurely scaling our infrastructure beyond it’s needs. This has definitely lead to some sleepless nights and the occasional hour of downtime but all in all we feel this approach is much better than over compensating from the beginning and building architecture that you may never need or use. These are a few of the key occasions when scaling became the only option…
1 Server: Launch to 20,000 users
The first version of Buffer was (perhaps famously) the embodiment of an MVP, a single pricing page that gauged potential interest in the idea before anything was even built. This was hosted on a tiny Linode VPS, shared with several other projects. Once the idea was validated and for the next year Joel was able to scale Buffer to 20,000 users by increasing the size of the VPS and optimising SQL queries, this meant a few hours of downtime each time the server was resized but allowed costs and time investment to remain low while product market fit was reached, or in Joel’s words:
Focus on “scaling” too early and you may well forget to focus on “building something people want”. Don’t make that mistake.
2 Servers: Moving to MongoDB
As I joined Buffer the first feature we decided to build was support for posting to Facebook - the main decision was how to change the code and database schema to accommodate more than just tweets. Over the course of several weeks the model layer was rewritten and the data migrated table by table from MySQL to collections in the document-based datastore MongoDB.
After the switchover, the new database was kept on the existing server, taking the place of MySQL. This worked well for around a month until early one Sunday morning Buffer suddenly disappeared from the internet. After tailing the error logs we quickly realised that we had reached an inherent limitation of MongoDB - 32-bit processes are limited to ~2gb of data and we had hit that limit.
We searched frantically for a way to get the database back in action on the existing hardware. After half an hour with no luck and Leo calming distressed users on twitter we realised we had to get in touch with the best person we knew. We called Eric Lubow from SimpleReach who put us on the right path in minutes - there was no other option but to accept the downtime and spin up another 64-bit server immediately. Thanks to clear, straight forward advice we got off very lightly with less than two hours of disruption in a quiet period as we setup a server, tarballed the existing data directory and shuttled it across!
4 Servers: Background Processes
By November 2011, Buffer had continued to grow to around 70,000 users and with 20,000 tweets per day being sent the website and browser extension were becoming noticeably slower at the peak time of the day (for us this is around 17:00 GMT, early evening in the UK and morning on the East Coast). At this point it was time to queue the sending tasks and have them processed by their own dedicated server. In hindsight this was the turning point where a Buffer ‘architecture’ started to emerge, with modular pieces of code and servers that we’re independent of each other.
As soon as the core task of sending tweets was moved to queues it became obvious that a lot of other processing could be completed this way such as gathering link info, analytics and image processing. We went ahead and moved these tasks to two new utility servers, I’ll be covering the details behind how we now use queues extensively at Buffer in a future post.
5 Servers: Upgrading MongoDB
In December our database server was humming along nicely but quickly filling up it’s available RAM headroom and running on an out of date version of MongoDB. In order to upgrade the software and the server to more RAM (as we would end up doing several times of the next few months) we needed to be in a replica set configuration that would allow us to take the individual database servers out of circulation one by one to upgrade them.
This proved to be another rocky period with several hours of downtime, we found this was almost entirely caused by issues in the immature PHP drivers which have since been addressed but also partly down to our own lack of experience managing replica sets, firewalls and all of the intricacies of a multi-server setup.
7 Servers: DiggDigg Acquisition
In January of this year we hit the 100,000 user milestone and completed our acquisition of the popular WordPress plugin DiggDigg which allows any WordPress user to quickly add a sharing bar to their blog or website. Part of the plan behind this move was to add the Buffer Button to DiggDigg and have it enabled by default for new installations, adding massively to the button’s reach and Buffer branding around the web.
In order to handle the millions of new button impressions and isolate the web interface from load generated by third party websites it was time to move the button to it’s own servers and introduce our first load balancer. It was also decided to use Nginx AND just for good measure re-write the entire button to become independent of the Buffer framework.
We definitely teetered on the edge of our seats after deploying all of this previously unused architecture at once, waiting for the load from DiggDigg to hit the servers. But everything went smoothly, the load balancer acted as expected and impressions gradually rose to new highs as people upgraded their plugin (it seems there is no sudden rush of traffic!). The button servers now handle around 4 million impressions per day and rising!
9 Servers: Sleepless Nights
Last month we reached 180,000 registered users and completed the acquisition of ShareFeed from KISSmetrics. Almost unbelievably, at this time the main website, API, blog and browser extension were all still running from a single Linode VPS (albeit one that had been upgraded on many occasions!). During peak times things we’re beginning to feel slow again and complaints started to appear on Twitter at predictable times.
Unfortunately late evening Hong Kong time (where we are currently based) is the peak time for traffic at Buffer. For almost a week I had incredibly restless sleep, I became acutely aware of every email received, waking up at the slightest sound convinced that the site would be down. It was definitely time to upgrade.
Thanks to all we had learnt creating the new environment for the Buttons several months before, we were able to make this move completely seamlessly. By setting up the load balancer, adding the new servers and testing thoroughly before moving the DNS we were able to ensure that the only thing users noticed was a performance increase.
Thats Not All Folks
It’s been an amazing ride so far with both highs and some serious lows, at one point working for nearly 30 hours straight just to keep the site online. We have learnt an incredible amount as a team in the last few months and I can’t think of any way we could have learnt more, faster.
Last week we migrated the whole of Buffer’s architecture to Amazon Web Services with less than two hours of scheduled downtime, making a number of huge changes and improvements in the process. I’ll cover some details of this migration in a future post.
I’d love to hear any comments your have or similar stories of battling scaling woes as a small startup!
