Oof. Yesterday was one of those days.
We love absolute transparency at Condiment, whether it’s showing you the quote from the printer to show you exactly how much your print will cost (we never mark-up print, just charge a small print management fee) or telling you the services we use for our online projects.
Hiding stuff from clients will always backfire. Maybe you’ll make a few quid out of them before they find out you’ve been marking up their print by 30% or something extortionate. But when they do discover what you’ve been up to, they’ll come down on you like a ton of bricks and never work with you again.
That isn’t a smart way to do business.
Instead, we rely on our own skills – as a creative agency – to make our money, and take great pride in the suppliers we work with. We trust them, pay them well for their services, and let them do what they do best.
We recently moved some of our higher-traffic sites over to PHP Fog, a new kind of cloud hosting with added Git goodness. Cloud hosting basically means we get a cost-effective, reliable service and can deploy new sites and apps far quicker than usual. It also means we can spend less time managing servers, and more time creating awesome websites and online tools for our clients. Cloud hosting also ensures that whenever the load gets high on our servers, the impact can be spread out, meaning a lower chance of failure.
We also use similar services for our First 65 apps, but specifically Rails-based providers such as Brightbox and Heroku.
However, yesterday our server (hosted with Amazon Web Services) threw an almighty hardware wobbler, and took two of our client sites out of action for the best part of Saturday. Hardware failures of this type are rare (it’s the first time we’ve ever experienced one), but do still happen, and cause even greater disruption on a weekend. After eight hours of engineers working hard to restore service, we were back online and running smoothly.
To a certain extent, there was nothing we could do but wait. We were in constant contact with the engineers and updated any affected clients as and when we had more information.
While it’s difficult to completely protect against things like this happening, we’ll be taking steps to ensure that our servers are monitored more often, and that there will be contingencies in place should something like this happen again.
One option, of course, is to move hosting providers; to simply declare PHP Fog to be worthless and unreliable. But move to whom? The list of hosting providers is endless, and flitting between them like Roman Abramovich in the hope the next one will never let you down is pointless. Much better is to work with suppliers, give honest feedback and help them to provide a cracking service.
So we’ll be sticking with the team at PHP Fog, just as we stuck with 34SP after they experienced a DDoS attack, and Media Temple after some tricky DNS issues on the day Up To The Moon went live.
We’re incredibly sorry to the clients that were affected by the outage, and we’ll get in touch with you individually to explain the measures we’ll be taking, one of which you’ll see below, with the introduction of New Relic monitoring on our PHPFog accounts. It will alert us to errors on our sites, and keep us up-to-date on anything we should be aware of when it comes to site performance.
