Imagine waking up to hundreds of unfulfilled customer orders because, due to an unforeseen systems failure, payments failed to get processed.
These orders are sitting in your system — caught in limbo — unable to go through to the warehouse for fulfillment.
And this happened even though your team did absolutely nothing wrong…
That’s what happened one morning while I was VP of Operations for FinalStraw, the fast-growing e-commerce company behind the foldable staw.
So what do you do when Shopify almost breaks your e-commerce store before you’ve had your 3rd coffee?
For the record, Shopify’s team didn’t do anything wrong either. Sometimes software just breaks. It happens to the best of us.
This isn’t a post about complaining at a great platform— it’s a story about what can be done when one of your systems goes haywire. Because when something like this happens to you— and if you’re in e-commerce operations long enough, it will— and you don’t have a system in place, the cost can be deadly.
Let’s go back to the morning it all went wrong…
It was around 10:30. I was going through the morning reports, checking fulfillments from the previous day and sales channels against inventory.
That’s when customer service noticed that we had a bunch of orders just sitting in our system. The system hadn’t collected payment. Which meant that there were no orders going through to the warehouse.
And the whole problem had been going on since about midnight.
An OK day suddenly turned into a frantic scramble. (Not an unusual thing in e-com ops.)
Our team dug into the problem. First we freaked out at our web developers (sorry Brooks & Adam, we jumped to conclusions). After an initial analysis, it looked like Shopify was authorizing but not charging our customers’ credit cards. On the customer side it looked like orders were going through, and they were expecting their orders. But the payment wasn’t being captured. Unpaid orders don’t flow to fulfillment. So they were just sitting there. Stuck. Sad. And the list was growing.
That’s when I had to move the entire customer service team and my ops team over to manually capturing credit cards so we could get orders to flow to the warehouse to get fulfilled. All. Day. Long.
By the end of the day, we’d processed about 500 orders. Manually.
Here’s what to do so you don’t have to spend your day entering 500 credit card details
Even the best software breaks. Sometimes the servers go down. Or AWS has an outage. Or a black swan event kills connections.
But when you work in ops, your job is to have a system to minimize the damage. And then have a second system in case the first one breaks. And a back-up for your back-up.
Let’s face it. Ops makes e-commerce work. Without our systems, the invisible magic that drives fulfillment wouldn’t happen.
So the answer to catching system breaks early?
Setting up a notification system.
Our development team wrote some custom code to set up an “if this, then that” notification system. Every time Shopify was down or payments didn’t get collected, I got a slack notification. We also signed up for Shopify’s text alert system so if the website is down, we get notified.
If we’d had alerts set up that day, the problem would have been caught much earlier. We wouldn’t have had to redirect resources to manual data entry for a whole day.
Ultimately, we were lucky it was only 500 orders. Imagine this happening on a day after a huge marketing promo, a new color release or a Shark Tank feature. Imagine inputting thousands of orders while sitting on the phone with the warehouse, telling them it’s coming…
That would not be a good day.
And setting up the right alerts can keep those kinds of days away.
