Imagine waking up to hundreds of unfulfilled customer orders because, due to an unforeseen systems failure, payments failed to get processed.
These orders are sitting in your system — caught in limbo — unable to go through to the warehouse for fulfillment.
And this happened even though your team did absolutely nothing wrong…
That’s what happened one morning while I was VP of Operations for FinalStraw, the fast-growing e-commerce company behind the foldable staw.
For the record, Shopify’s team didn’t do anything wrong either. Sometimes software just breaks. It happens to the best of us.
This isn’t a post about complaining at a great platform— it’s a story about what can be done when one of your systems goes haywire. Because when something like this happens to you— and if you’re in e-commerce operations long enough, it will— and you don’t have a system in place, the cost can be deadly.
Let’s go back to the morning it all went wrong…
It was around 10:30. I was going through the morning reports, checking fulfillments from the previous day and sales channels against inventory.
That’s when customer service noticed that we had a bunch of orders just sitting in our system. The system hadn’t collected payment. Which meant that there were no orders going through to the warehouse.
And the whole problem had been going on since about midnight.
An OK day suddenly turned into a frantic scramble. (Not an unusual thing in e-com ops.)
Our team dug into the problem. First we freaked out at our web developers (sorry Brooks & Adam, we jumped to conclusions). After an initial analysis, it looked like Shopify was authorizing but not charging our customers’ credit cards. On the customer side it looked like orders were going through, and they were expecting their orders. But the payment wasn’t being captured. Unpaid orders don’t flow to fulfillment. So they were just sitting there. Stuck. Sad. And the list was growing.
That’s when I had to move the entire customer service team and my ops team over to manually capturing credit cards so we could get orders to flow to the warehouse to get fulfilled. All. Day. Long.
By the end of the day, we’d processed about 500 orders. Manually.
Even the best software breaks. Sometimes the servers go down. Or AWS has an outage. Or a black swan event kills connections.
But when you work in ops, your job is to have a system to minimize the damage. And then have a second system in case the first one breaks. And a back-up for your back-up.
Let’s face it. Ops makes e-commerce work. Without our systems, the invisible magic that drives fulfillment wouldn’t happen.
So the answer to catching system breaks early?
Setting up a notification system.
Our development team wrote some custom code to set up an “if this, then that” notification system. Every time Shopify was down or payments didn’t get collected, I got a slack notification. We also signed up for Shopify’s text alert system so if the website is down, we get notified.
If we’d had alerts set up that day, the problem would have been caught much earlier. We wouldn’t have had to redirect resources to manual data entry for a whole day.
Ultimately, we were lucky it was only 500 orders. Imagine this happening on a day after a huge marketing promo, a new color release or a Shark Tank feature. Imagine inputting thousands of orders while sitting on the phone with the warehouse, telling them it’s coming…
That would not be a good day.
And setting up the right alerts can keep those kinds of days away.
Kathleen Sullivan Garman has been an Ecommerce Operations Leader for 20+ years, helping dozens of companies set up the right backend processes. She’s currently running ecommerce operations at Remaker Labs (the masterminds behind CarryHitch).
As Pipe17’s Ecommerce Operations Advisor, Kathleen provides valuable input to our product team and advises customers on best practices. Kathleen is an avid snow skier and scuba diver and would rather be under the ocean than above it. She volunteers as the executive director of Mended Hearts of Spokane, a cardiac charity.