Every release took the business offline. Now it runs 24x7 at 60% the cost.

The situation

An SMS marketing platform had reached the point where its architecture was setting the pace of the business. Every update meant shutting down the whole platform. For a service that sends texts and routes revenue-generating replies, that pause hit every client at once, and their revenue with it.

The goal was simple to state: deploy at any time of day without taking the platform down, and without users noticing or message delivery being affected.

The product manager knew the outages were wrong. What they couldn’t do was explain why. Without a cause there was no estimate, no place on the roadmap, and no way to judge whether an engineering answer was real.

The engagement started as two fixed-price projects, one to diagnose the platform and one to design the fix, so the company could find out what it was dealing with before committing to anything ongoing.

Diagnosis

The first project was an architectural review: how the platform was built, what it depended on, and where it would fail under load, all framed by the one business constraint that needed fixing.

It found three problems:

• An in-memory message queue. The queue lived inside the application server, so the platform could only grow by buying a bigger server, and the queue had to drain completely before every deployment. That was the planned outage.
• Manual deployments and hand-patched servers. Patching production under pressure is error prone, and fixes made there were easy to forget to carry back into the main code.
• Queue consumers inside the monolith. Different kinds of messages arrive at very different rates, but every consumer had to scale together.

The finding that surprised the business was a consequence of the first problem that had never been surfaced. Because the queue was in memory, only one server could run at a time. If that server crashed, restarted or ran out of resources, every message waiting to be sent was lost, and nobody would know it had happened. The planned outage was the visible symptom. Silent data loss was the real risk, and it would have cost customer trust far faster than a maintenance window.

The approach

The second project designed the new architecture and the order to build it in. I put silent data loss at the top of that order. Preventing lost messages wasn’t the brief, but fixing it also removed the reason for the planned outage.

1. Protect the work first. Replace the in-memory queue with a distributed one, so messages survive a server failure and more than one server can run. Growing a platform that still loses work just loses it faster.
2. Then scale each part on its own. Move the queue consumers out of the monolith so each type of message gets its own capacity, added and removed as its demand rises and falls. A spike in one kind of message no longer slows the rest.
3. Automate deployment. Move to automated deployments and infrastructure as code, so releases are repeatable and capacity can scale up and down without a person needing to do it by hand.
4. Respect the carriers’ limits. Add controls that keep sending rates within what the carriers and texting providers allow. Push volume past those limits and a growth fix becomes an outage.

Just as deliberate was what I chose not to fix. The existing data store was doing its job, and replacing it would have solved nothing the business was asking for. The instinct is to fix everything you can see, and it’s usually wrong: size the work for where the company is now, and leave the wrong-but-working parts alone until they start costing you - as the dynamics can change and you may be fixing something that will never become a problem.

After the two fixed-price projects, the work moved to a monthly retainer. I met with the product manager on a regular cadence to keep the work moving, surface problems early, and look ahead at the next constraint. The translation mattered more than it sounds. Naming the link between “every update means an outage” and the queue design turned a vague sense that something was wrong into a sequenced plan, with a tradeoff the product manager could defend in a roadmap conversation.

Results

The platform runs around the clock, with each part scaling up and down as demand requires, and the business no longer plans around maintenance windows. The downtime problem went away as a side effect of protecting the work, and there was an operational cost saving of ~40%.

What stayed after the engagement

The engineering team built every piece, so it stayed with them: the distributed queue, the independently scaled consumers, the automated deployment pipeline and the sending-rate controls.

So did a roadmap for the constraints still ahead, each paired with the business metric that decides when work on it has to start, so the next architecture problem shows up on the plan before it shows up as an outage.