There’s a certain kind of issue that only reveals itself when you’re successful. Not a logic error, not a race condition, but something quieter and more humbling: a solution that simply wasn’t built for the size you’ve grown into. This is the story of one of those moments, what we found when we looked closely, and how we learned from it, then turned it into a meaningful step forward.
The Business Context: When Your Biggest Customers Feel the Pain First
Not all performance problems are created equal. A delay of a second for a small hotel with a few hundred bookings per month is a very different story when it affects one of your largest enterprise customers, and the delay gets scaled up. And in any B2B business, that distinction matters enormously. Large customers represent not just significant revenue, but also trust, long-term contracts, and reputation.
So when one of our biggest hotel customers flagged that searching for a guest profile was painfully slow (we’re talking around 30 seconds per search), it immediately became a priority. Thirty seconds is an eternity in a hotel lobby. Front desk staff are standing face-to-face with guests. That kind of lag isn’t just a UX problem; it’s an operational one.
The first question we asked ourselves wasn’t “How do we fix this?” It was “Why is this only happening now, and only for them?”
Suboptimal by Design Is Okay, Until It Isn’t
Here’s something worth saying out loud: the code that caused this problem wasn’t written carelessly. It was written pragmatically, at a time when pragmatism was exactly the right call. In this case it meant doing what every engineer hates – accepting that the performance isn’t optimal.
When you’re an early-stage product or a growing startup, shipping features fast is survival. You make reasonable assumptions — “our customers will have tens of thousands of guest profiles” — and you build for that reality. You don’t over-engineer. You don’t prematurely optimize. That’s healthy. That’s how products get built.
The challenge is that as your company grows, those reasonable assumptions quietly stop always being true. Data volumes grow. Customer profiles diversify. And one day, a solution that served you perfectly well for years, and still does for most customers, hits a wall for your most important customer. Not because it was ever wrong, but because it was designed for a different profile of customer. This is a natural part of scaling.
Digging Into the Problem
The Culprit: A Query That Didn’t Know What It Was Up Against
The guest search functionality was backed by a SQL query that had lived comfortably in our monolith for a long time. It worked for hotels with guest databases in the tens of thousands. But this particular customer had approximately 13 million guest records — an order of magnitude beyond our typical profile.
When we pulled the query out and examined it under load, the issues became clear:
- Missing indexes on the columns being searched. The database was performing range scans across millions of rows on every single search request.
- Suboptimal query structure that wasn’t leveraging the database engine efficiently: non-sargable predicates that prevented index usage and result sets far larger than needed before filtering was applied.
- Searching on multiple columns even when it doesn’t make sense (e.g. looking for phone number when the search input didn’t include digits).
- Searching for guest names diacritic insensitively, but the database was built diacritic sensitively.
- No pagination or result-set limiting built into the core query logic, meaning even a narrow search could pull back far more data than the user would ever see.
For 50,000 records, these inefficiencies are very minor. For 13 million records, they’re catastrophic.
Where to Fix?: A Push Forward on Monolith Decomposition
Performance crises have a way of creating architectural momentum, and this one was no exception.
Where to put the fix isn’t the obvious first question, but maybe it should be. The default solution would have been to change the existing search in the monolith, but why apply that constraint? Sometimes taking a step back and looking at the whole system architecture then considering what the ideal solution would be is the best plan.
Success creates new engineering problems.
Those are our favorite kind.
We had been on a gradual journey of decomposing our monolith into microservices — a process that is never as fast as you’d like, because the monolith always has more surface area than you remember. The guest search work gave us a clear, well-scoped opportunity to move one piece of that functionality into a different service. Fortunately, we’d already created a service for guest profiles before the search issue surfaced. We already had a copy of guest data in a new database, we just needed to implement search functionality from scratch in our microservice.
The new service gave us better isolation, independent scalability for search traffic, as well as the benefit of reducing the load on our monolith database, along with the freedom to evolve the search infrastructure without being constrained by the monolith’s assumptions.
Obviously not every problem will have such a convenient solution, where you can fix the customers issue at the same time as moving your own architecture forward, all in a reasonable time-scale. But unless we’d taken the small amount of time to consider what the best solution would look like, we would probably have just done the default and applied a patch to the old monolith code.
The Solution: Smarter Search from the Ground Up
Once we understood the problem fully, the path forward was clear.
The rewritten query was built with scale as a first-class requirement:
- Ideal indexes were already present in our microservice, with composite keys where appropriate to match common search patterns. All we had to do was apply diacritic insensitivity at the database level, as that’s what the application required.
- Query restructuring to ensure the database engine could apply indexes effectively. Rewriting conditions to be sargable, eliminating redundant subqueries, and pushing filtering as early in the execution plan as possible.
- Smarter search logic that prioritized exact and prefix matches before falling back to broader searches, dramatically reducing the result set the database needed to evaluate.
- Breaking down the huge query into smaller, optimized search steps.
- Does the search input look like an email address? Search on email address first.
- Does the search input look like a phone number? Search on phone number first.
- No special characters? Might be name of the guest → Search on name first.
- Breaking down the huge query into smaller, optimized search steps.
The result? Guest search for that customer dropped from ~30 seconds to under 200ms. A 150x improvement. The front desk could breathe again.
The Observability Gap: How Did We Miss This?
This is the question we sat with the longest, and honestly, it’s the most instructive part of the story.
The short answer: our monitoring was calibrated for our average customer, not our outliers.
Our performance dashboards and alerting thresholds were set based on typical usage patterns. Average query times looked fine in aggregate, because the vast majority of our customers were fine. The slow queries from the large customer were statistically drowned out. There was no per-customer performance segmentation and no alerting for long-tail outliers.
In hindsight, this makes intuitive sense. You monitor what you know to watch for. We hadn’t yet developed the instinct to ask: “What happens to this feature at 100x the data volume we normally see?” That’s a question born from experiences like this one.
The fix here wasn’t just technical. It was about expanding how we think about observability: not just “Is the system healthy on average?” but “Is the system healthy for every customer segment, including the ones at the edges?”
What We Took Away
Looking back, there’s no single villain in this story. Not the original developer, not the product team, not the infrastructure. A pragmatic solution met the limits of its design when our world changed around it. That’s a very human, very normal thing to happen in software.
What matters is the response: the curiosity to understand why rather than just patching the symptom, the discipline to fix it properly rather than just fast, and the awareness to improve our systems so we catch this class of problem earlier next time.
Scale reveals everything eventually. The best you can do is stay curious and build the muscle to respond well when it does.
