At Mews, we try to go above and beyond to help hotels increase their bottom-line revenue. One way we do this is by continuously exploring how our clients use Mews products : what works, what doesn’t, and where there’s untapped opportunity.
As part of this effort, we were investigating upsell performance across properties, looking for patterns we could turn into concrete recommendations. The goal was simple: identify what drives upsell revenue, and help hotels replicate those wins.
We started by measuring upsell rates, the percentage of check-ins including at least one upsell. This way, we could rank properties by performance and try to uncover a correlation between upsell success and how hotels were using Mews. But as we looked at the numbers, we quickly noticed how misleading some results might be.
The Problem with Small Sample Sizes
Properties with few check-ins could show very high upsell rates, simply because the sample size wasn’t large enough for the rate to converge toward its true value. Raw percentages, on their own, failed to reflect the uncertainty behind the numbers. As a result, properties with limited data were often ranked higher than those with significantly more check-ins.
Take two hypothetical examples:
- Hotel A had just 5 check-ins last month. Three of them resulted in upsells, yielding a 60% upsell rate.
- Hotel B had 100 check-ins, with 40 upsells , so only 40%.
If we took those numbers at face value, Hotel A would seem like the clear leader. But with such a small sample, Hotel A’s result could easily be a statistical fluke. One guest deciding not to buy would drop the rate to 40%. Two guests? Now it’s 20%. Small changes lead to huge deltas, reducing the confidence in the results.
Hotel B’s rate, on the other hand, is supported by a hundred data points. It’s much more likely to reflect the true behavior of its guests.
This is the challenge of working with proportions based on small counts: the numbers can swing wildly, leading to low confidence in the results which can distort analysis and mislead decisions.
If you reach into the deepest corners of your mind, you might recall from your statistics class that from the binomial variance, we can obtain the variance of the sample proportion as:
This explains how when the number of check-ins n is small, the variance of p-hat is large — meaning the observed upsell rate can swing wildly from the true rate. As n increases, the variance decreases, and the sample proportion becomes a more reliable estimate of the true upsell probability p.
Measuring Not Just Performance, But Confidence
We needed a way to measure not just how high the upsell rate was, but how much we could trust it. A method that rewarded sustained performance over one-off spikes.
That’s when we turned to the Wilson Confidence Intervals : a statistical approach that gives you a confidence interval for the true success rate, while accounting for the uncertainty introduced by small sample sizes.
Originally proposed by Edwin B. Wilson in 1927, this method is widely used in problems involving binary distributions and limited data. It’s commonly applied in A/B testing, online ratings and review systems, and machine learning classification . Basically, anywhere you need to estimate the probability of success while accounting for how much evidence supports it.
Rather than taking the observed rate at face value, the Wilson Score produces a lower and an upper bound where the true success rate likely falls. We chose to use the lower bound of that interval as our new metric, which can be calculated with the following formula:
where:
- p-hat is the observed success rate (e.g. 3/5 = 0.6 or 60%)
- n is the number of trials (e.g. 5 check-ins)
- z is the z-score for your confidence level (1.96 for 95%)
Why the lower bound? Because it answers the question:
“Given the amount of data we have, what’s the lowest upsell rate we can be 95% confident about?”
It’s a cautious but realistic way of reading the data.
Want to help us interpret our data?
Check out our open roles and join the team!
Revisiting the Example
Let’s go back to our two hotels:
- Hotel A: 3 upsells out of 5 → 60% raw rate
- Hotel B: 40 upsells out of 100 → 40% raw rate
Using the Wilson score interval at a 95% confidence level, we get the following confidence ranges:
- Hotel A: 23% — 88%
- Hotel B: 31%— 49%
Intuitively, the smaller the sample size, the wider the confidence interval. Since we’re interested in penalizing properties with fewer check-ins, we take the lower bound for each.
Suddenly, the rankings flip , and for good reason. Hotel B comes out top with 31% upsell rate The Wilson method gives more weight to results backed by more evidence. It ensures that we don’t overestimate performance just because of lucky timing or a small number of check-ins.
What Changed for Us
Rather than relying on raw rates, we began using this adjusted score to rank property upsell performance, monitor trends, and study potential recommendations.
Properties with previously inflated rates due to tiny sample sizes fell back into place. High performers with consistent volumes rose to the top. The entire leaderboard became more stable, fair, and representative of true performance.
This small change gave us a much more accurate view of reality , and that’s critical when you’re advising clients on what actions to take.
More Than Just Upsells
Although we used this method to improve our understanding of upsell performance, the principle applies much more broadly. Anywhere you’re working with binary outcomes — opened vs. unopened, converted vs. not, clicked vs. ignored — you’ll run into the same problem, and Wilson confidence intervals offer a solution.
Looking ahead, this same technique will help us interpret Mews product surveys more accurately, so we can identify customer pain points with greater confidence and prioritize solutions that deliver the biggest impact.
At Mews, that’s what we’re always striving for: not just measuring what happened, but helping our clients make sense of it — so they can improve, grow, and increase their revenue in ways that last.
This is just one of many ways we’re working to turn data into action.
