Introduction
The times when web services consisted only of a database, an API, and a simple frontend (built with jQuery—remember that?) are a thing of the past. Every developer working with today’s technologies will sooner or later face the question: is my service sufficiently performant, accessible, and secure? At Mews, we manage data for over 5,500 users scattered across the globe, which means we pay particular attention to these issues.
Welcome to the first article in a series dedicated to the broad topic of observability & monitoring. In this series, we will explore the challenges developers and managers face, tested patterns, and practical implementation examples. Our goal is to keep the content quasi-technical, making it accessible to everyone. Happy reading.
What is observability & monitoring?
As with any other technical article, let’s start with some basics. The internet offers many definitions of these two terms (admittedly, they sound professional), but the one that best captures the essence of the matter, in my opinion, is this from IBM:
- Observability is the ability to understand a complex system’s internal state based on external outputs. When a system is observable, a user can identify the root cause of a performance problem by looking at the data it produces, without additional testing or coding.
- Monitoring is the task of assessing the health of a system by collecting and analyzing aggregate data from IT systems based on a predefined set of metrics and logs.
Does this sound complicated to you? The first definition might give you a headache. How do you prepare a service to be observable? In practice, both concepts often go hand in hand, and observability & monitoring tools do most of the work for us. Moreover, they are easily accessible and configurable as code snippets, SDKs (Software Development Kits), or out-of-the-box services within CDNs (Content Delivery Networks) or the cloud.
Fantastic! Let’s dive into the work, then. Not so fast! Before setting out on an observability & monitoring adventure, consider that most solutions available on the market are paid. While the configuration itself is relatively straightforward and not challenging, the initial enthusiasm can quickly give way to reality when you discover how much data you’ve consumed (in external data analysis services like Sentry or New Relic, you usually pay for the data volume).
For example, one feature we implemented to examine potential website performance issues, dubbed “Property Doctor”, consumed about 30% of our free data limits for just one client in seven days. Of course, there are methods to optimize data usage (sampling, using your own infrastructure), but the fact remains that these functionalities are generally not free.
How can observability & monitoring benefit your business?
Now that we’ve covered the definitions, let’s look at how observability & monitoring can help. At Mews, as a global leader in hospitality software development, we set high standards for customer service, including the availability of our services 24/7.
One of the key factors influencing the choice of our services is the overall ease of use and availability of the features we offer. This also means the general performance and speed of our service, which, depending on the needs of our clients, can be pushed to its limits (viewing and editing tens of thousands of records at once is nothing unusual for us).
Thanks to the tools we use, we can quickly respond to various types of issues, such as:
- Availability problems caused by infrastructure defects
- Long page load times
- Unavailability of specific services on the site
But also:
- Effectively identifying problematic parts of the application (like inefficient code)
- Tracking interactions on individual elements of the site
Expanding observability beyond the backend
Wait a minute! We were just talking about system monitoring. How does this relate to the website itself?
In most companies I’ve worked with, observability & monitoring were associated with and reserved for backend solutions. Interestingly, this was never due to developers’ misunderstanding of the subject (even mid-level developers could identify that not all information about a web service could be collected solely on the server), but always a business decision. Websites are sometimes treated as a necessary evil—something that just needs to “look good.”
However, it is often said that “websites make the money.” In an ideal world, a website is not just a set of images, but a complete tool designed to fulfill a specific goal. So, what if we looked at the possibilities of observability & monitoring tools from a different perspective?
Let’s assume we’ve just become marketing managers for a medium-sized online store. Our task is to gather data on the most frequently viewed products, learn about our users, identify inefficient parts of our site, optimize the sales process, and generally increase sales (we love ambitious goals!).
Typically, our first choice would be analytical tools like Google Analytics or Mixpanel—but these only provide partial answers, focusing on specific aspects like click counts or page views, and at best, giving us basic user data.
Optimizing UX with monitoring tools
In practice, we will also use various tools of this type or data aggregators like Databricks (which excels in analysis). Yet, all the tasks we face can be accomplished with just one service.
Using features from one of the most well-known monitoring tools—Sentry (unfortunately, we don’t receive extra funds for this promotion, hey Sentry!)—let’s see how easily we can do our job:
- With the Insights feature, we can easily identify the most frequently viewed URLs (products) and learn user details (including the types of devices used to browse our pages).
- By using the Transaction Summary, we can evaluate the effectiveness of our solution, locate its slowest components, and, thanks to Profiles, pinpoint which specific parts of the code are not performing as expected.
Finally, using Traces we can determine the greatest opportunities for speeding up the purchase process; from the moment the CTA (Call to Action) is clicked to the full processing of the request by the server.
Passionate about monitoring and observability like Aleksander?
Check out our open positions!
Need more? Modern browser APIs (Application Programming Interface) allow for the analysis of user activity down to individual clicks on specific elements on a web page. Sounds like heatmaps? Exactly! But instead of relying on yet another external service, all data is presented on a silver platter, in an accessible graphical format, in one place.
And as a “dessert”, thanks to automatic error reporting, we’ll also know when users couldn’t access the application.
Leveraging observability for a competitive advantage
What if marketing isn’t your destiny? Let’s explore other potential uses for observability & monitoring capabilities.
One of the key advantages of such tools is data accumulation. This represents great potential (and a challenge, but more on that shortly) for analysts, and offers considerable opportunities to leverage information about your service compared to others.
For example, a few months ago, one of our clients managing a network of hotels in the EU reported complaints about the low performance of our services: long page load times, slow CRUD (Create Read Update Delete) operations.
This complaint piqued our interest, as we found no confirmation of the issues described in our backend statistics. Only by comparing data regarding our client’s device communication with other networks, particularly ASN (Autonomous System Number) did we uncover the problem: a connectivity issue between a specific AS and our CDN (confirmed by other users of that CDN).
This problem was identified through excellent analysis provided by Browser Monitoring from another major market player, New Relic, which we also use. With such information, we can support our Customer Support team by providing them with dynamic data, significantly reducing the time required to resolve user-reported issues.
Data interpretation: A double-edged sword
Let’s revisit the data for a moment. The paradox of data is that everyone wants it, but only some know how to use it. As our example demonstrates, having the right data is one side of the coin, but analyzing it is another story entirely.
One of the first questions we asked ourselves when implementing broad monitoring on our websites was “What exactly are we observing?”. Learning how to interpret our data took time, and we understand that this can be a blocker for many (sadly, services like New Relic do not include a free analyst in their package).
However, once you realize the potential of the data you possess and learn to use more advanced search options (SQL forever!), solving potential problems, like the one described above, becomes much simpler. Don’t underestimate the power of AI, which can often assist in this analysis!
Of course, you might argue that not everyone needs to buy the entire dairy farm just to drink a glass of milk (let’s face it, tools like Sentry are now multi-functional powerhouses). And that’s true. Fortunately, most features offered by external services rely on publicly available APIs. This means there are solutions available on the market that can be implemented at a lower cost/effort.
For example, to investigate basic site performance metrics known as Web Vitals, data can be collected using the browser’s Performance API. A basic tool for implementation can be easily found on GitHub.
If we need a systemic solution that provides an overview of communication performance between our services (including the frontend), the OpenTelemetry standard, already supported by various programming languages, comes to the rescue.
And if we only need to examine the availability of our services, additional tools may not even be necessary—most internet providers offer such services out of the box.
In summary
Does every service truly need such functionality? Experience tells me that the answer depends on its purpose. In our case, running a blog with static content probably would not justify implementing observability & monitoring solutions; instead, we’d use widely available tools. However, if you run a recognizable e-commerce site and care about the quality of your service and customer satisfaction, investing in such solutions is highly advisable.
What’s next?
If you’ve made it this far, it might mean:
- It’s an exceptionally slow day at work, and our article popped up first in your search results by sheer luck (congratulations to our Community Team!)
or
- You care about your services and want to learn more about monitoring and optimization.
In either case, stay tuned for the upcoming parts of our series, where we’ll delve deeper into topics such as:
- Frontend monitoring, including why Web Vitals don’t always give a complete picture of site performance, the features worth investing in, alternatives available on the market, and cost-optimization techniques.
- Where and how to use observability & monitoring tools beyond website performance assessment.
- Understanding the OpenTelemetry standard and its role in building efficient, robust services.
- Experiences in tracking potential user-reported issues.
- Leveraging AI in data analysis.
See you soon!
