It’s been some time since we began thinking about using the durable execution framework called Temporal. Why? The reason is fairly simple: it helps us implement actual business cases while allowing us to worry less about how we reliably execute the code. But as with everything in life, nothing is black and white — rather, it’s 255 shades of gray.
I had heard about Temporal quite a few times, read articles about it, and noticed they have a pretty big customer base, so I thought there must be something to it.
One concern we had was the risk of vendor lock-in, as it can significantly impact your code since you build your crucial business logic around it. On the other hand, you can easily migrate between cloud providers because Temporal doesn’t care where your code runs.
Initial considerations
We began by evaluating exactly where and how to use it. The usual recommendation is to introduce new technology in a non-critical area so that if anything goes wrong, the impact is minimal. Easier said than done, right? Then, suddenly, customer issues started rolling in related to a problem with automatic payments. We stepped back and said, “Hey, even though it’s a somewhat critical piece of logic, it’s not huge, and we can always hide it under a feature flag, right?” So…
Pre-implementation challenges
As part of the payment automation logic, we needed to close a bill. The problem was that it took a variable amount of time, which unexpectedly prolonged the overall payment automation process. Not ideal. While the closing logic itself could be improved, you can’t fix everything at once.
Another issue was observability. Previously, we had very little insight into the actual performance of the bill-closing process—such as the p99 execution duration or the most common errors. This made us eager to leverage OpenTelemetry more effectively in this area.
While diagnosing issues, we also discovered the need to improve exception handling, as it wasn’t entirely clear what exceptions might be thrown by the code.
Are you ready to dive in and get to work?
Check the open positions.
Setting up Temporal.io infrastructure
We have an internal guideline that strongly favors managed solutions over self-hosted ones. Temporal fits this requirement nicely, as they offer a managed version of their server through the Temporal Cloud service. Among other benefits, they continuously work to improve customer experience—such as by introducing custom-tailored enhancements in cloud persistence.
We requested an account and had our first instance deployed within a day.
Implementing bill closure with Temporal.io
One of the strongest arguments for using an independent workflow for this task was that the automation doesn’t have to wait for the bill to be closed. It’s essentially a “fire and forget” scenario. The key point is that the workflow is executed reliably, so we don’t need to worry about restarting it after a deployment, for instance.
Here’s the diagram of actions that need to happen as part of the workflow:
The actual flow of things was quite straightforward, so we decided to implement it. However, it made us consider the details of the implementation—are they idempotent? How do we track the success of the migration in our observability tool?
It was crucial to start small by enabling the new functionality for just a subset of customers and addressing potential issues along the way. To do this, we used LaunchDarkly for feature flag management.
While implementing the workflow, Temporal’s architecture and built-in retry mechanism helped guide us in designing the code properly. You must be explicit about which exceptions trigger retries and which ones cause workflows to fail without additional retries.
I don’t need to elaborate on integration tests, as they are expected to be an inherent part of a process like this. The Temporal .NET SDK also makes it easy to spin up a custom development server for testing, and its usage was smooth and intuitive.
Temporal support & guidance
I was pleasantly surprised by how the Temporal team guided us throughout the entire process. The account manager was in constant contact with me and connected me with the right people when needed. The maintainers of the .NET SDK responded within a day and helped us find the right solution for our problem.
A week before going live, they even provided us with a design review session where we had the opportunity to walk them through our implementation and perform basic sanity checks. Honestly, I really appreciated this part.
Lastly, their support, managed through Zendesk, was fast and helpful. There’s a minimum price for Basic support at $200, but it’s totally worth it. A few times when we needed a feature, it was already in preview, and they simply enabled it for us.
Results & key takeaways
Once deployed to production, we gradually enabled the feature flag for customers while monitoring a dashboard in our observability tool to keep an eye on critical metrics. It took us nearly a week before we enabled it for everyone.
Currently, we are successfully closing around 6,000 bills every day via Temporal. While that number isn’t huge, it’s a significant step forward for us. Personally, I was surprised by how minimal the latency was, even with the external service running in a different data center.
Allow me to sum up what we gained and learned on this journey:
- Payments automation is now faster since it no longer has to wait for bills to be closed (you can also read about our automatic payments in this article).
- We verified that Temporal can be a viable solution for a whole set of problems.
- The refactorings done alongside the implementation of Temporal improved our codebase overall. We converted certain functionalities into proper command handlers (see CQRS description for more details), making the new workflow usable throughout our product and enhancing the reusability and testability of the underlying code.
Looking ahead
So far, everything has been working well without any major issues. We’re eager to implement additional use cases with the durable execution approach and see how much more we can accomplish with Temporal.
