When building resilient systems, it’s easy to forget how subtle design choices, like where you put a Kafka call, can become major bottlenecks. We hit this firsthand during a recent outage that caused thousands of Temporal workflows to stall. Here’s how we fixed it, and what we learned.
TL;DR:
A Kafka outage caused thousands of our Temporal workflows to get stuck before doing any meaningful work. We shipped a fix to decouple core logic from Kafka by running steps in parallel—using the patching API and replay tests to keep things safe. We also learned a few hard lessons about versioning, workflow longevity, and staying focused on what really matters.
What happened?
We recently had a rough run-in with Kafka that reminded us how fragile things can get when side effects block core processing.
The issue stemmed from a Temporal workflow we use to process Stripe webhooks. It’s a pretty straightforward flow—at least on paper:
- Extract metadata from the event
- Send a “processing started” event to a Kafka topic
- Execute the actual processing logic
- Send a “processing finished” event to Kafka
Simple enough. Except that when Kafka became temporarily unavailable, all those workflows stalled at step 2. They didn’t fail—they just… waited. Thousands of them.
The actual business logic (step 3) was perfectly fine, but it never ran—because a messaging side effect was placed before it in the sequence. With Kafka down, nothing progressed.
The good news? Temporal did what Temporal does best: once Kafka came back online, all those workflows quietly resumed. No human intervention needed. But the damage was done, and it was clear we had to rethink the design.
This post covers what we changed, what we considered but didn’t do, and what we learned from the whole experience.
What we changed
We updated the workflow to run steps 2 and 3 in parallel, ensuring that processing isn’t blocked by Kafka outages.
Instead of launching a new workflow version (which we did consider), we used Temporal’s patching API to safely update the logic without impacting in-flight workflows. Replay tests helped validate the changes and build confidence.
Surprisingly, we discovered that some very old, stuck workflows were still hanging around. That meant we had to keep the patch live longer than expected.
Want to help us build resilient systems?
Check out our open positions!
What we learned
- Don’t block core functionality on side effects like Kafka.
- Patching > forking, especially for targeted changes.
- Long-lived workflows are real—and sometimes resurface at the worst times.
- Replay tests are invaluable for safely validating changes.
Key takeaways
- Prioritize what’s essential in your workflow. Core processing should always run.
- Parallelize non-critical steps to avoid coupling with external systems.
- Plan to support patches longer than you think.
- Use replay tests as a safety net for evolving sensitive logic.
Summary
This incident was a humbling reminder: even reliable systems like Temporal (if you want to learn more about how we got started with Temporal, check out the previous blog post) can be tripped up by how we design around external dependencies. By decoupling side effects, leaning on patching, and testing with replays, we made our workflows more robust—and hopefully helped future us sleep a little better.
