The Tangled Web We Weave: How Moving Fast Led Us Down the Synchronous Coupling Rabbit Hole
“The chains of habit are too weak to be felt until they are too strong to be broken.” — Warren Buffett
Buffett was talking about personal habits, but he might as well have been describing our monolithic systems four years ago.
The Wake-Up Call That Made Us Rethink Everything
Picture this: You’re sitting in your favorite Bangkok café, sipping your morning coffee, when suddenly partner-facing systems starts throwing errors. Content pages won’t load. Search results are timing out. Partners are calling, and you’re frantically trying to figure out what’s broken.
The culprit? The finance API went down due to a production issue.
Wait, what? Why would content pages care about finance APIs? They shouldn’t. But they did, and that’s was the result of what our monolithic architecture had created, something dangerous — invisible coupling cascades that would only get worse as we grew. We needed to break the monolith into domain-based microservices, but the critical question was: how do we avoid simply recreating the same coupling problems in a distributed system?
“What we learn from history is that people don’t learn from history.” — Warren Buffett
We wanted to learn.
How “Move Fast and Break Things” Broke things
At Agoda, moving fast is one of our core values (without the break things, that was from a Zuck quote), and it’s served us incredibly well. We ship features rapidly, iterate quickly, and respond to market changes with agility that makes our competitors envious. But in our monolithic systems, this strength had created invisible coupling problems that were becoming increasingly dangerous as we scaled.
Here’s what we discovered in our monolith: in our monolithic BFF (Backend for Frontend) and orchestration layers, adding another service dependency to a client request felt as natural as referencing a namespace. Need user preferences? There’s an API for that, and the client is already configured. Want to show personalized content? Just add another API call — it’s already wired up and ready to go. Your IDE might even auto-complete the namespace for you, so you barely notice you’re adding another cross-domain dependency.
“Every system is perfectly designed to get the results it gets.” — W. Edwards Deming
We were getting exactly what our system was designed for: maximum developer convenience at the cost of system resilience. Every “quick” API call was another thread in our web of dependencies. A single HTTP request to our BFF could trigger 30–100 backend API calls and SQL requests. We were trading away fault tolerance for development velocity, and the technical debt was compounding.
The Invisible Coupling Cascade in Our Monolith
The insidious thing about this pattern in our monolithic architecture was how invisible it became. We discovered that pages inadvertently dependent on different backend services across many different domains.
None of these dependencies made business sense, but they’d accumulated organically as developers took the path of least resistance in the monolith to delivery requests from product. After all, why build a local cache from an event stream and deal with eventual consistency when you can just make another API call to a service that’s already wired up?
This was precisely the problem we set out to solve by decomposing our monolith into domain-based microservices. But we quickly realized that simply breaking apart the monolith wouldn’t automatically solve the coupling problem — we needed a different approach, to not repeat the same mistakes.
The Strategic Shift: Domain Separation as the Solution
We knew that simply breaking apart the monolith into microservices wouldn’t automatically solve our coupling problem — we needed intentional friction around cross-domain dependencies. We say friction and not “blocking” as sometimes they are ok. Friction can be your friend.
“A good system should make it easy to do the right things and hard to do the wrong things.” — Jeff Atwood
Our solution was to implement domain separation as a design constraint. By decomposing our monolithic orchestration layers into domain-specific microservices, we’re creating beneficial friction around cross-domain dependencies. In these domain-separated services, adding a call to an API outside your domain boundary requires deliberate effort.
You have to set up new clients. You have to configure authorization. You have to justify the dependency in code reviews. This friction forces the critical question: “Should I be doing this?”
The New Architecture: Loose Coupling Through Asynchronous Patterns
Instead of defaulting to synchronous API calls, we’re shifting toward:
Event Sourcing Patterns: Capturing business state changes as immutable events that other domains can subscribe to, rather than calling across service boundaries synchronously.
Eventual Consistency Models: Prioritizing system resilience over immediate consistency where business requirements allow. Yes, this means rethinking some UX patterns, but it also means systems that gracefully degrade instead of catastrophically failing.
Message Buses and Event-Driven Architectures: Making asynchronous communication an easy common alternative, with synchronous calls reserved for cases where they’re genuinely necessary.
Why This Matters for You
If you’re an engineering leader, you’ve probably seen similar patterns in your own systems. In a monolith, the symptoms are different but equally problematic: cascading failures where unrelated features break together, difficulty understanding the impact of changes, and invisible dependencies that accumulate over time.
When you move to microservices to solve these problems without thinking it through, you often trade monolith issues for distributed monolith challenges, like deployments that require coordination across multiple teams, “quick fixes” that take weeks to release because they span service boundaries, and mystery failures where debugging feels like playing six degrees of Kevin Bacon with your service dependencies.
By putting our boundaries at the business domain level, we found that product work is usually isolated withing a single business domain (its them that helps us draw these lines), so generally we have found that this helps reduce the co-ordination across many services problems.
Drawing these boundaries though for individual engineers, this architectural shift means thinking differently about how you solve problems that do need to cross them. Instead of asking “Which API has this data?” you start asking “How can I get this data without creating a runtime dependency?”
The Implementation Philosophy: Strategy, Not Dogma
We’re not mandating specific technologies or patterns. Teams have full autonomy to choose appropriate message bus technologies, determine which interactions truly need synchronous communication, and design event schemas that serve their business requirements.
The goal isn’t technology adoption — it’s building systems that serve our partners reliably at scale, even when individual services are having a bad day.
Trade-offs and Reality Checks
Let’s be honest: this approach introduces new complexities. Debugging distributed, eventually consistent systems is harder than tracing synchronous calls (Open Telemetry helps). Operational overhead increases. Some features will take longer to build initially.
But here’s what we’ve learned from the past six months of gradual migration: the short-term complexity pays massive dividends in long-term stability and team velocity. Teams can now deploy independently without coordinating release schedules. Partner-facing APIs maintain consistent response times even during peak traffic or partial outages.
The Path Forward
We didn’t take a gradual, organic approach to this problem. Instead, we launched a dedicated project to systematically move everything out of the monolith, focusing on high-change areas first to maximize the impact on developer velocity. By prioritizing the parts of our system that changed most frequently, we could unlock the biggest wins for team productivity while proving out our domain separation approach.
“You can’t go back and change the beginning, but you can start where you are and change the ending.” — C.S. Lewis
We couldn’t undo years of coupling decisions in our monolith, but we could be intentional about how we structured our new microservices. By introducing intentional friction around cross-domain dependencies from day one of each service extraction, we’re building systems that can evolve independently while still serving our partners reliably at web scale.
What patterns have you seen in your own systems? Have you fallen into the “just call the API” trap? I’d love to hear your war stories and solutions in the comments below.