The Prototype That Became Production

It was 2:15 on a Thursday when Somchai pulled up the dependency graph on the big monitor near the window on level 22. Nobody had asked him to. He’d been trying to trace a bug in a service that was supposed to be a temporary bridge between two systems — built eighteen months ago, over a long weekend, by an engineer who’d since moved to another team. The graph looked like someone had dropped a plate of spaghetti onto a circuit board. The vinyl floor creaked as three people quietly rolled their chairs closer to look. Nobody said anything. The aircon hummed its flat, indifferent hum. On Somchai’s desk, a plastic cup of Thai iced tea sat sweating in the afternoon heat, that distinctive milky orange slowly separating — the condensed milk sinking, the tea rising — the way things do when you leave them sitting long enough. He took a long pull through the straw and finally muttered, “So… this is load-bearing now, right?”

He wasn’t wrong. What started as a quick prototype — a “let’s just prove the concept” hack built under deadline pressure — had quietly become the backbone of a system serving real traffic, generating real revenue, and terrifying anyone who got close enough to read the code. Nobody decided it should be permanent. Nobody had to. It just… survived. Like that cup of Thai tea separating on the desk, the original intent had slowly drifted apart from reality, and nobody noticed because nobody was watching.

Nothing Is More Permanent Than a Temporary Solution

The economist Milton Friedman said that. He was talking about government programmes, but he might as well have been describing your microservices architecture. Every codebase I’ve ever worked on — across two countries and more companies than I’d like to admit — has at least one system that started as a prototype and is now infrastructure. Most have five. Some have entire platforms built on foundations that were never meant to bear weight.

The pattern is always the same. Someone builds something quickly to prove an idea works. It works. It ships. It serves traffic. And now it has defenders — not because anyone loves it, but because touching it is risky and there’s always something more urgent than replacing something that technically functions.

Here at Agoda, we had a framework — let’s call it what it was, Chronos — that started life as a modular .NET Core framework. Beautiful vision: clean separation of concerns, plug-and-play components, the kind of architecture that looks great on a whiteboard. Then came a high-pressure migration. Deadlines were tight. The gatekeeping that kept Chronos modular broke down under delivery pressure. “Just put it in Chronos” became the default answer to every architectural question, not because it was the right answer, but because it was the fastest one. Over the next few years, one trade-off at a time, the modular framework quietly became an all-or-nothing monolith. No single decision killed it — it was death by a thousand cuts, each one individually justifiable, none of them individually catastrophic. By the time anyone stepped back to look at the full picture, the cost of unwinding it was enormous.

This isn’t a story about incompetence. That’s what makes it uncomfortable. These were competent engineers making individually reasonable decisions that collectively created a mess. Every shortcut made sense in isolation. Every “just this once” was justified by a real deadline, a real business need, a real constraint. The problem isn’t that anyone made a bad decision. The problem is that nobody made the next decision — the one where you go back and undo the compromise, restore the boundary, reclaim the modularity that got sacrificed for the deadline. Chronos itself was never temporary. But every compromise made inside it was supposed to be.

Why Smart People Let This Happen

There’s a psychological dimension to this that we don’t talk about enough. We like to frame technical debt as a rational trade-off — speed now, cost later — but the forces that keep prototypes alive are deeply irrational.

The IKEA Effect is the tendency for people to overvalue things they’ve built, even if they built them quickly and badly. “It’s not that bad” is the most dangerous sentence in an architecture review. It’s the sentence that keeps prototypes alive for years past their intended lifespan.

Here’s a red flag I watch for: when people give their systems pet names. There’s an old infrastructure adage — treat your servers like cattle, not pets. The same applies to systems. If you’ve named your pricing service “Cuddly Dragon Baby,” you’ve formed an attachment. And when the right engineering decision is to kill it and build something better, you don’t want to kill your pet, do you? Give your systems meaningful, descriptive names. Call it “Pricing API.” It helps with communication and it keeps emotional distance. You’re an engineer, not a zookeeper.

Loss Aversion makes replacing a working prototype feel like risk, while keeping it feels like safety. But keeping it is just deferred risk that compounds daily. The prototype isn’t stable — it’s a debt instrument accruing interest. Instead, try to make the right path the easy path. Build golden paths that make standing up a new service with proper observability, testing, and deployment pipelines as easy as hacking together a prototype. When doing it right is as fast as doing it wrong, people do it right.

The Planning Fallacy in reverse. When building the prototype, teams chronically underestimate how long it’ll last. “Two sprints, max.” But there’s no mechanism to enforce the expiry date. Nobody puts a calendar reminder that says “remember, this was supposed to be temporary.” Two sprints becomes two quarters becomes two years, and somewhere along the way, temporary became permanent without anyone noticing the transition.

As the psychologist Daniel Kahneman observed, “We can be blind to the obvious, and we are also blind to our blindness.” That’s the prototype trap in one sentence. You don’t see it calcifying because you’re too busy building on top of it.

Sunk Cost meets Revenue Coupling. Once the prototype serves real traffic and generates real revenue, it acquires organisational gravity. Every day it runs successfully makes it harder to justify replacing. “It works” becomes the enemy of “it should work differently.” The revenue it enables makes it politically untouchable, even as its maintenance cost — invisible, spread across dozens of teams as friction — quietly eats your velocity.

The Missing Role. There’s usually nobody whose job it is to notice that temporary things became permanent. Product owners don’t track architectural intent — they love that you built it so fast and ask if you can do it again. Tech leads are fighting today’s fires. The prototype quietly ages into infrastructure while everyone’s looking the other way.

We saw this pattern with shared libraries too. An engineer — let’s say Somchai — builds a useful utility library. “The community will maintain it,” everyone agrees. Community ownership becomes Somchai ownership. Somchai’s ownership becomes Somchai’s burden. Somchai leaves. And now nobody maintains it, but seventeen services depend on it. The library wasn’t a prototype, but the ownership model was — and nobody noticed it had become permanent until the person holding it together walked out the door.

The Real Cost: It’s Not Just Debt, It’s Constraint

Here’s what separates this from a typical “tech debt is bad” conversation. The cost of a calcified prototype isn’t just the mess inside it. It’s the shape it imposes on everything built around it.

A prototype that becomes permanent doesn’t just accumulate technical debt — it becomes an architectural constraint. Future systems are designed to accommodate its quirks. New features route around its limitations. Integration patterns emerge that make sense only in the context of “well, that service does things a bit differently.” The prototype didn’t just create mess; it shaped what was buildable next.

This is how you end up with systems that sat untouched for so long that nobody knows if they still work. Merge requests that sit open for seventy-five days because the code they touch is so fragile that nobody wants to approve them. Architecture that looks irrational from the outside but makes perfect sense once you understand which decisions were supposed to be temporary.

The quality management pioneer W. Edwards Deming warned that “a system must be managed. It will not manage itself. Left to themselves, components become selfish, competitive, independent profit centres, and thus destroy the system.” When one service is a calcified prototype that everyone routes around, it doesn’t just rot in isolation — it teaches the rest of the architecture to compensate, to work around, to accept less. The dysfunction becomes load-bearing.

Throwaway Code That Actually Gets Thrown Away

Let’s be practical. You’re not going to stop engineers from building prototypes — nor should you. Prototyping is how we test ideas cheaply. The problem isn’t the prototype. The problem is the missing mechanisms that would turn “temporary” from a vague intention into an enforceable contract.

Put Expiry Dates on Architecture Decisions. Literally. Whatever form your architecture decisions take — ADR documents, PowerPoint decks, markdown files in your repo, napkin sketches photographed and uploaded to Confluence — they should have a “review by” date. If the decision was “we’ll use this temporary approach,” the document should say when the permanent approach is due and who’s accountable for it. I’m not prescriptive about the format. Use whatever your team actually reads. But make it expire.

Run a Prototype Postmortem. When a prototype ships to production — and yes, sometimes that’s the right call, because the whole point of a prototype is to test something with a low chance of working, and sometimes it works — run a mini-retro immediately. Ask two questions: “What would need to be true for us to replace this?” and “What would prevent us from replacing it?” Document the answers. Congratulations, you’ve just built an early warning system. Most prototypes calcify not because replacement is impossible, but because nobody articulated the conditions early enough.

Have Product in the Room. Make sure your product owner understands the trade-off. This is debt you’ll be paying down, not a free lunch in velocity. When product celebrates the speed of a prototype without understanding the deferred cost, you’ve set the stage for every future prioritisation conversation to go badly. Make the deal explicit: “We can ship this fast, but here’s what we’re borrowing from our future selves.”

Give Revenue-Coupled Code a Budget. If a prototype is serving real traffic, it should show up in your capacity planning and tech debt tracking. Make the cost visible. The reason prototypes survive is that their maintenance cost is invisible — distributed across dozens of teams as friction that shows up in lead times and incident counts but is never attributed to the thing actually causing it.

Schedule Sunset Sprints. Not “tech debt sprints” — that’s too vague. Specifically: dedicate time to evaluating whether temporary solutions should become permanent (with proper investment) or actually be replaced. The question isn’t “should we fix tech debt?” The question is “which of our temporary solutions are now permanent, and do we accept that intentionally?”

Build Kill Switches by Design. When you build a prototype, spend the extra ten percent of effort to include explicit seams — feature flags, abstraction layers, separate configuration — that make replacement possible later. This connects directly to the strangler pattern: you can’t gradually replace something that’s entangled with everything. The cost of building in a clean boundary at prototype time is small. The savings when you actually need to replace it are enormous.

Make the Right Path the Easy Path. Here’s the thing — if building a proper service takes three weeks and hacking together a prototype takes three days, people will prototype. You can’t fight that with policy. You fight it with tooling. We’ve invested heavily in golden path templates — project scaffolds that encode our architectural decisions, our logging patterns, our CI/CD configuration, our testing setup. Run the generator, and you get a new service that already looks like a real system. When standing up something properly is nearly as fast as hacking it together, the “temporary” prototype becomes a lot less tempting. I wrote more about this in my golden paths post — the short version is that the best governance doesn’t look like governance. It looks like a project template that makes the right decisions for you before you’ve written a line of code.

Sign the IOU. Here’s a practice that sounds dramatic but works: when a prototype ships to production — and sometimes it should, because the whole point is to test something with a low chance of success cheaply — make the team sign a metaphorical IOU. Not literally, but close. Document the compromise explicitly. And critically, make sure everyone understands the real math:

The true cost of a prototype = the cost to build it + the cost to deintegrate it + the cost to build the real solution.

That middle term is the one people conveniently forget. Building the prototype isn’t free just because it’s fast. You’re not just deferring the cost of the proper solution — you’re adding the cost of ripping the prototype out first. The longer it stays, the more things integrate with it, and the more expensive that deintegration becomes. Get product, engineering, and leadership to acknowledge this equation in writing: “We are shipping this knowing it’s not production-grade. We accept the debt. We commit to paying it down by [date].” When everyone has skin in the game and understands that the prototype actually makes the eventual real solution more expensive, not less, “we’ll fix it later” stops being a vague intention and starts being an actual commitment with names attached. The IOU doesn’t prevent the prototype from shipping — it prevents everyone from conveniently forgetting that it was supposed to be temporary.

The Uncomfortable Truth

Here’s the part that makes this genuinely hard rather than just a “best practices” lecture: some prototypes should become permanent.

Not every quick solution needs to be replaced with a proper one. Sometimes the prototype is good enough. Sometimes the cost of replacement genuinely exceeds the cost of living with it. Sometimes the business context has changed so much that the “proper” solution you originally envisioned no longer makes sense.

The skill isn’t preventing prototypes from becoming permanent. The skill is deciding intentionally rather than letting it happen by default. There’s an enormous difference between “we evaluated this prototype, decided it’s fit for purpose, and invested in hardening it” and “nobody ever got around to replacing it and now we’re stuck with it.” The outcome might look the same on a dependency graph, but the organisational health implications are worlds apart.

As the architect Christopher Alexander wrote, “When you build a thing you cannot merely build that thing in isolation, but must repair the world around it, and within it, so that the larger world at that one place becomes more coherent.” When a prototype becomes permanent by decision, you repair the world around it — you add tests, documentation, monitoring, clear ownership. When it becomes permanent by neglect, the world around it becomes less coherent, one workaround at a time.

The Bottom Line

Every codebase has ghosts — systems that were supposed to be temporary, built by people who’ve moved on, serving traffic that nobody planned for, maintained by engineers who inherited them without context. These ghosts aren’t the result of bad engineering. They’re the result of missing mechanisms: missing expiry dates, missing postmortems, missing conversations with product about the true cost of speed.

Your prototype is not the problem. Your prototype is doing exactly what prototypes do — it proved the concept. The problem is what happens next. Or more precisely, what doesn’t happen next.

The next time someone on your team says “it was only supposed to take two sprints,” ask them the follow-up question that nobody asks: “And when it’s done, what’s the plan for replacing it?” If the answer is silence, you’re not building a prototype. You’re building your next piece of permanent infrastructure. You might as well do it intentionally.

Now, if you’ll excuse me, I need to go check on a “temporary” service we stood up in 2022. I’m told it’s still running, still serving traffic, and still has a TODO comment at the top that says “replace with proper implementation.” At this point, I’m thinking of framing it.