Code Lies, Data Speaks: Splitting a Monolith on Data, Not Code

Wait 5 sec.

Breaking a monolith into microservices gets planned as a code and architecture problem. You draw the services on a board, you argue about where the boundaries go, you agree on an order of extraction, and almost the whole conversation happens in domain and code terms. The data tends to show up late in that conversation, usually as a line item called "migrate the data", as if it were the mechanical step at the end. In practice, it's the opposite: the data decides whether the whole thing ships, and by the time that becomes clear, it's already too expensive to move as the plan assumed.This is true of almost any system that ran as one database for years before anyone tried to split it, and ours was a clear example. We had over 500 tables in a single database, some of them hundreds of columns wide, and one grew to billions of rows and, at the far end, into the dozens of billions. The target was roughly 30 services, some owning a handful of tables and some a couple of dozen. Getting from the first shape to the second took several attempts over several years, and most of what we learned came from the ones that failed rather than from the version that finally held.Why this class of problem arisesA monolith database is shaped for a monolith. Joins across domains are free, so people use them everywhere. Tables get shared because sharing is one foreign key away, and reporting assumes it can reach any table from any other, because it always could. None of that is bad engineering; it's the natural shape of a system where everything lives in one place. The catch is that this shape quietly encodes an assumption that there are no boundaries or that the boundaries don't cost anything.Microservices need the opposite, because each service is supposed to own its data and nobody else gets to reach into it. So the real work of a decomposition isn't rewriting the code into services, which is mechanical and mostly well understood. It's that you're trying to impose clean data ownership on a database that was built for years on the assumption that ownership didn't matter.The part that took us a while to see was that service boundaries are really domain boundaries, since a domain is what a service is meant to wrap. In a legacy monolith, though, those domain boundaries have washed out over the years, and the first place a washed-out boundary shows up is the data. You won't see it in a design doc, where every domain looks tidy. You see it in the tables: the one column half the system writes to, the join that three "separate" domains all depend on, the table that nobody can quite say who owns.Our clearest example was the custom fields table. Users could create custom fields freely, on more or less any form and in any quantity, so the values table grew without a ceiling. On top of that, nearly every create action in the product spawned a good dozen records per user, and that one table climbed into the dozens of billions of rows.Volume was only half of what made it hard, though. Every domain wrote custom fields, so the table belonged to all of them and none of them at once. It's the same table that sat underneath a search feature I've written about before: there, it was a performance problem, and here, it was the table that refused to sit on one side of any boundary we tried to draw.The plan that kept dying.We didn't reach the working approach in one go. There were about four runs at modernization over several years, and most of them died. Some were shallow proof-of-concepts that ran for a few months or a quarter, coming at the problem from a different surface, mobile or web, on the hope that the boundaries would be clearer from that angle. One was a multi-year stretch handed to contractors that sat at "almost done" for most of its life. Each attempt looked reasonable on paper, and then came apart on contact with the data.They came apart the same way each time, and it's worth being precise about why. The boundaries were drawn top-down, from dependency diagrams and an assumed picture of how data flowed through the app, by people senior enough to own the architecture but far enough from the data that they were mostly guessing at its real shape. The intent was always to go incrementally, one service at a time, which is the right instinct, but the increments were carved along lines the data didn't actually follow.So, you'd start extracting a service and find that the table it needed was also written by four other domains, or that the "clean" boundary ran straight through the dozens-of-billions-of-rows table you had no cheap way to move, or that a reporting screen quietly depended on joining across three of your new services. The plan didn't fail for lack of detail; it failed because it was a hypothesis about the data, written by people looking at diagrams instead of at the data.What finally forced it through had nothing to do with a better plan; we'd simply run out of room to abandon it. Security patches were getting harder to apply, the legacy platform was going out of date, and the system was falling behind on two fronts at once: shipping features without new bugs, and holding up under load. Stopping wasn't an option anymore, so we had to grind. The grind was ugly for a while: false starts, buggy versions, slow ones, and a few that were just meh and clearly weren't going to hold. The mistake across all of those earlier attempts wasn't too little planning, but planning in the wrong layer.What the data actually said.The approach that finally worked started from the data instead of the diagram. Rather than deciding the boundaries up front and forcing the data through them, we let the seams show us where the data actually separated cleanly. We'd peel off one service along a real seam, move its data, check it in production, and only then go for the next one. When a boundary turned out to be wrong, and some did, it was one service's worth of wrong and cheap to correct, not a year of committed planning to unwind.The mechanism that made this possible was change data capture. CDC let the legacy database stay the source of truth while new services consumed its changes and built up their own copies. That lets us stand a service up, run it alongside the old path, compare the two, cut over once we trust it, and retire the old piece one step at a time. That "run alongside and compare" step is the whole game, because it's the only point where the data gets to tell you a boundary is wrong before you've committed to it.The custom fields table is where that discipline paid off. One early boundary assumed the table could belong to the domain that leaned on it most. The run-alongside comparison proved that wrong, because writes from other domains kept showing up on the legacy side that the new service never received. Forcing the table into one service would have meant every other service reaching across the boundary to write into it, which is exactly the coupling we were trying to remove.So the fix was to stop assigning it to a domain at all and treat the shared data as its own concern, with its own owner. That correction cost one service's worth of rework, caught during comparison, rather than surfacing deep into a committed plan the way it had in the earlier runs.We tried the other obvious thing first, and it was a trap: dual writes, where the app writes to both the legacy tables and the new service on every change, coordinated with events. On paper, it keeps both sides in sync during the transition, but in practice, it meant we were still writing to legacy constantly, which is the opposite of what you want when the goal is to retire legacy.The coupling got deeper rather than looser, because every new service stayed wired into the old schema instead of growing away from it. CDC won because it let data flow out of legacy without us having to keep feeding legacy by hand. It's the same instinct behind a talk I gave on turning a messy codebase into something you can actually work with: you work with what the system actually is, not the clean version you wish you had.Telling a diagram-shaped boundary from a data-shaped oneAfter enough of those corrections, you start to recognise the pattern early, and you can usually tell whether a boundary was drawn before or after anyone looked at the data. A few tells were worth watching for:The boundary is described entirely in domain nouns and never in tables. If nobody can point out which tables move and which stay, it's still a hypothesis rather than a boundary.One table has to be "split" or "shared" between two services for the boundary to work, and a boundary that needs a shared table isn't a boundary yet.The plan assumes a big table can just be moved, when anything into the billions of rows can't be moved on the schedule that a diagram implies. That constraint should shape the boundary rather than be discovered after it.Reporting isn't in the plan at all, which matters because cross-cutting queries are where washed-out boundaries hide. If reporting hasn't been accounted for, the hard part hasn't been looked at yet.The data-shaped version is less satisfying to look at because it doesn't carve into a clean org chart of services. It carves along where the data will actually let go, which is messier and sometimes leaves a domain split across two services longer than you'd like. But it has the one property that matters: it moves, and the diagram-shaped version doesn't.The tax nobody quotes you.There's a cost that never makes it onto the board, and it's the one I most wish we'd priced in from the start. In the monolith, a cross-cutting query was one join. A report that spanned orders, customers, and billing was a single statement against a single database, effectively free, and the product was built on the assumption that it always would be.Once you split those into three services with three databases, that join is gone, because the tables no longer live together, and the services don't share a database on purpose.-- in the monolith: one query, one database, effectively freeSELECT o.id, c.name, i.totalFROM orders oJOIN customers c ON c.id = o.customer_idJOIN invoices i ON i.order_id = o.idWHERE o.created_at > ?;To get the same result after the split, you have to rebuild it somewhere else, either as a reporting pipeline that pulls from every service into a separate store or as a materialized view that recombines the split data so the old style of query still has something to run against. Either way, you're now paying for it: extra infrastructure to run, another moving system to keep working, and a maintenance cost that never ends. That last one bites hardest, because the view or pipeline has to stay aligned with 30 services whose schemas now evolve independently, on their own teams' schedules, without asking you first.We ended up needing both for different jobs. The materialized view carried the interactive side, the screens where a user still expects to filter across what used to be one database and get an answer back straight away. It recombines the split data so those legacy-style queries still have something to run against. The reporting pipeline carried the heavy analytical side, pulling from each service into a separate store that could be queried hard without touching the live services.Splitting the work that way is reasonable, but it doubles the surface that has to stay aligned. Every time a service changes a column the view or the pipeline reads, someone has to keep the recombination honest, and that someone is usually not the team that made the change.When it does fall out of alignment, it tends to fail quietly, as a report that's subtly wrong or a screen that drops rows nobody notices until someone reconciles the numbers by hand. That kind of maintenance never shows up in the migration estimate, and it runs indefinitely, because the product still expects one database at the edges and something has to keep faking that for it.This is the part that reframed how I think about legacy complexity in general. We tend to talk about legacy systems getting "complex" as if the code rotted on its own, but a lot of the time it's simpler than that. The product and the design were built on a cost model, and the free cross-domain join was a huge part of it. A decade of features was designed assuming it. Then the architecture changed underneath them and quietly deleted that assumption, so now every one of those features is paying a tax it was never designed to pay. The complexity isn't rot. It's a stack of features carrying a bill nobody quoted when they were built.What it bought, and what it didn'tEven with that tax, it was worth doing, and I don't want to undersell that. Splitting the hot domains out let us scale them on their own instead of scaling the whole monolith to feed one busy corner. Giving teams real ownership of their services sped their cycles up and made them more predictable, and the bug count came down as the blast radius of any one change shrank. Those gains are real, and they're the reason the last attempt was the one that stuck.What it hasn't bought us yet is a finished modernization. The work is still deep in progress, though the data decomposition is mostly through, which was the part that had killed every earlier attempt, so that is the big one behind us. The cross-cutting and reporting side, on the other hand, is a standing trade-off we carry now rather than a solved problem, and honestly, it's the part we'll be watching for a long time. The floor is much better than the monolith's, but the ceiling is a cost we chose knowingly and still have to keep managing.What actually generalizesService boundaries are domain boundaries, and the domain is where they should be reasoned about. But in a system old enough to need decomposing, the domain boundaries have already blurred, and the data is where you find out whether the boundaries you believe in are real. So plan in domain terms, but let the data confirm or break each boundary before you commit to it, one reversible step at a time. A decomposition is a technical roadmap, not a diagram you execute.It also helps to price the tax before you start rather than after. The cross-cutting queries that were free in the monolith won't be free again, and the pipeline or view you build to fake them back is a permanent line item rather than a migration cost you pay once. Knowing that going in doesn't change whether you should split, since most of the time you still should, but it changes what you promise, what you sequence first, and how honest the plan is about what the clean architecture actually costs to run.The lesson took four tries to sink in: a decomposition plan is a claim about what the data will let you do, and the data is the only thing that can tell you whether that claim holds. When the two disagree, the data is what's real, and the sooner you let it correct you, the cheaper the correction.