Everyone wants their database to scale. In fact, if you are old enough, you might even remember when saying your database was web scale was all the rage.Xtranormal screenshotIn principle, “scale” just means the ability to do more. But more what? More data? More queries? Talking about scale only makes sense if we define the axis on which scaling takes place.It’s time to move beyond thinking of databases in terms of just capacity and throughput. The rise of AI agents requires a new axis of scalability: How many databases can you create and maintain? Multitenancy is ending; the age of hyper-tenancy has begun.Sponsor note: If you’re into database scalability, there’s a whole conference for that! Monster Scale Summit is a free + virtual conference on extreme scale engineering, with a focus on data-intensive applications. Join your peers and learn from luminaries like antirez, creator of Redis; Camille Fournier, author of “The Manager’s Path” and “Platform Engineering”; Martin Kleppmann, author of “Designing Data-Intensive Applications” and more than 50 others, including engineers from Discord, Disney, Pinterest, Rivian, Datadog, LinkedIn and UberEats. Register and join the community for some lively chats. Changing Dimensions for Scalability: Two ExamplesThis certainly wouldn’t be the first time that new technologies triggered us to rethink scalability.Around 2005, booting your typical Linux distribution took more than a minute. The kernel alone took dozens of seconds to initialize and, after that, myriad services had to be brought online. That was a mostly sequential operation, driven by handcrafted bash scripts (SysV Init!).There were many efforts to reduce that time, led by engineers (myself included) who simply couldn’t stand it. It offended our engineering sensibilities. You know engineers — if something can be fast, why would we let it be slow?But there was no real pressure to improve this metric, no real business need. As a result, many in the community saw those efforts as futile. After all, in server hardware, the firmware sanity checks alone sometimes took more than that. And even on desktop PCs, how often do you boot your computer anyway?Everybody wanted Linux to be fast and wanted Linux to scale, but that meant executing specific system calls fast, being able to run multiprocessor systems, etc. Time to boot was never a part of the scalability axis.What changed the calculation was the rise of virtualization, the technology that led the cloud revolution. Machines were no longer physical entities; they were logical entities that were created on demand. Initially, those machines just played the same role as the old physical ones, except much cheaper and with more flexibility. But that soon started to change.Taking advantage of the new capabilities afforded by the cloud, developers started running all sorts of ephemeral machines: They would come online, execute a specific task, then go away. Suddenly, you had an army of those machines instead of the one big server. The business pressure to make the operating system boot time a part of the scalability matrix was suddenly real. Today, for some specialized applications, especially those being deployed in unikernels, the time to boot can be around a dozen milliseconds.A transistorAn even more dramatic example comes from the hardware itself. The first commercial transistors were around 10mm in size. That is, by all worldly standards, small. But there was no meaningful pressure to make it even smaller.Engineers working on transistors wanted to make them better, but the axis on which “better” was judged was just different. For example, one could measure how the switching speed (how fast the transistor could switch a high-frequency signal) or the noise figure (the amount of noise introduced in the circuit by this element). But how physically small they were? Not a factor.Size only started to become a concern with the rise of integrated circuits, the grandfather of modern CPUs (and now GPUs). As miniaturization became an important aspect, the transistor size became a key business concern. Today, a modern CPU holds more than 200 billion transistors, each of them in the order of a couple of nanometers — 1000 times smaller than their predecessors.A New Scalability Axis for DatabasesIn each case, the pattern is identical: A technological disruption fundamentally changes the way we interact with and think about an existing building block. Suddenly, a new axis of scalability emerges.Databases are facing similar pressures today, driven by the rise of agentic systems. Up until recently, the term “database” was commonly preceded by the definite article: “the database.” You write an application and that application has a database that goes with it. Scaling “the database” traditionally meant one of two things: either store more data or execute more queries.But the rise of AI agents changes the game. Agents are spun up by the millions. Each agent is responsible for a piece of a larger task, with some agents having the task of coordinating the work of their subagents. Some agents exist for just a couple of seconds. And all of them have data needs that are private to the agent. Which tools were called? Did the tool succeed? What was the result of the last tool call? Which files were generated? Which files, data and metadata can be added to the context of this agent to improve its performance?Traditional multitenancy breaks down when you need microsecond provisioning and strict data isolation guarantees. This calls for hyper-tenancy, a much finer-grained isolation with a lot more elastic scalability.Agent builders want this data to be quickly available during the agent operation, as well as summarized for later use for auditability and observability reasons. Some agents deal with sensitive data that must be encrypted, isolated and never leave the context of that one agent.Consider a customer service AI handling a complex request. It spawns one agent to check inventory, another to verify the customer’s purchase history and a third to calculate refund amounts considering current promotions. A coordinator agent manages these three, while a compliance agent logs every decision for regulatory audit. Each exists for a couple of seconds. Each needs to track its state, cache its context and record its tool calls. Once a new user opens a new ticket, a whole new collection of agents gets created. Now imagine that serving millions of users. And that is for a single system in an organization.Databases do all of that — and have been doing this for the past five decades. But the time it takes for a database to come online, and the number of databases you can maintain at once, was never a pressing concern — until now.Databases will face the pressure to provide trillions of instances, working independently. Picture a fleet of hundreds of agents serving millions of users and now multiply that by all the companies across the economy.Traditional multitenancy — sharing one database with logical separation — breaks down when you need microsecond provisioning and strict data isolation guarantees. This calls for hyper-tenancy, a much finer-grained isolation with a lot more elastic scalability.Databases will face the pressure to provide trillions of instances, working independently. They will evolve from being deployed to being spawned. If the “trillions” order of magnitude sounds absurd, just look at our previous examples and imagine we are living at the beginning of the agentic curve. Picture a fleet of hundreds of agents serving millions of users and now multiply that by all the companies across the economy.The Right Tool for the JobAs it turns out, a database that can be spawned by the trillions already exists: SQLite. On its website, SQLite claims a total of trillions of installations worldwide, making it the most deployed database in the world. It can do this because it doesn’t follow the traditional client-server model: It is just a file, accompanied by an in-process library code. SQLite has also inspired a generation of other in-process databases with the same model, like Turso and DuckDB.To understand why this model works so well, consider what an agent actually needs from its local, private database:Instant availability: When an agent spawns, its database must already be there. Not “available after connection establishment,” but there. The moment you have a file handle, you have a database.True isolation: Each agent’s data must be separate from every other agent’s data. Separate tables could work, but then you would need billions of tables. Separate database files allow for hard isolation between agent data, with the added benefit of being easy to encrypt data for different agents with separate encryption keys.Co-location: The database must live where the agent lives. If the agent is in a container, the database is in that container. If the agent runs in a browser, the database runs in that browser. If the agent moves to a different machine, the database moves with it. Data and computation travel together. As agents get deployed everywhere, databases must follow them everywhere.Zero coordination: An agent shouldn’t need to negotiate with a database cluster about connections, transactions or resource limits. It owns its database completely. No connection pools to exhaust. No other tenants to affect performance. No shared fate.Client-server databases simply can’t provide these properties, no matter how well-optimized they are. In-process databases can.The GapBut here’s the challenge: SQLite was designed for a different era and has been slow to evolve. It’s brilliant at what it does, but agents need capabilities that go beyond what it can do. They need:Vector search: Agents work with embeddings constantly. Semantic search over documents, similarity matching for Retrieval-Augmented Generation (RAG), clustering of results. SQLite has no native understanding of vector operations. You can bolt on extensions, but they’re not first-class features integrated into the query planner.Native encryption: SQLite can encrypt data, but it requires extensions and key management becomes the developer’s problem. Agents dealing with sensitive data need encryption that’s transparent, with keys that live and die with the agent life cycle.Observability: When you have billions of database instances, you can’t SSH into each one to debug. You need structured logging, metrics and traces that aggregate automatically. SQLite is silent, optimized for embedded contexts where observability was someone else’s problem.Network capabilities: While the core database should be embedded, agents often need to sync state, replicate to durable storage or coordinate with other agents. SQLite is purely local. The moment you need to move data between machines, you’re on your own.Developer abstractions: Agent builders aren’t database administrators. They need high-level APIs that handle common patterns: “Give me a database for this agent,” “Sync this data when you can,” “Clean up everything when I’m done.” SQLite gives you SQL and file handles. You build everything else.The New GenerationLooking at the past for lessons on what to expect for the future, I believe we will see the emergence of a new category: databases that inherit SQLite’s embedded architecture, but evolve it for modern needs. These systems are forged to be building blocks for larger systems, not monolithic solutions. They provide the primitives that agent frameworks need: instant instantiation, built-in encryption, vector-native operations and automatic synchronization.Databases like DuckDB fit the bill. DuckDB, often dubbed “SQLite for OLAP,” is an in-process online analytical processing (OLAP)-oriented database that provides local, actionable intelligence for agents. But there’s also a need for an online transaction processing (OLTP) equivalent that will play the same role as the original SQLite.That’s what led to the creation of Turso, an OLTP-oriented database that is fully file- and API-compatible with SQLite. It’s a full modern rewrite in Rust that aims to provide a more modern and feature-rich telling of what SQLite can be (with native encryption, change data capture, vector search, among others), while keeping the file-based nature of SQLite.I, for one, am excited about the future!The post AI Agents Create a New Dimension for Database Scalability appeared first on The New Stack.