SQLite runs on billions of devices and counts its installations in the trillions. Since it runs anywhere, including some satellite and spacecraft systems, it is known for its reliability and widespread use — not just around the world, but in the solar system. I won’t say “the universe” because I don’t know what aliens are using in other galaxies.Given SQLite’s ubiquity and reputation for reliability, you might wonder: Why would anyone outside the confines of a mental institution even think to rewrite it?It’s not that SQLite is doing something wrong. Rather, it’s because today’s applications need capabilities that SQLite, constrained by decades of architectural decisions, cannot easily provide. Although SQLite has achieved remarkable reach, the vision of truly having databases everywhere — powering AI agents in real time, streaming analytics at the edge and intelligent systems that adapt instantly to changing data — remains just out of reach with traditional SQLite’s limitations.SQLite: Beloved Yet LimitedSQLite used to hit the sweet spot: simple, yet powerful enough for many use cases. But that balance has shifted with the pace of evolution. What was once “just right” is now too limiting for many applications.In my view, the main limitations are performance and dependability, lack of key modern features and its synchronous design.Performance and DependabilityWait, what? Isn’t SQLite insanely fast and reliable? The issue is that there is a large gap between what SQLite can do for reads and writes.Compared to any other database, reading from SQLite is one of the fastest things there is. But SQLite’s single-writer architecture locks the entire database for writes. This makes it unsuitable for applications that need to ingest data with high throughput, like logging and at-the-edge telemetry. Even at low throughput, users report seeing the dreaded “database is locked” error, since writes coming from multiple threads cannot occur at the same time.Lack of Key Modern FeaturesSQLite lacks features that developers take for granted today. For example:Change data capture (CDC), the ability to log changes to the databaseEncryptionVector searchCDC is usually best done as a core feature of the database, while other capabilities, like vector search and encryption, could be done as extensions. SQLite does have an extension system. However, it is not memory safe and doesn’t allow the database to be changed in a powerful way. Moreover, extensions cannot register new data or index types. So while there are some decent vector or encryption extensions for SQLite, the experience of using them lags behind what you see in other databases.Synchronous DesignSQLite’s API is synchronous. This means that once it starts an operation, it will run to completion, blocking anything else. This makes it hard to port it to environments like web browsers. It is also hard to replace its backing storage with object storage — something that is becoming increasingly popular in modern databases.Isn’t Open Source the Answer?It tends to be. For popular open source projects, if you see room for improvement, you are free to make those improvements. That is what made the Linux kernel, for example, great.But SQLite is different: The project is “open source, but not open contribution.” This means that contributing to SQLite is, if not outright impossible, at least very hard.This closed development model inhibits the database’s evolution. Forking is always an option, but one that comes with a problem. While SQLite’s legendary reliability rests on an extensive test suite (reportedly containing five times more code than SQLite itself), this test suite is proprietary. This makes deep, core changes to SQLite extremely difficult.Our Tipping Point: Adding Vector SearchMy team maintained a fork of SQLite for almost two years. We had carefully extended it in many ways, but adding vector search to SQLite was an eye-opener. We wanted to make the syntax as straightforward and natural as possible. That would require changes to the bytecode generation, which we did. We are able to expose vectors as a data type, query relational and vector data together in the same table, and (as long as the query doesn’t need an index) use perfectly boring SQL syntax. Adding all of that while maintaining certainty that nothing else broke was challenging, but we pulled it offBut for searches with an index, without making very invasive changes, it was hard to achieve the syntax that we wanted:SELECT title, year FROM movies ORDER BY vector_distance_cos(embedding, vector('[4,5,6]')) LIMIT 3;And we ended up settling for:SELECT title, year FROM vector_top_k('movies_idx', vector('[4,5,6]'), 3) JOIN movies ON movies.rowid = id;The index is represented as a separate table, and we have to explicitly join it with the main table.At that point, we thought: How much effort would be required to rewrite SQLite from scratch?Turso: A Rewrite of SQLite in RustThis is where Turso enters the picture. At the end of 2024, we decided to go all in on a complete rewrite of SQLite in Rust, maintaining full compatibility while enabling fundamental architectural innovations for modern workloads. We just released a beta version and are moving fast toward our upcoming first stable release.Aside from the choice of language, Turso drifts from SQLite in key ways:Asynchronous I/O: Turso’s architecture follows asynchronous I/O, with io_uring being used as a backend on Linux. This allows Turso to be used natively in challenging environments, like web browsers.Native encryption, CDC, vector search: Turso has built-in support for encrypted databases, vector search and CDC.Concurrent writes: Backed by multiversion concurrency control (MVCC), writes in Turso are designed to proceed in parallel.Live materialized views: Turso implements incremental view maintenance techniques. This allows materialized views to be kept up to date incrementally, without the need for recomputing the entire view.Open contribution: With over 150 contributors, Turso has a thriving community of contributors who are pushing the boundaries of what it can be.Architectural Decisions … and Why Rust?There’s a lot more to say about all of these points, of course. At the upcoming P99 CONF, I’ll take a detailed look at two threads relevant for that latency-obsessed community: why Rust not Zig and how we made SQLite even faster. We’ll walk through some key design decisions such as adopting an async architecture, making multithreading a first-class citizen and using incrementalism to avoid recomputing heavy operations every time.The conference is free and virtual (free passes at p99conf.io), so I hope all you SQLite, Rust and low-latency programming enthusiasts will be there live and spark some friendly debate in the legendary P99 CONF chat.The post Why We Created Turso, a Rust-Based Rewrite of SQLite appeared first on The New Stack.