Today, more and more customers are embracing AI with one clear goal in mind: they want to chat with their data. And by "data," they mean all of it: structured tables, unstructured documents, images, audio, and more.This demand has given rise to hybrid approaches that blend semantic understanding with precise data retrieval. But here’s the challenge: while embeddings and vector search play a critical role in semantic search, they’re not enough on their own — especially when the question is highly specific and demands an exact answer.As I discussed in Improve the “R” in RAG and Embrace Agentic RAG in Azure SQL article, a smarter, multi-step approach is needed.There’s a code sample waiting for you at the end—but don’t scroll just yet! This article lays the foundation you’ll need to really understand the strategy behind it. That way, you’ll be able to apply the same pattern to your own data, not just the example. So read through it all… and no, asking an AI to summarize it won’t cut it this time 😉Two Kinds of Questions, Two Kinds of AnswersWhen users ask a question, it typically falls into one of two categories: Unstructured Data Search: The question targets unstructured content (text, image, audio), and the goal is to find semantically similar results — this is where vector search and embeddings shine. Precise, Structured Query: The question is clear-cut and requires an exact answer from structured data. Embeddings offer little help here. Instead, we need to generate and run a valid SQL query against the correct set of tables.This post focuses on the second case, as it's arguably the most complex.Getting a precise, correct answer from a database using natural language is where LLMs struggle the most. They're inherently designed to provide semantically relevant responses, not necessarily factually or structurally accurate ones. That’s why this problem requires a carefully crafted multi-step approach, where the LLM is guided to do what it does best while minimizing the risk of error.So how do you apply this to a real-world enterprise database with hundreds of tables?“Divide and Conquer” Still WorksThe classic strategy of "divide and conquer" is your best friend here. You can’t simply dump the full schema of 100+ tables into a large language model (LLM) and expect magic. LLMs have limitations — especially when it comes to context window size and relevance filtering.The more irrelevant data you include, the more noise you'll actually generate and so the less effective the model becomes. If you’re asking a question about product pricing, providing the LLM with address type definitions only adds confusion. So, the problem needs to be solved with two separate steps: first, help the LLM to figure out what table are really needed, second help the LLM model to have detailed understanding of the schema of those tables, to that the best SQL query can be generate.Agent 1: Create a high-level, domain expertStart by giving the LLM a high-level map of your schema. Not every column, not every data type — just the purpose of each table.Here’s an example from the AdventureWorks2022 database, that can be used to define the prompt for the first agent:[Production].[ProductListPriceHistory]: Changes in the list price of a product over time.[Person].[Address]: Street address information for customers, employees, and vendors.[Sales].[SpecialOfferProduct]: Cross-reference table mapping products to special offer discounts.[Production].[ProductModel]: Product model classification.[Person].[AddressType]: Types of addresses stored in the Address table.[Person].[StateProvince]: State and province lookup table.The prompt would look like the following:You are an AI assistant that helps users to query the database. The tables in the database, with the related description, are:{await database.GetTablesAsync()} Use a professional tone when aswering and provide a summary of data instead of lists. If users ask about topics you don't know, answer that you don't know. Today's date is {DateTime.Now:yyyy-MM-dd}. You must answer providing a list of tables that must be used to answer the question and an explanation of that you'll be doing to answer the question.You must use the provided tool to query the database.If the request is complex, break it down into smaller steps and call the plugin as many time as needed. Ideally don't use most than 5 tables in the same query. With this context, you ask the LLM to identify the most relevant tables needed to answer a specific question. Think of the LLM in this phase as a knowledgeable data architect — someone who knows where the data lives, but not exactly how it’s structured inside each table.Agent 2: Create a SQL expertOnce you have the list of candidate tables, then — and only then — you provide full schema details: column names, types, relationships, and even quirks like embedded JSON blobs in “details” columns.This is your opportunity to give the LLM the context it actually needs, and nothing more. If a column holds JSON, you should explain its structure and provide examples of how to extract data from it.Here’s a sample of how you might present that information to the LLM for the [Sales].[SalesTerritory] table:[TerritoryID] int -- Primary key for SalesTerritory records.[Name] Name -- Sales territory description[CountryRegionCode] nvarchar -- ISO standard country or region code. Foreign key to CountryRegion.CountryRegionCode.[Group] nvarchar -- Geographic area to which the sales territory belongs.In this second agent, the prompt is the following:You create T-SQL queries based on the given user request and the provided schema. Just return T-SQL query to be executed. Do not return other text or explanation. Don't use markdown or any wrappers.The schema is provided in the format: Table1Name: Column1Name (Column1Type) -- Column1DescriptionColumn2Name (Column2Type) -- Column2Description...ColumnNName (ColumnNType) -- ColumnNDescriptionTable2Name: Column1Name (Column1Type) -- Column1DescriptionColumn2Name (Column2Type) -- Column2Description...ColumnNName (ColumnNType) -- ColumnNDescriptionThe schema for the avaiable tables is the following: {table_schemas.ToString()} Generate the T-SQL query based on the provided schema and the user request. The user request is in the next message.This type of structured metadata, minimal yet meaningful, gives the LLM enough signal to reason about what this table is for, without overwhelming it with noise. You can then repeat the process for other selected tables, tailoring the level of detail based on what the question requires.By feeding the model only relevant, targeted information like this, you dramatically reduce confusion and increase the accuracy of the generated SQL query. You have now created a SQL expert that knows exactly how to query the involved tables.Making the Agents Working TogetherThis is where the Agentic approach shines. Think of this system as having two expert agents, each with a specific role: Agent 1 is the domain expert. It knows the full database layout at high-level and can identify which tables are relevant based on the user's question. Agent 2 is the query expert. It writes SQL queries using only the relevant tables and a clear goal, as instructed by Agent 1.Here’s how a real interaction might play out:Human: Hi, I want to know who is the most profitable customer.Agent1: I need to query the database using [Sales].[Customer], [Sales].[SalesOrderHeader], [Sales].[SalesOrderDetail]. Asking Agent2 to: "Identify the customer who has contributed the highest total profit. Sum up the profit for each customer using order amounts and subtracting related costs if available. Return the most profitable customer details including customer ID, name if available, and total profit."Agent2: Running the following query:SELECT TOP 1 C.CustomerID, C.AccountNumber, SUM(SOD.LineTotal) AS TotalProfitFROM [Sales].[Customer] AS CJOIN [Sales].[SalesOrderHeader] AS SOH ON C.CustomerID = SOH.CustomerIDJOIN [Sales].[SalesOrderDetail] AS SOD ON SOH.SalesOrderID = SOD.SalesOrderIDGROUP BY C.CustomerID, C.AccountNumberORDER BY TotalProfit DESC;Answer: The most profitable customer is the one associated with Customer ID 29818 and Account Number AW00029818. This customer has contributed a total profit of approximately $877,107.19 to the business. If you need further details about this customer, let me know!Human: Can you do a breakdown per year and then also return to me the name and some details about the customer?Agent1: Use the tables: [Sales].[Customer], [Sales].[SalesOrderHeader], [Sales].[SalesOrderDetail]. Asking Agent2 to: "Break down the total profit data of Customer ID 29818 by year and provide yearly profit details for this customer."Agent2: Agent1: Agent2, now use [Sales].[Customer], [Person].[Person] to: "Provide the name and additional details of the customer with Customer ID 29818, including contact or relevant information."Agent2: Answer: This flow mirrors real-world collaboration: schema design and data retrieval are distinct tasks, and each expert focuses on what they do best. The LLMs emulate that by acting as coordinated, specialized agents.What’s Next: From Queries to ConversationsThe approach we’ve outlined, combining schema-aware reasoning with agentic collaboration, is a powerful way to enable natural language access to structured data. But in most real-world scenarios, structured data is only part of the story.To truly “chat with your data,” you need to handle both structured and unstructured sources — and that’s where RAG pattern comes in.In practice, this means your system needs to make an early decision: Is this question best answered with a SQL query, or does it require semantic search over documents, images, or other unstructured content?This decision is the first and most critical task for the LLM. As discussed in [Improve the “R” in RAG and Embrace Agentic RAG in Azure SQL](#), the best systems combine both approaches: Use Agentic SQL generation when the question demands precision and structured data (e.g., “What was our top-selling product in Q4 2023?”). Use RAG Pattern when the question is exploratory or based on unstructured content (e.g., “What are customers saying about our new product line?”). And in some cases, combine both. For example, retrieving a list of customers from SQL, then enriching it with sentiment analysis from support tickets or reviews.This hybrid model ensures that each question is routed to the right tool for the job, maximizing both accuracy and relevance.Final WordConversational access to enterprise data isn’t just about writing better SQL or retrieving better documents. It’s about orchestrating the right tools in the right order. That’s where agentic patterns shine. By combining classic RAG with structured query generation, and by guiding LLMs through a multi-step reasoning process, you can build systems that are not only intelligent — but also trustworthy, scalable, and enterprise-ready.Let’s make data talk — no matter what form it takes.Try it yourselfThe source code, as usual, is available on GitHub, here: https://github.com/Azure-Samples/azure-sql-nl2sql