Large Language Models (LLMs) are powerful tools for natural language processing, but they have inherent limitations when dealing with structured relational data. Many assume that providing an LLM with a database schema and a natural language question is enough for it to generate correct SQL queries. However, this approach overlooks the fundamental nature of LLMs: they are designed to understand and generate natural language, not to inherently grasp the intricacies of database schemas, table structures, and field relationships.

This blog explores why an LLM alone is insufficient for accurate SQL generation and why a proper solution requires a semantic reasoning layer that offers rich contextual understanding of database objects. By providing natural language explanations of tables, fields, keys, and values, filtering relevant schema elements dynamically, and continuously updating semantic information, we can create a system that bridges the gap between natural language and structured data. This ensures higher accuracy, reliability, and efficiency—critical requirements for any enterprise-grade data querying solution.

We will use examples from publicly available ERP schemas and sample databases to illustrate these points, emphasizing the need for structured semantic enrichment in AI-driven data querying.

The Challenge: One-Word Labels with Deep Business Meaning

Relational databases are designed to store structured, highly accurate data, with each table, view, field, and key carefully named according to business logic. However, these names are typically single words or abbreviations that carry tremendous business meaning—meaning that an LLM cannot infer from the schema alone.

For example, in an ERP system like SAP or Oracle ERP, a table named ORDERS might store sales transactions, while CUSTOMERS holds client information. The field STATUS in ORDERS might store values like P for pending, C for completed, and X for canceled. Without a semantic reasoning layer explaining what these values represent, an LLM may misinterpret the meaning and generate incorrect SQL queries.

Example: Misinterpreting Schema Naming Conventions

Consider an SQL query request:

User Query: “Retrieve all completed orders placed by VIP customers last month.”

A simple schema might look like this:

TABLE ORDERS (
    ORDER_ID INT,
    CUSTOMER_ID INT,
    STATUS CHAR(1),
    ORDER_DATE DATE
);

TABLE CUSTOMERS (
    CUSTOMER_ID INT,
    CUSTOMER_TYPE CHAR(1)
);

A naive LLM-based approach might generate:

SELECT * FROM ORDERS WHERE STATUS = ‘COMPLETED’ AND ORDER_DATE >= ‘2024-01-01’; This query is incorrect for two reasons:

1. The STATUS field does not use COMPLETED as a value; it uses C.
2. The CUSTOMER_TYPE field in CUSTOMERS determines VIP status, but the LLM did not join ORDERS with CUSTOMERS.

Without semantic reasoning, the LLM fails to interpret that STATUS = ‘C’ represents completion and that CUSTOMER_TYPE = ‘V’ indicates a VIP customer. A well-designed system should provide the LLM with a mapping of field values and business rules to ensure accuracy.

The Importance of a Semantic Reasoning Layer

To bridge the gap between relational database structures and natural language understanding, a semantic reasoning layer is necessary. This layer provides contextual information about tables, fields, and relationships, allowing the LLM to generate more accurate queries.

Key Components of Semantic Reasoning

1. Table and Field Descriptions: Each table, field, and key should have a natural language explanation. For example:
   • ORDERS.STATUS → “Order status: ‘P’ = Pending, ‘C’ = Completed, ‘X’ = Canceled.”
   • CUSTOMERS.CUSTOMER_TYPE → “Customer type: ‘V’ = VIP, ‘R’ = Regular.”

1. Business Rules and Constraints: The reasoning layer should define logical relationships, such as “Only CUSTOMERS with CUSTOMER_TYPE = ‘V’ should be included when searching for VIP customers.”
2. Example Queries and Mappings: Providing examples of valid SQL queries mapped to similar natural language questions can guide the LLM in generating accurate queries.
3. Filtering Mechanisms: If a database has 1,000 tables, sending the entire schema to an LLM for every question is inefficient. Instead, the system should intelligently filter and send only the relevant tables, keys, and constraints based on the question.

Example: Semantic Reasoning in Action

If we enhance our system with semantic reasoning, it can generate the correct SQL for the previous example:

SELECT * FROM ORDERS
JOIN CUSTOMERS ON ORDERS.CUSTOMER_ID = CUSTOMERS.CUSTOMER_ID
WHERE ORDERS.STATUS = 'C'
AND CUSTOMERS.CUSTOMER_TYPE = 'V'
AND ORDER_DATE >= '2024-01-01';

This improved accuracy comes from having explicit mappings and constraints in the semantic reasoning layer.

Addressing the Inconsistency of LLM Responses

Unlike relational databases, which are deterministic and return exact results, LLMs generate responses probabilistically. This means that the same input might not always yield the same SQL query. A robust AI-driven SQL generation system must mitigate this inconsistency by implementing:

• Query Validation Mechanisms: Pre & Post-processing SQL statements and outputs to check for errors, expected resultset size and classifying the query’s complexity before execution.
• Audit and Security Controls: Ensuring queries adhere to security policies and do not expose sensitive data, providing organizations with control, visibility and management capabilities to align with known security best practices as well as descriptive AI to understand why a specific prompt generated that specific query.
• Feedback Loops: Allowing users to refine queries and correct mistakes, improving future responses.

The Role of Dynamic Semantic Updates

A key advantage of a semantic reasoning layer is its ability to evolve. Businesses frequently update database schemas, add new fields, and change value mappings. The reasoning layer should support dynamic updates to maintain accuracy.

Example: Handling Code Lists Dynamically

An ERP system might store order priority levels as follows:

PRIORITY_LEVEL
----------------
H = High
M = Medium
L = Low

If a company introduces a new priority level (E = Emergency), the semantic layer should be updated so the LLM understands this new classification. Without this update, queries involving priority levels might become inaccurate.

Scalability and Performance Considerations

For an AI-powered SQL system to be scalable and efficient, it must:

• Optimize Schema and Example Filtering: Instead of passing 1,000 tables to the LLM, the system should identify and send only the necessary four tables for a specific query.
• Use Caching and Preprocessing: Frequently asked questions should have precomputed query templates to reduce LLM processing time.

Implement Efficient Parsing Pipelines: The system should intelligently parse user queries, identify required tables, and construct optimal SQL queries.

Conclusion

LLMs are not inherently designed to interpret database schemas accurately. Relying solely on sending an LLM a schema and expecting perfect SQL queries is a limited and somewhat flawed approach. Instead, a robust AI-driven SQL generation system must incorporate at least a semantic reasoning layer, such as that provided by GigaSpaces eRAG that enriches database metadata with business context. To provide a holistic solution addressing security, compliance, scale, performance and accuracy – an end-to-end design, such as eRAG, must be implemented.