The key to making data useful is by adding meaning and context. Data experts can extract meaningful information from the organization’s data using built-in metadata mechanisms, such as SQL-based (Data Definition Language) DDLs in RDBMSs, semi-structured schemas (JSON, AVRO, XML) for semi-structured data sources, or a combination of both. This process can be time consuming and does not give non-technical users the independence to obtain accurate responses on their own.  

Just as the average human user is unable to extract meaningful information from metadata, LLMs face the same difficulty. Built-in metadata in its raw form is too obscure to provide meaningful context for LLMs. Lacking this context, LLMs struggle to deliver accurate answers to questions which rely on structured data.

eRAG’s Reasoning Engine fills this gap, by:

• Extracting metadata and enriching it with organizational context
• Enriching the data with meaningful table/column names and descriptions, formulated value formats, dictionaries and usage instructions
• Storing this enriched knowledge in an internal knowledge graph layer 
• Enabling additional calibration using organizational know-how from data experts and users (human in the loop)

With its ability to connect to multiple databases simultaneously, eRAG automatically generates reasoning for each data catalog. Using the built-in metadata (e.g., DDL) as a starting point, eRAG creates an AI-enriched knowledge base of the metadata including meaningful table and column names and descriptions, formulated value formats, dictionaries and usage instructions.

While the heavy lifting of reasoning generation is automatic, ‘human in the loop’ plays a significant role in calibrating the results to best fit the specific domain knowledge and organizational know-how. 

Explicitly, a data expert assigned to calibrate the model may edit the AI-generated reasoning at any point in time, to fine tune and further enrich the model with relevant information. Implicitly, feedback provided by conversing users is also taken into account for calibration.