Agentic RAG with Apache Iceberg and Dynamiq on IBM watsonx.data

Agentic RAG is retrieval-augmented generation in which an AI agent, rather than a fixed pipeline, decides how to answer each question: which sources to query, in what order, and whether the results are good enough. On IBM watsonx.data, that means one agent can run SQL against governed Apache Iceberg tables and search unstructured documents in Milvus, then combine both into a cited answer. Dynamiq provides the agent layer, with guardrails, approvals and a trace of every step.

This post explains how agentic RAG differs from classic RAG, the architecture on watsonx.data, an HR assistant example, and what to watch for in production.

What is agentic RAG?

Classic RAG follows one fixed path: embed the question, retrieve similar passages, and generate an answer from them. That works for questions whose answer sits in a document. It fails when the answer needs numbers from a table, several lookups in sequence, or a check that the retrieved text actually answers the question.

Agentic RAG puts an agent in charge of retrieval:

See our glossary entry on retrieval-augmented generation for the basics.

Why combine structured and unstructured data?

Most real business questions need both. "How many vacation days do I have left, and can I carry them into next year?" needs the employee's balance from an HR table and the carry-over rule from a policy document. A document-only assistant cannot answer the first part; a SQL-only assistant cannot answer the second. An agent that can use both tools answers the whole question and cites each source.

What does the architecture look like on watsonx.data?

The design has three layers.

Structured data: Apache Iceberg tables

IBM watsonx.data is a lakehouse that uses Apache Iceberg as its open table format. Iceberg gives agentic RAG several useful properties:

• Time travel. Queries can read a table as it was at an earlier snapshot, for example to compare current and past transactions or reproduce an answer for an audit.
• Schema evolution. Columns can be added or changed without rewriting data, so new compliance fields do not break the agent's queries.
• ACID transactions. Concurrent updates stay consistent, so an agent never reads a half-written table.
• Open format. The same tables are readable by several query engines and stay in your object storage.

Unstructured data: embeddings in Milvus

Policies, contracts and other documents are parsed, split into chunks and embedded, using embedding models from watsonx.ai or another provider, and stored in Milvus, the vector database available in watsonx.data. Keeping document versions in the index lets the agent cite the policy that was in force.

The agent layer: Dynamiq

A Dynamiq agent receives the question and chooses its tools: a read-only SQL tool against watsonx.data's query engine for structured data, and a retrieval tool over the Milvus index for documents. For more complex work, a manager agent can delegate to a SQL specialist and a document specialist. Guardrails detect PII before text reaches a model, and every query, retrieval and model call is recorded in a trace.

Example: an HR policy and entitlement assistant

HR teams answer the same questions every day, and most combine a personal record with a policy: leave balances and carry-over rules, eligibility for benefits, stock plan vesting and its terms. The assistant works like this:

1. Employee records such as leave balances and plan enrollments live in Iceberg tables, with standardized schemas and consistent updates.
2. HR policies such as leave, benefits and stock plan documents are embedded into a Milvus index with their effective dates.
3. The agent receives a question, queries the employee's own records through the SQL tool, retrieves the relevant policy passages, and answers with citations to both.
4. Permissions restrict every query to the asking employee's own data, and questions outside policy are handed to the HR team.

Because the agent reads the current policy at question time, a new or changed policy is used as soon as it is indexed, without retraining a model.

What are the benefits of agentic RAG on watsonx.data?

• Complete answers. Questions that need both numbers and rules get one answer, with a citation for each part.
• Audit-ready. Traces show every query and retrieval, and Iceberg snapshots let you reproduce the data an answer was based on.
• One governed source. The agent reads the same tables as your analytics and reporting, instead of a copy that drifts.
• Fast adaptation. New policies and schema changes flow into answers without rebuilding pipelines or retraining models.
• Storage efficiency. Iceberg tables use compressed columnar files in object storage, and Milvus supports quantized indexes for large document sets.

What should you watch out for in production?

• Text-to-SQL errors. Give the agent read-only access to curated views, validate generated queries, and test with real questions before launch.
• Access control. Enforce row-level permissions in the data layer, not only in the prompt, so an agent cannot read records the user may not see.
• Personal data. Detect PII in inputs and outputs, and keep traces in the same governed environment as the data.
• Stale indexes. Re-index documents when policies change, and keep effective dates so old versions are not cited as current.
• Cost and latency. Cap the number of retrieval rounds, and route simple lookups to smaller models.

How do Dynamiq and IBM work together?

Dynamiq is an IBM AI Technology Partner. Teams can build agents and RAG applications with watsonx.ai models and watsonx.data vector stores alongside other providers, and deploy Dynamiq on IBM Cloud. Read more about the IBM partnership.

For a production example on IBM technology, see how we built a cost-aware legal research workflow with IBM watsonx, which uses Milvus on watsonx.data for internal contracts.

FAQ

What is agentic RAG?

Agentic RAG is retrieval-augmented generation where an AI agent decides how to retrieve information for each question: which sources to query, in what order and whether to try again. It can combine structured data, such as SQL tables, with unstructured documents in one answer.

What is the difference between RAG and agentic RAG?

Classic RAG runs one fixed retrieval step, usually a vector search, before generating an answer. Agentic RAG lets an agent plan multiple retrieval steps across different sources and check its results, which handles multi-step questions and mixed data types.

What is Apache Iceberg used for in AI applications?

Apache Iceberg is an open table format for data lakes. In AI applications it gives agents consistent, governed tables to query, with time travel for reproducing past answers and schema evolution for changing data without breaking queries.

Can an AI agent query a data lakehouse directly?

Yes, through a SQL tool. In production, give the agent read-only access to curated views, enforce row-level permissions in the data layer and validate generated queries before they run.

Does Dynamiq work with IBM watsonx.data?

Yes. Dynamiq is an IBM AI Technology Partner with access to watsonx.ai models and watsonx.data vector stores, and Dynamiq can be deployed on IBM Cloud.