RAG vs Agentic RAG: when retrieval is no longer enough

💥 The question that breaks classic RAG

The repository's demonstration case is a multi-hop question: answering it takes two different documents — the 2024 return policy and the 2025 one — plus a comparison between them. The classic pipeline turns the question into one embedding and retrieves the top-k most similar chunks: if the query “compare the policies” looks more like the 2025 document than the 2024 one, part of the evidence may never reach the model.

The problem is neither the vector database nor the model: it's the architecture. A linear pipeline has no point where it can notice that retrieval went poorly. No retry, no evaluation, no second query. Whatever comes in on the first shot is all the model will ever see.

• Single query: a composite question gets squashed into one embedding.
• No evaluation: nobody checks whether the retrieved chunks are enough.
• No retry: poor retrieval means a poor answer. The end.

📏 Classic RAG: the linear pipeline

The naive pipeline is this, no frills: embed the question, one hybrid search on Qdrant, chunks stuffed into the prompt, answer. In the repo it's a service with about thirty lines of logic: that's the beauty — and the limit — of this approach.

The search underneath is already hybrid (dense vectors + BM25-style sparse, fused with server-side Reciprocal Rank Fusion on Qdrant): the baseline retrieval is good. But however good the single shot is, it remains a single shot.

// 1. Retrieve — one shot, no second chances.
var chunks = await vectorSearchService
    .HybridSearchAsync(request.Question, topK, ct);

// 2. Stuff everything into the prompt.
var context = string.Join("\n---\n",
    chunks.Select(c => $"[{c.Source}#{c.ChunkIndex}]\n{c.Text}"));

List<ChatMessage> messages =
[
    new(ChatRole.System,
        "Answer using ONLY the context below. If the context is not " +
        "enough to answer, say so explicitly. Cite sources as " +
        $"[file#chunk].\n\nContext:\n{context}"),
    new(ChatRole.User, request.Question)
];

// 3. Answer. Whatever was retrieved is all the model will ever see.
var response = await chatClient.GetResponseAsync(messages, cancellationToken: ct);

🤖 Agentic RAG: the LLM drives retrieval

In Agentic RAG the flow can become iterative: besides the question, the model receives three tools and a strategy in the system prompt — split the request into focused queries, search, evaluate whether the evidence is enough and, when needed, refine. The model chooses whether and when to invoke them.

The conceptual difference matters: the classic pipeline is linear — input, retrieval, answer. The agentic one makes a loop with explicit evaluation available. But the code does not guarantee that the model will execute every step or call the judge before answering: these are instructions and tools, not a deterministic workflow.

🛠️ The three tools: search, evaluate, refine

The tools are C# methods with a Description attribute: search_docs runs hybrid search on Qdrant, rerank_evaluate asks an auxiliary LLM call for a verdict — SUFFICIENT or INSUFFICIENT — on the chunks collected so far, and refine_query rewrites a query that performed poorly into one to three alternatives. Despite its name, rerank_evaluate does not reorder the chunks: it only judges sufficiency; final sources remain ordered by Qdrant score.

The run state — collected chunks, step trace, tokens spent in the auxiliary calls — lives in a per-request context shared by the three tools. No framework: just a class with three methods and their descriptions.

[Description("Search the company knowledge base with a focused query. " +
             "Returns the most relevant document chunks. Call it multiple " +
             "times with different queries for multi-part questions.")]
public async Task<string> SearchDocsAsync(
    [Description("A short, focused search query")] string query,
    [Description("How many chunks to retrieve")] int topK = 5,
    CancellationToken ct = default)
{
    var chunks = await vectorSearchService.HybridSearchAsync(query, topK, ct);
    // ... deduplication, result construction and trace as in the repo
    return result;
}

[Description("Evaluate whether the chunks retrieved so far are sufficient " +
             "to fully answer the user's question. Returns a verdict " +
             "(SUFFICIENT or INSUFFICIENT) with a short reason.")]
public async Task<string> RerankEvaluateAsync(
    [Description("The original user question")] string question,
    CancellationToken ct = default)
{
    // ... message construction and LLM call as in the repo
    return response.Text;
}

[Description("Rewrite a query that returned poor or incomplete results " +
             "into up to three better search queries.")]
public async Task<string> RefineQueryAsync(
    string originalQuery, string reason,
    CancellationToken ct = default)
{
    // ... LLM call producing one to three alternative queries
    return response.Text;
}

🔁 The agent loop with Microsoft.Extensions.AI

The model → tool → model cycle is handled by function invocation in Microsoft.Extensions.AI. I register the client with UseFunctionInvocation and pass the three tools in ChatOptions: the middleware continues while the model produces function calls, then returns the final answer or stops at the configured limit.

The demo configures five total iterations, while the system prompt asks the model to retry at most twice. The middleware does not guarantee that search_docs, rerank_evaluate and refine_query will be called or used in the suggested order. No additional agent framework is required, but behavior remains model-driven.

// In InfrastructureServiceExtensions, Ollama branch:
services.AddChatClient(
        serviceProvider => (IChatClient)new OllamaApiClient(
            serviceProvider.GetRequiredService<IHttpClientFactory>()
                .CreateClient(OllamaHttpClientName),
            configuration["AI:Ollama:ChatModel"]!))
    .UseFunctionInvocation(configure: c =>
        c.MaximumIterationsPerRequest = maxIterations);

// In the agentic service, per request:
var options = new ChatOptions
{
    Tools =
    [
        AIFunctionFactory.Create(context.SearchDocsAsync, "search_docs"),
        AIFunctionFactory.Create(context.RerankEvaluateAsync, "rerank_evaluate"),
        AIFunctionFactory.Create(context.RefineQueryAsync, "refine_query")
    ]
};

// UseFunctionInvocation() runs the loop: model -> tool -> model -> ...
var response = await chatClient.GetResponseAsync(messages, options, ct);

🪵 How to read an agentic run trace

The demo records every tool call through Serilog, includes it in the response's trace field and persists the completed run to MongoDB. Each entry contains the step, tool name, arguments, a result summary and duration. For search_docs, that summary stores only chunk counts, not chunk text; rerank_evaluate and refine_query can instead retain their complete short responses. The sample below illustrates the format documented by the repository; it is not output from a versioned end-to-end test.

The trace helps explain which query ran and what verdict the evaluator produced. In production, however, queries, answers and results can contain sensitive data: redact or exclude them from logs and apply appropriate levels, access controls and retention.

[10:42:01 INF] AGENT step 1 | search_docs(
  query="return policy 2024" topK=5) -> 5 chunks (5 new) (38 ms)
[10:42:02 INF] AGENT step 2 | search_docs(
  query="return policy 2025" topK=5) -> 5 chunks (4 new) (35 ms)
[10:42:05 INF] AGENT step 3 | rerank_evaluate(
  question="Compare..." chunks=9) -> SUFFICIENT: both policies retrieved (2810 ms)

📊 Benchmark: same questions, two pipelines

In the repo, POST /benchmark/run reads six questions — two simple, two multi-hop, one tricky and one out-of-domain — and runs each sequentially through the naive pipeline and then the agentic one. This produces twelve individual documents in the runs collection and one summary document in benchmarks, available through GET /benchmark/latest.

For each execution the demo records latency, input and output tokens, LLM calls and tool calls. It does not include ground truth, an evaluator or accuracy: it measures operational behavior but does not prove that one answer is better. A serious comparison must fix the model, quantization, hardware and dataset, repeat runs, and evaluate correctness and completeness too.

6 questions × 2 pipelines = 12 sequential runs

runs:
  mode · question · answer · sources · trace · metrics

benchmarks:
  ranAt · chatModel · 12 entries

metrics:
  durationMs · inputTokens · outputTokens · llmCalls · toolCalls

answer quality: not evaluated automatically

⚖️ When to use which (an honest decision table)

Agentic RAG is not an automatic upgrade: it can require more calls, tokens and latency and introduces a new error surface because the model may skip or misuse tools. The right question isn't “which one is better?”, it's “does this question actually benefit from the loop?”.

My practical rule is to start naive, define a quality evaluation and measure. I switch to agentic only where results show incomplete retrieval and a repeatable benefit. A router in front of both pipelines is a possible evolution, but it is not implemented in this demo: the repository exposes two separate endpoints so you can experiment on your own corpus.

• Stay naive when: pointed questions on a single document, small homogeneous corpus, latency and cost matter more than completeness.
• Evaluate agentic RAG when: multi-hop or comparative questions, multiple sources to cross-reference, measured retrieval gaps.
• Prerequisite: a model that can actually do tool calling — it's the engine of the loop, not an optional.
• In production: router in front, naive by default, agent on demand. And an iteration cap, always.

🚀 The repo: two endpoints, one docker compose

All the code is public: .NET 9 Minimal API, Microsoft.Extensions.AI as the LLM abstraction (local Ollama by default, OpenAI via configuration), Qdrant for hybrid search, MongoDB for runs and benchmarks, demo dataset included. Docker Compose starts Qdrant and MongoDB; Ollama runs separately on the host. The manual POST /admin/seed call recreates the collection and re-ingests the seven Markdown documents.

The most instructive way to use it: same question on both endpoints, then compare trace and sources in the two responses. The difference between “it retrieved what was there” and “it searched for what was needed” shows better in a diff than in a thousand words.

git clone https://github.com/fscamuzzi/rag-vs-agentic-rag && cd rag-vs-agentic-rag
docker compose up -d              # Qdrant + MongoDB
ollama pull llama3.1:8b && ollama pull nomic-embed-text
dotnet run --project src/RagVsAgenticRag.Api

curl -X POST localhost:5210/admin/seed
curl -X POST localhost:5210/rag/naive   -H 'Content-Type: application/json' \
  -d '{"question":"Compare the 2024 and 2025 return policies"}'
curl -X POST localhost:5210/rag/agentic -H 'Content-Type: application/json' \
  -d '{"question":"Compare the 2024 and 2025 return policies"}'
curl -X POST localhost:5210/benchmark/run