LLM chat memory in ASP.NET Core: IChatReducer and MongoDB

🧠 The problem: the LLM remembers nothing

An `IChatClient` is stateless: every call starts from zero and only sees the messages I pass in that request. A chat's “memory” is therefore an illusion built by the caller: I keep the list of turns and replay it all to the model with every new message.

The naive approach — a `List<ChatMessage>` that grows forever — has three problems that show up in this order: tokens (hence cost and latency) grow with every turn; the model's context window eventually overflows and providers truncate or refuse; and the in-memory list vanishes at the first restart of the process. Two distinct things are needed: persistence of the history and control over how much history reaches the model.

• Tokens and latency: ever-longer prompts on every turn, linear with the number of messages.
• Context window: past the model's limit the conversation breaks, it doesn't just slow down.
• Volatility: history in RAM dies with the process; a real chat survives a deploy.

// Works. Then the conversation hits 60 turns…
List<ChatMessage> history = [];
while (true)
{
    history.Add(new(ChatRole.User, Console.ReadLine()!));

    // The WHOLE history, EVERY turn: linear tokens and latency,
    // a context window on a countdown, zero persistence.
    var response = await client.GetResponseAsync(history);
    history.AddMessages(response);
}

🏗️ Architecture: the store remembers everything, the model sees little

The pattern I implement separates two responsibilities the naive approach conflates: what I keep and what I send to the model. MongoDB holds the complete transcript of every conversation — it's the audit trail, never touched. The prompt for the model is instead rebuilt and reduced on every turn: system prompt, plus whatever view the reducer decides.

The flow of every message: load the conversation from Mongo, recompose the full prompt, pass it to the `IChatReducer`, send the model only the reduced view, persist the new turn (question + answer) with a partial update. The reduction is a read-side view: it deletes nothing from the store.

⚙️ Setup: project and packages

Three packages: Microsoft.Extensions.AI for `IChatClient` and the reducers, OllamaSharp which implements `IChatClient` on top of Ollama, MongoDB.Driver for persistence. Chat reduction is marked experimental (diagnostic `MEAI001`): I silence it in the `.csproj` with a deliberate `NoWarn` — the API may change, the pattern won't.

The project is a clean .NET 9 Minimal API: no Controllers, typed record DTOs, one service for the chat flow and one repository for Mongo. The full structure is in the repo linked at the end of the article.

$ dotnet new webapi -n ChatMemory.Api --no-https
$ cd ChatMemory.Api
$ dotnet add package Microsoft.Extensions.AI      # 10.9.0
$ dotnet add package OllamaSharp                  # 5.4.30
$ dotnet add package MongoDB.Driver               # 3.11.0

# in the .csproj: reducers are experimental (MEAI001)
# <NoWarn>$(NoWarn);MEAI001</NoWarn>

🗄️ MongoDB persistence: $push, not replace

Every conversation is one document: id, title, timestamps and the `messages` array with role, text and date of each turn. The choice that matters is how I write: append with `$push`, never replace the document. Two concurrent turns on the same conversation both get appended instead of overwriting each other, and I never rewrite bytes that didn't change.

The repository exposes three operations — create, read, append — behind an interface: in tests I swap it for an in-memory version without touching the rest of the code.

// Partial update on purpose: $push the new turn,
// never replace the whole document.
public Task AppendMessagesAsync(
    string id, IReadOnlyList<StoredMessage> messages, CancellationToken ct) =>
    collection.UpdateOneAsync(
        c => c.Id == id,
        Builders<Conversation>.Update
            .PushEach(c => c.Messages, messages)
            .Set(c => c.UpdatedAt, DateTime.UtcNow),
        cancellationToken: ct);

✂️ IChatReducer: hard cut or summary

`Microsoft.Extensions.AI` defines IChatReducer — a single operation, `ReduceAsync`, that takes the message list and returns a shorter one — and ships two implementations. MessageCountingChatReducer keeps system messages plus the last N: zero cost, but everything that falls out of the window is forgotten. SummarizingChatReducer instead, past a threshold, asks an LLM to summarize the older turns and replaces those messages with the summary, keeping the recent tail.

The summarizing one has two parameters worth understanding: `targetCount` (how many recent messages survive) and `threshold` — how many messages past the target I tolerate before re-summarizing. The threshold avoids paying an extra LLM call on every single turn: the summary is redone only once the tail has grown enough. In the project the strategy is configuration, not code: a factory reads `appsettings.json` and wires the right reducer.

• counting: free and predictable; old context is gone entirely.
• summarizing: keeps the gist of old turns; costs one LLM call when it triggers.
• none: no reduction — useful as the baseline to measure the difference.

Counting cuts, summarizing compresses: either way the Mongo history stays whole.

🏭 The factory: strategy from configuration

The factory is the only place that knows the three strategies. The `SummarizingChatReducer` receives the same `IChatClient` as the chat: the summary is generated by the same local model, no extra dependency. In production I can point it at a smaller, faster model dedicated to summarization.

public sealed class ChatReducerFactory(IOptions<ChatMemoryOptions> options)
{
    // "counting" keeps the last N; "summarizing" folds older
    // turns into an LLM summary. Anything else = no reduction.
    public IChatReducer? Create(IChatClient chatClient) =>
        options.Value.Strategy.ToLowerInvariant() switch
        {
            "counting" => new MessageCountingChatReducer(
                options.Value.TargetMessageCount),
            "summarizing" => new SummarizingChatReducer(
                chatClient,
                options.Value.TargetMessageCount,
                options.Value.SummarizationThreshold),
            _ => null,
        };
}

🔁 The core: load, reduce, ask, persist

The `ChatService` is the whole flow in four steps. Note the order: the reduction happens after recomposing the full prompt and before the model call; persistence saves the unreduced turn. The store doesn't even know a reducer exists.

The HTTP response carries three numbers that make the memory observable: how many messages the full history holds, how many reached the model, and a preview of the actual prompt. They're the difference between “I think it's summarizing” and watching it happen turn by turn.

// 1. Full prompt: system + persisted history + the new turn.
List<ChatMessage> fullPrompt =
[
    new(ChatRole.System, options.Value.SystemPrompt),
    .. conversation.Messages.Select(m =>
        new ChatMessage(new ChatRole(m.Role), m.Text)),
    new(ChatRole.User, request.Text),
];

// 2. Reduce: the store keeps everything, the model gets the reduced view.
var prompt = reducer.Value is null
    ? fullPrompt
    : (await reducer.Value.ReduceAsync(fullPrompt, ct)).ToList();

// 3. Ask the model — reduced prompt only.
var response = await chatClient.GetResponseAsync(prompt, cancellationToken: ct);

// 4. Persist the turn untouched (no reduction on write).
await repository.AppendMessagesAsync(conversationId, turn, ct);

return new SendMessageResponse(
    response.Text, fullPrompt.Count, prompt.Count, preview);

🌐 Endpoints and wiring: Minimal API all the way

Three endpoints are enough: create a conversation, send a message, read the transcript back. The wiring in `Program.cs` registers Mongo, the `OllamaApiClient` as `IChatClient` via `AddChatClient`, the factory and the service — no Controllers, no ceremony.

builder.Services.AddSingleton<IConversationRepository,
    MongoConversationRepository>();

builder.Services.AddChatClient(sp =>
{
    var ollama = sp.GetRequiredService<IOptions<OllamaOptions>>().Value;
    return new OllamaApiClient(new Uri(ollama.Endpoint), ollama.Model);
});

builder.Services.AddSingleton<ChatReducerFactory>();
builder.Services.AddSingleton<ChatService>();

var app = builder.Build();
app.MapConversationEndpoints();   // POST /, POST /{id}/messages, GET /{id}
app.Run();

🐳 Dependencies in Docker: Mongo, Ollama and the model pull

Outside the app, two services are needed: MongoDB and Ollama. The repo's `docker-compose.yml` brings both up with healthchecks, plus a one-shot container that downloads `llama3.2:3b` into the Ollama volume and exits: the first start takes a few minutes (~2 GB), from the second on it's instant. Nothing gets installed on the laptop besides Docker.

`appsettings.json` already points at `localhost:27017` and `localhost:11434`, the ports published by compose: `docker compose up -d`, `dotnet run`, and the chat answers.

services:
  mongodb:
    image: mongo:7
    ports: ["27017:27017"]
    volumes: [mongo-data:/data/db]

  ollama:
    image: ollama/ollama:latest
    ports: ["11434:11434"]
    volumes: [ollama-data:/root/.ollama]

  ollama-init:    # one-shot: pulls the model, then exits
    image: ollama/ollama:latest
    depends_on: { ollama: { condition: service_healthy } }
    environment: ["OLLAMA_HOST=http://ollama:11434"]
    entrypoint: ["ollama", "pull", "llama3.2:3b"]

🔬 Observable memory: the numbers on every reply

This is where the pattern shows its work. After a dozen turns the Mongo history is at 23 messages, but `messagesSentToModel` stays at 6: system, the reducer-generated summary, the recent tail and the new question. The `promptPreview` shows exactly what the model saw — including the summary message, recognizable because it condenses the early turns into one line.

In production these three fields become metrics: if `messagesSentToModel` grows without a ceiling, the reducer isn't working; if the summary drops information that mattered, I see it in the preview before users notice it in the quality of the answers.

The yellow line in the promptPreview is the summary: the early turns compressed into one message.

🧪 Tests without Mongo and without Ollama

The reducers and the chat flow are testable without external services: a fake `IChatClient` with a scripted reply (which records the messages it received) and an in-memory repository. The most useful test verifies the pattern's central property: after the reduction the model saw 5 messages but the store keeps 22.

The fake doubles as the summarizer: when testing the `SummarizingChatReducer`, the summarization “LLM call” returns a fixed string and I can assert it lands in the reduced prompt. Six tests, seven milliseconds, zero containers.

var response = await service.SendAsync(
    conversation.Id, new SendMessageRequest("Latest question"), ct);

Assert.Equal(22, response.MessagesInHistory);   // system + 20 + new
Assert.Equal(5, response.MessagesSentToModel);  // system + last 4
Assert.Equal("Latest question",
    chatClient.LastMessages![^1].Text);         // the fake records everything

var stored = await repository.GetAsync(conversation.Id, ct);
Assert.Equal(22, stored!.Messages.Count);       // the store keeps EVERYTHING

⚠️ Limits and trade-offs to know

Chat reduction is experimental (`MEAI001`): the API may change across `Microsoft.Extensions.AI` releases — the full-store-plus-reduced-view pattern, though, stays valid even if tomorrow I write the reducer by hand. The `SummarizingChatReducer` excludes messages containing function calls from the summary: with tool calling, the “tools” part of the conversation doesn't get compressed.

The summary is itself an LLM output: it can drop details, and with a small model like `llama3.2:3b` it does. The practical defenses: a generous recent tail (high `targetCount`), a threshold that avoids constant re-summarizing, and an eye on the `promptPreview` when answers degrade. For hard requirements — “remember the order number mentioned 40 turns ago” — a summary isn't enough: that calls for semantic memory with embeddings and retrieval, the natural next step of this architecture.

• Experimental API: a deliberate NoWarn; the pattern is stable even if the API shifts.
• Lossy summaries: fine details can vanish; the preview surfaces it immediately.
• Summarizing cost: one extra LLM call when it triggers — the threshold amortizes it.
• Beyond summaries: precise long-term facts → embeddings + retrieval, not summarization.

📦 GitHub repo

All the article's code is in a public repository: the complete Minimal API (Mongo repository, reducer factory, chat service, endpoints), the six xunit tests with the fake `IChatClient`, the `docker-compose.yml` with Mongo, Ollama and the automatic model pull, plus a ready-to-use `requests.http` for the three endpoints.

Once cloned: `docker compose up -d`, `dotnet run`, and the first conversation with memory is one `curl` away. The strategy changes from `appsettings.json` without touching code.

$ git clone https://github.com/fscamuzzi/chat-memory-llm-dotnet-mongodb.git
$ cd chat-memory-llm-dotnet-mongodb

$ docker compose up -d        # Mongo + Ollama + pull llama3.2:3b
$ docker logs -f chatmemory-ollama-init   # first start: ~2 GB

$ dotnet test                 # 6 tests, zero external services
$ dotnet run --project src/ChatMemory.Api

$ curl -s -X POST http://localhost:5210/api/conversations \
    -H "Content-Type: application/json" -d '{"title":"Rome"}'

✅ Final checklist: memory in six moves

Recapping the journey. The rule I take home: the store and the prompt are two different things — the former is the complete truth and never gets reduced, the latter is a throwaway projection I keep small with the right reducer for the use case. Everything else — strategy, thresholds, model — is configuration.