In this article
- 🧨 Why an LLM needs the same protection as any remote dependency
- 🧭 What I build: two IChatClient decorators
- ⬇️ Step 1 · Project and packages
- 🐳 Step 2 · Ollama with Docker Compose
- ⚙️ Step 3 · Typed options
- 🎯 Step 4 · Which errors to retry
- 🛡️ Step 5 · ResilientChatClient: the Polly pipeline
- 🧱 ResilientChatClient: the code
- 🌊 Step 6 · Streaming: retry only until the first token
- 🔁 The streaming flow at a glance
- 🔀 Step 7 · FallbackChatClient
- 💥 Step 8 · ChaosChatClient: breaking the primary on demand
- 🧩 Step 9 · Registration and endpoints
- 🌐 The endpoints: a clean 503 instead of a generic error
- 📄 Program.cs and configuration
- ▶️ Step 10 · Try it
- 🖥️ What I see in the terminal
- 🧪 Tests: exercising failures without a model
- ⚠️ Mistakes to avoid
- 📦 GitHub repo
- 🏁 Conclusions
🧨 Why an LLM needs the same protection as any remote dependency
A language model, local or cloud, is a network dependency like any other: sometimes it answers in two seconds, sometimes in twenty, sometimes not at all. If my API calls the model with no defences, every provider failure becomes my failure.
In practice I see four situations. They need four different answers, and they are not the same answer: retrying helps with a passing error, but makes an already struggling provider worse.
- Passing error (network, 503, 429): try again after a short wait → retry with backoff.
- Very slow answer: stop waiting → timeout per attempt.
- Provider in trouble: stop calling it for a while → circuit breaker.
- Provider down: answer with another model → fallback.
🧭 What I build: two IChatClient decorators
I build a Minimal API on .NET 10 on top of the IChatClient abstraction from Microsoft.Extensions.AI. The defences are decorators: classes that implement IChatClient, wrap another one and add a behaviour. The rest of the app knows nothing about retry or fallback: it only depends on IChatClient.
The policy engine is Polly v8 (Polly.Core). Microsoft.Extensions.AI has no built-in fallback between models, so I write it myself: it is a few dozen lines.
- ResilientChatClient: a DelegatingChatClient with a Polly pipeline (retry → circuit breaker → timeout). One per model.
- FallbackChatClient: tries the clients in order and moves on only for errors that justify it.
- ChaosChatClient: a development decorator that simulates an outage, to watch the policies at work.
Two clients, each with its own pipeline; the fallback only kicks in when needed.
⬇️ Step 1 · Project and packages
I need the .NET 10 SDK and Docker. I create the solution, the Minimal API and the test project, then add the packages. OllamaSharp provides an Ollama client that already implements IChatClient.
dotnet new sln -n LlmResilience
dotnet new web -n LlmResilience.Api -o src/LlmResilience.Api
dotnet new xunit -n LlmResilience.Tests -o tests/LlmResilience.Tests
dotnet sln add src/LlmResilience.Api tests/LlmResilience.Tests
dotnet add tests/LlmResilience.Tests reference src/LlmResilience.Api
cd src/LlmResilience.Api
dotnet add package Microsoft.Extensions.AI
dotnet add package OllamaSharp
dotnet add package Polly.Corefscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
🐳 Step 2 · Ollama with Docker Compose
The only external dependency is Ollama. I start it with Docker Compose, with a healthcheck and a one-shot service that downloads the two models: a more capable one as primary (llama3.2:3b) and a tiny one as fallback (qwen2.5:0.5b).
In production the primary would typically be a cloud model and the fallback a local model or one from another provider: the code does not change, only the two registrations do.
services:
ollama:
image: ollama/ollama:latest
container_name: llm-resilience-ollama
ports:
- "11434:11434"
volumes:
- ollama-data:/root/.ollama
healthcheck:
test: ["CMD", "ollama", "list"]
interval: 10s
timeout: 5s
retries: 12
start_period: 10s
ollama-init:
image: ollama/ollama:latest
container_name: llm-resilience-ollama-init
depends_on:
ollama:
condition: service_healthy
environment:
OLLAMA_HOST: http://ollama:11434
entrypoint: ["/bin/sh", "-c"]
command:
- |
ollama pull llama3.2:3b &&
ollama pull qwen2.5:0.5b
restart: "no"
volumes:
ollama-data:fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
⚙️ Step 3 · Typed options
Every threshold lives in a typed class bound to appsettings.json: no magic numbers in the code. Changing them without recompiling is the useful part, because the right thresholds are found by observing the system, not on paper.
namespace LlmResilience.Api.Options;
public sealed class ResilienceOptions
{
public const string SectionName = "Resilience";
public int MaxRetryAttempts { get; init; } = 2;
public TimeSpan RetryBaseDelay { get; init; } = TimeSpan.FromMilliseconds(500);
public TimeSpan AttemptTimeout { get; init; } = TimeSpan.FromSeconds(30);
public double BreakerFailureRatio { get; init; } = 0.5;
public int BreakerMinimumThroughput { get; init; } = 4;
public TimeSpan BreakerSamplingDuration { get; init; } = TimeSpan.FromSeconds(30);
public TimeSpan BreakerBreakDuration { get; init; } = TimeSpan.FromSeconds(20);
}fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
🎯 Step 4 · Which errors to retry
This is the most important decision. Retrying a non-transient error (an invalid prompt, a wrong key) wastes time and, on the cloud, tokens. I only retry what can go away on its own: network errors, 5xx, 429, 408 and timeouts.
I separate two questions: is it worth retrying on the same client? (IsTransient) and is it worth moving to the next client? (ShouldFallBack). The second also includes the open circuit: with the breaker open retrying makes no sense, but moving on does.
using System.Net;
using Polly.Timeout;
namespace LlmResilience.Api.Resilience;
public static class TransientFailure
{
// Errors worth another attempt on the same provider: network, 5xx, 429, 408, timeouts.
public static bool IsTransient(Exception exception) => exception switch
{
HttpRequestException { StatusCode: null } => true,
HttpRequestException { StatusCode: HttpStatusCode status } => IsTransientStatus(status),
TimeoutRejectedException => true,
TimeoutException => true,
IOException => true,
_ => false,
};
// Errors that justify moving to the next provider: transient ones plus an open circuit.
public static bool ShouldFallBack(Exception exception) =>
IsTransient(exception) || exception is Polly.CircuitBreaker.BrokenCircuitException;
private static bool IsTransientStatus(HttpStatusCode status) =>
(int)status >= 500
|| status == HttpStatusCode.TooManyRequests
|| status == HttpStatusCode.RequestTimeout;
}fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
🛡️ Step 5 · ResilientChatClient: the Polly pipeline
The core is a DelegatingChatClient that runs every call inside a ResiliencePipeline. The order of the strategies matters: retry outside, circuit breaker in the middle, timeout inside. That way the timeout applies per attempt and the breaker sees every single failure, not only the final outcome of the retry.
The predicate is the same for retry and breaker (IsTransient). A BrokenCircuitException is not transient, so it is not retried: it exits at once and leaves the decision to the fallback. The breaker opens when at least 50% of the calls in the 30-second window fail, with a minimum of 4 calls.
- Retry: exponential backoff with jitter, so that many clients do not retry together.
- Circuit breaker: keeps the state (Closed, Open, HalfOpen) and exposes it through CircuitBreakerStateProvider.
- Timeout: 30 seconds per attempt; it produces a TimeoutRejectedException, which is transient.
The timeout applies per attempt; the breaker sees every failure.
🧱 ResilientChatClient: the code
The constructor builds the pipeline once. Each client has its own pipeline and therefore its own circuit breaker: if the primary is down, the fallback keeps working.
using LlmResilience.Api.Options;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging;
using Polly;
using Polly.CircuitBreaker;
using Polly.Retry;
namespace LlmResilience.Api.Resilience;
public sealed class ResilientChatClient : DelegatingChatClient
{
private readonly ResiliencePipeline _pipeline;
private readonly CircuitBreakerStateProvider _breakerState = new();
public string Name { get; }
public CircuitState Circuit => _breakerState.CircuitState;
public ResilientChatClient(
string name,
IChatClient inner,
ResilienceOptions options,
ILogger logger) : base(inner)
{
Name = name;
_pipeline = BuildPipeline(name, options, _breakerState, logger);
}
public override Task<ChatResponse> GetResponseAsync(
IEnumerable<ChatMessage> messages,
ChatOptions? options = null,
CancellationToken cancellationToken = default) =>
_pipeline.ExecuteAsync(
async token => await base.GetResponseAsync(messages, options, token),
cancellationToken).AsTask();
public override async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
IEnumerable<ChatMessage> messages,
ChatOptions? options = null,
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default)
{
// Retry and breaker only protect the opening of the stream (up to the first update).
var opened = await _pipeline.ExecuteAsync(
async token => await StreamOpening.OpenAsync(
base.GetStreamingResponseAsync(messages, options, token), token),
cancellationToken);
if (opened is null)
yield break;
await foreach (var update in StreamOpening.ReplayAsync(opened, cancellationToken))
yield return update;
}
private static ResiliencePipeline BuildPipeline(
string name,
ResilienceOptions options,
CircuitBreakerStateProvider breakerState,
ILogger logger)
{
var handled = new PredicateBuilder().Handle<Exception>(TransientFailure.IsTransient);
return new ResiliencePipelineBuilder()
.AddRetry(new RetryStrategyOptions
{
ShouldHandle = handled,
MaxRetryAttempts = options.MaxRetryAttempts,
Delay = options.RetryBaseDelay,
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
OnRetry = args =>
{
logger.LogWarning(
args.Outcome.Exception,
"LLM client {Client}: attempt {Attempt} failed, retrying in {Delay} ms",
name, args.AttemptNumber + 1, (int)args.RetryDelay.TotalMilliseconds);
return default;
},
})
.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
ShouldHandle = handled,
FailureRatio = options.BreakerFailureRatio,
MinimumThroughput = options.BreakerMinimumThroughput,
SamplingDuration = options.BreakerSamplingDuration,
BreakDuration = options.BreakerBreakDuration,
StateProvider = breakerState,
OnOpened = args =>
{
logger.LogError("LLM client {Client}: circuit OPEN for {Duration} s", name, args.BreakDuration.TotalSeconds);
return default;
},
OnClosed = _ =>
{
logger.LogInformation("LLM client {Client}: circuit CLOSED", name);
return default;
},
})
.AddTimeout(options.AttemptTimeout)
.Build();
}
}fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
🌊 Step 6 · Streaming: retry only until the first token
Streaming changes the picture. Once the first token has gone to the client, I can no longer retry: I would have to restart the answer from scratch and the client has already seen half a sentence. So the protection only acts on opening the stream, up to the first update.
StreamOpening.OpenAsync asks for the first element of the enumerator: if it fails, the exception goes through the pipeline (retry, breaker, timeout) and the fallback. If it succeeds, ReplayAsync returns that first element and then the rest of the stream. An error mid-stream, afterwards, ends the stream: it is a protocol limit, not a code limit.
using Microsoft.Extensions.AI;
namespace LlmResilience.Api.Resilience;
// A stream that already produced its first update: from here on a retry is no longer safe.
public sealed record OpenedStream(IAsyncEnumerator<ChatResponseUpdate> Enumerator, ChatResponseUpdate First);
public static class StreamOpening
{
public static async Task<OpenedStream?> OpenAsync(
IAsyncEnumerable<ChatResponseUpdate> stream,
CancellationToken cancellationToken)
{
var enumerator = stream.GetAsyncEnumerator(cancellationToken);
try
{
if (!await enumerator.MoveNextAsync())
{
await enumerator.DisposeAsync();
return null;
}
return new OpenedStream(enumerator, enumerator.Current);
}
catch
{
await enumerator.DisposeAsync();
throw;
}
}
public static async IAsyncEnumerable<ChatResponseUpdate> ReplayAsync(
OpenedStream opened,
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken)
{
await using var enumerator = opened.Enumerator;
yield return opened.First;
while (await enumerator.MoveNextAsync())
{
cancellationToken.ThrowIfCancellationRequested();
yield return enumerator.Current;
}
}
}fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
🔁 The streaming flow at a glance
The diagram sums up where the defences work and where they do not. In the safe zone I can retry and switch model; after the first token the stream is committed and every error propagates to the client.
After the first token there is no way back.
🔀 Step 7 · FallbackChatClient
The fallback tries the clients in order. It moves on only if the error justifies it (ShouldFallBack); a non-transient error, such as an invalid prompt, exits at once because another model would not fix it. If all of them fail, it throws an LlmUnavailableException with all the collected errors.
I also record which client answered in the response AdditionalProperties (llm.client): it is the value I then put in logs and metrics, because an answer from the fallback must be recognisable.
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging;
namespace LlmResilience.Api.Resilience;
public sealed class FallbackChatClient(
IReadOnlyList<ResilientChatClient> clients,
ILogger<FallbackChatClient> logger) : IChatClient
{
public const string ClientProperty = "llm.client";
public IReadOnlyList<ResilientChatClient> Clients { get; } = clients;
public async Task<ChatResponse> GetResponseAsync(
IEnumerable<ChatMessage> messages,
ChatOptions? options = null,
CancellationToken cancellationToken = default)
{
var chat = messages.ToList();
var failures = new List<Exception>();
foreach (var client in Clients)
{
try
{
var response = await client.GetResponseAsync(chat, options, cancellationToken);
response.AdditionalProperties ??= [];
response.AdditionalProperties[ClientProperty] = client.Name;
return response;
}
catch (Exception exception) when (TransientFailure.ShouldFallBack(exception))
{
logger.LogWarning(exception, "LLM client {Client} failed, trying the next one", client.Name);
failures.Add(exception);
}
}
throw new LlmUnavailableException(failures);
}
public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
IEnumerable<ChatMessage> messages,
ChatOptions? options = null,
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default)
{
var chat = messages.ToList();
var opened = await OpenFirstAvailableAsync(chat, options, cancellationToken);
await foreach (var update in StreamOpening.ReplayAsync(opened, cancellationToken))
yield return update;
}
public object? GetService(Type serviceType, object? serviceKey = null) =>
serviceKey is null && serviceType.IsInstanceOfType(this) ? this : null;
public void Dispose()
{
foreach (var client in Clients)
client.Dispose();
}
private async Task<OpenedStream> OpenFirstAvailableAsync(
List<ChatMessage> chat,
ChatOptions? options,
CancellationToken cancellationToken)
{
var failures = new List<Exception>();
foreach (var client in Clients)
{
try
{
var opened = await StreamOpening.OpenAsync(
client.GetStreamingResponseAsync(chat, options, cancellationToken),
cancellationToken);
if (opened is not null)
return opened;
failures.Add(new InvalidOperationException($"{client.Name} returned an empty stream."));
}
catch (Exception exception) when (TransientFailure.ShouldFallBack(exception))
{
logger.LogWarning(exception, "LLM client {Client} failed to open the stream, trying the next one", client.Name);
failures.Add(exception);
}
}
throw new LlmUnavailableException(failures);
}
}fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
💥 Step 8 · ChaosChatClient: breaking the primary on demand
To watch the policies work I need a reproducible failure. The ChaosChatClient wraps only the primary and, based on a mode that can be set at runtime, throws a 503 (Fail) or hangs until the timeout (Slow). It is switched on with Llm:ChaosEnabled and is meant for development and demos, not for production.
using Microsoft.Extensions.AI;
using System.Net;
namespace LlmResilience.Api.Resilience;
public enum ChaosMode
{
Off,
Fail,
Slow,
}
public sealed class ChaosState
{
private int _mode;
public ChaosMode Mode
{
get => (ChaosMode)Volatile.Read(ref _mode);
set => Volatile.Write(ref _mode, (int)value);
}
}
// Development-only decorator: injects failures so the resilience policies can be observed.
public sealed class ChaosChatClient(IChatClient inner, ChaosState state) : DelegatingChatClient(inner)
{
private static readonly TimeSpan SlowDelay = TimeSpan.FromMinutes(5);
public override async Task<ChatResponse> GetResponseAsync(
IEnumerable<ChatMessage> messages,
ChatOptions? options = null,
CancellationToken cancellationToken = default)
{
await InjectAsync(cancellationToken);
return await base.GetResponseAsync(messages, options, cancellationToken);
}
public override async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
IEnumerable<ChatMessage> messages,
ChatOptions? options = null,
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default)
{
await InjectAsync(cancellationToken);
await foreach (var update in base.GetStreamingResponseAsync(messages, options, cancellationToken))
yield return update;
}
private async Task InjectAsync(CancellationToken cancellationToken)
{
switch (state.Mode)
{
case ChaosMode.Fail:
throw new HttpRequestException("Chaos: simulated provider outage.", null, HttpStatusCode.ServiceUnavailable);
case ChaosMode.Slow:
await Task.Delay(SlowDelay, cancellationToken);
break;
}
}
}fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
🧩 Step 9 · Registration and endpoints
A single extension method, AddResilientChat, builds the two clients, wraps them and registers the FallbackChatClient as IChatClient. The HttpClient has an infinite timeout on purpose: the deadline is decided by Polly, per attempt, not by a hidden global value.
using LlmResilience.Api.Options;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Options;
using OllamaSharp;
namespace LlmResilience.Api.Resilience;
public static class ServiceCollectionExtensions
{
public static IServiceCollection AddResilientChat(this IServiceCollection services, IConfiguration configuration)
{
services.Configure<LlmOptions>(configuration.GetSection(LlmOptions.SectionName));
services.Configure<ResilienceOptions>(configuration.GetSection(ResilienceOptions.SectionName));
services.AddSingleton<ChaosState>();
services.AddSingleton(provider =>
{
var llm = provider.GetRequiredService<IOptions<LlmOptions>>().Value;
var resilience = provider.GetRequiredService<IOptions<ResilienceOptions>>().Value;
var chaos = provider.GetRequiredService<ChaosState>();
var loggers = provider.GetRequiredService<ILoggerFactory>();
IChatClient primary = CreateOllama(llm.Endpoint, llm.PrimaryModel);
if (llm.ChaosEnabled)
primary = new ChaosChatClient(primary, chaos);
var fallback = CreateOllama(llm.Endpoint, llm.FallbackModel);
return new FallbackChatClient(
[
new ResilientChatClient("primary", primary, resilience, loggers.CreateLogger("LlmResilience.primary")),
new ResilientChatClient("fallback", fallback, resilience, loggers.CreateLogger("LlmResilience.fallback")),
],
loggers.CreateLogger<FallbackChatClient>());
});
services.AddSingleton<IChatClient>(provider => provider.GetRequiredService<FallbackChatClient>());
return services;
}
// The HttpClient timeout is disabled on purpose: the Polly timeout owns the deadline.
private static OllamaApiClient CreateOllama(string endpoint, string model) =>
new(new HttpClient { BaseAddress = new Uri(endpoint), Timeout = Timeout.InfiniteTimeSpan }, model);
}fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
🌐 The endpoints: a clean 503 instead of a generic error
If all the clients fail, the API answers 503 Service Unavailable with a ProblemDetails, not a 500. It is the right signal for a client that knows how to retry later. For streaming I open the flow before returning the response: that way a total outage becomes a real 503 and not an interrupted 200 response.
using System.Diagnostics;
using System.Net.ServerSentEvents;
using LlmResilience.Api.Contracts;
using LlmResilience.Api.Resilience;
using Microsoft.AspNetCore.Http.HttpResults;
using Microsoft.Extensions.AI;
namespace LlmResilience.Api.Endpoints;
public static class ChatEndpoints
{
public static IEndpointRouteBuilder MapChatEndpoints(this IEndpointRouteBuilder app)
{
app.MapPost("/chat", ChatAsync);
app.MapPost("/chat/stream", StreamAsync);
app.MapGet("/llm/status", Status);
app.MapPut("/chaos", SetChaos);
return app;
}
private static async Task<Results<Ok<ChatReply>, ProblemHttpResult>> ChatAsync(
ChatRequest request,
IChatClient client,
CancellationToken cancellationToken)
{
var clock = Stopwatch.StartNew();
try
{
var response = await client.GetResponseAsync(request.Prompt, cancellationToken: cancellationToken);
var used = response.AdditionalProperties?[FallbackChatClient.ClientProperty]?.ToString() ?? "unknown";
return TypedResults.Ok(new ChatReply(response.Text, used, clock.ElapsedMilliseconds));
}
catch (LlmUnavailableException)
{
return Unavailable();
}
}
private static async Task<Results<ServerSentEventsResult<string>, ProblemHttpResult>> StreamAsync(
ChatRequest request,
IChatClient client,
CancellationToken cancellationToken)
{
var updates = client.GetStreamingResponseAsync(request.Prompt, cancellationToken: cancellationToken);
var enumerator = updates.GetAsyncEnumerator(cancellationToken);
try
{
// Opening the stream here lets a total outage become a clean 503 instead of a broken 200.
if (!await enumerator.MoveNextAsync())
return TypedResults.ServerSentEvents(Empty(), "token");
}
catch (LlmUnavailableException)
{
await enumerator.DisposeAsync();
return Unavailable();
}
return TypedResults.ServerSentEvents(Tokens(enumerator), "token");
}
private static async IAsyncEnumerable<string> Tokens(IAsyncEnumerator<ChatResponseUpdate> enumerator)
{
await using (enumerator)
{
do
{
if (!string.IsNullOrEmpty(enumerator.Current.Text))
yield return enumerator.Current.Text;
}
while (await enumerator.MoveNextAsync());
}
}
private static async IAsyncEnumerable<string> Empty()
{
await Task.CompletedTask;
yield break;
}
private static Ok<ProviderStatus[]> Status(FallbackChatClient client) =>
TypedResults.Ok(client.Clients.Select(c => new ProviderStatus(c.Name, c.Circuit.ToString())).ToArray());
private static NoContent SetChaos(ChaosRequest request, ChaosState state)
{
state.Mode = request.Mode;
return TypedResults.NoContent();
}
private static ProblemHttpResult Unavailable() =>
TypedResults.Problem(
title: "All LLM clients are unavailable",
statusCode: StatusCodes.Status503ServiceUnavailable,
extensions: new Dictionary<string, object?> { ["retryAfterSeconds"] = 20 });
}fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
📄 Program.cs and configuration
The composition root stays at a few lines. The example configuration points to localhost:11434, the port published by Docker Compose.
using System.Text.Json.Serialization;
using LlmResilience.Api.Endpoints;
using LlmResilience.Api.Resilience;
var builder = WebApplication.CreateBuilder(args);
builder.Services.ConfigureHttpJsonOptions(options =>
options.SerializerOptions.Converters.Add(new JsonStringEnumConverter()));
builder.Services.AddProblemDetails();
builder.Services.AddResilientChat(builder.Configuration);
var app = builder.Build();
app.MapChatEndpoints();
app.Run();
public partial class Program;
// appsettings.json (excerpt)
// "Llm": {
// "Endpoint": "http://localhost:11434",
// "PrimaryModel": "llama3.2:3b",
// "FallbackModel": "qwen2.5:0.5b",
// "ChaosEnabled": true
// },
// "Resilience": {
// "MaxRetryAttempts": 2,
// "RetryBaseDelay": "00:00:00.500",
// "AttemptTimeout": "00:00:30",
// "BreakerFailureRatio": 0.5,
// "BreakerMinimumThroughput": 4,
// "BreakerSamplingDuration": "00:00:30",
// "BreakerBreakDuration": "00:00:20"
// }
// }fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
▶️ Step 10 · Try it
I start Ollama, wait for the models to download and launch the API. Then I break the primary and repeat the same request: the fallback answers and in the logs I see the two attempts with their backoff. After a few failures the primary circuit is Open: later calls no longer touch it and the fallback answers in a few milliseconds.
docker compose up -d
docker compose logs -f ollama-init # attendi exit code 0
dotnet run --project src/LlmResilience.Api --urls http://localhost:5050fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
🖥️ What I see in the terminal
This is real output from a session: the retry warn with its delay, the move to the next client, the answer with client: fallback and the circuit state. The elapsedMs of the first answer includes the retry waits; the following ones, with the circuit open, are almost instant on the primary side.
After the failures the primary is Open and the fallback answers.
🧪 Tests: exercising failures without a model
A fake IChatClient with a script of errors lets me test every policy without Ollama and without a network. I set one-millisecond delays, so the suite runs in about two seconds. I check: retry until success, no retry on non-transient errors, timeout, circuit opening, caller cancellation, fallback and streaming.
using LlmResilience.Api.Options;
using LlmResilience.Api.Resilience;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging.Abstractions;
using Polly.CircuitBreaker;
using Polly.Timeout;
namespace LlmResilience.Tests;
public class ResilienceTests
{
private static readonly ResilienceOptions Fast = new()
{
MaxRetryAttempts = 2,
RetryBaseDelay = TimeSpan.FromMilliseconds(1),
AttemptTimeout = TimeSpan.FromMilliseconds(200),
BreakerMinimumThroughput = 4,
BreakerFailureRatio = 0.5,
BreakerSamplingDuration = TimeSpan.FromSeconds(30),
BreakerBreakDuration = TimeSpan.FromSeconds(30),
};
private static ResilientChatClient Wrap(string name, IChatClient inner) =>
new(name, inner, Fast, NullLogger.Instance);
private static List<ChatMessage> Prompt => [new(ChatRole.User, "ciao")];
[Fact]
public async Task Transient_failures_are_retried_until_success()
{
var inner = new ScriptedChatClient("ok", ScriptedChatClient.Outage(), ScriptedChatClient.Outage());
var client = Wrap("primary", inner);
var response = await client.GetResponseAsync(Prompt);
Assert.Equal("ok", response.Text);
Assert.Equal(3, inner.Calls);
}
[Fact]
public async Task Non_transient_failures_are_not_retried()
{
var inner = new ScriptedChatClient("ok", new ArgumentException("bad request"));
var client = Wrap("primary", inner);
await Assert.ThrowsAsync<ArgumentException>(() => client.GetResponseAsync(Prompt));
Assert.Equal(1, inner.Calls);
}
[Fact]
public async Task Slow_attempts_hit_the_timeout()
{
var inner = new ScriptedChatClient("ok") { Delay = TimeSpan.FromSeconds(5) };
var client = Wrap("primary", inner);
await Assert.ThrowsAsync<TimeoutRejectedException>(() => client.GetResponseAsync(Prompt));
Assert.Equal(3, inner.Calls);
}
[Fact]
public async Task Circuit_opens_after_repeated_failures_and_stops_calling_the_provider()
{
var inner = new ScriptedChatClient("ok", Enumerable.Repeat<Exception?>(ScriptedChatClient.Outage(), 20).ToArray());
var client = Wrap("primary", inner);
await Assert.ThrowsAsync<HttpRequestException>(() => client.GetResponseAsync(Prompt));
await Assert.ThrowsAsync<BrokenCircuitException>(() => client.GetResponseAsync(Prompt));
var callsWhenOpen = inner.Calls;
await Assert.ThrowsAsync<BrokenCircuitException>(() => client.GetResponseAsync(Prompt));
Assert.Equal(CircuitState.Open, client.Circuit);
Assert.Equal(callsWhenOpen, inner.Calls);
}
[Fact]
public async Task Caller_cancellation_is_not_retried()
{
var inner = new ScriptedChatClient("ok") { Delay = TimeSpan.FromSeconds(5) };
var client = new ResilientChatClient("primary", inner, Fast, NullLogger.Instance);
using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(50));
await Assert.ThrowsAnyAsync<OperationCanceledException>(() => client.GetResponseAsync(Prompt, null, cts.Token));
Assert.Equal(1, inner.Calls);
}
[Fact]
public async Task Fallback_answers_when_the_primary_is_down()
{
var primary = new ScriptedChatClient("from primary", Enumerable.Repeat<Exception?>(ScriptedChatClient.Outage(), 20).ToArray());
var secondary = new ScriptedChatClient("from fallback");
var client = new FallbackChatClient([Wrap("primary", primary), Wrap("fallback", secondary)], NullLogger<FallbackChatClient>.Instance);
var response = await client.GetResponseAsync(Prompt);
Assert.Equal("from fallback", response.Text);
Assert.Equal("fallback", response.AdditionalProperties![FallbackChatClient.ClientProperty]);
}
[Fact]
public async Task Fallback_is_skipped_when_the_primary_works()
{
var primary = new ScriptedChatClient("from primary");
var secondary = new ScriptedChatClient("from fallback");
var client = new FallbackChatClient([Wrap("primary", primary), Wrap("fallback", secondary)], NullLogger<FallbackChatClient>.Instance);
var response = await client.GetResponseAsync(Prompt);
Assert.Equal("primary", response.AdditionalProperties![FallbackChatClient.ClientProperty]);
Assert.Equal(0, secondary.Calls);
}
[Fact]
public async Task Fallback_throws_when_every_client_fails()
{
var down = Enumerable.Repeat<Exception?>(ScriptedChatClient.Outage(), 20).ToArray();
var client = new FallbackChatClient(
[Wrap("primary", new ScriptedChatClient("a", down)), Wrap("fallback", new ScriptedChatClient("b", down))],
NullLogger<FallbackChatClient>.Instance);
var failure = await Assert.ThrowsAsync<LlmUnavailableException>(() => client.GetResponseAsync(Prompt));
Assert.Equal(2, failure.Failures.Count);
}
[Fact]
public async Task Streaming_is_retried_before_the_first_update()
{
var inner = new ScriptedChatClient("uno due tre", ScriptedChatClient.Outage());
var client = Wrap("primary", inner);
var text = string.Concat(await Collect(client.GetStreamingResponseAsync(Prompt)));
Assert.Equal("unoduetre", text);
Assert.Equal(2, inner.Calls);
}
[Fact]
public async Task Streaming_falls_back_when_the_primary_cannot_open()
{
var primary = new ScriptedChatClient("a b", Enumerable.Repeat<Exception?>(ScriptedChatClient.Outage(), 20).ToArray());
var secondary = new ScriptedChatClient("x y");
var client = new FallbackChatClient([Wrap("primary", primary), Wrap("fallback", secondary)], NullLogger<FallbackChatClient>.Instance);
var text = string.Concat(await Collect(client.GetStreamingResponseAsync(Prompt)));
Assert.Equal("xy", text);
}
private static async Task<List<string>> Collect(IAsyncEnumerable<ChatResponseUpdate> updates)
{
var parts = new List<string>();
await foreach (var update in updates)
parts.Add(update.Text);
return parts;
}
}fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
⚠️ Mistakes to avoid
These are the mistakes I see most often when retry and fallback are added to an LLM integration.
- Retrying everything: a 400 or an invalid prompt does not improve on the second attempt.
- Retry without jitter: many clients retrying together form a wave that crushes the provider.
- A single global timeout: a per-attempt timeout avoids three 100-second retries turning into five minutes of waiting.
- A shared breaker: every provider needs its own, otherwise a primary failure also shuts down the fallback.
- A silent fallback: a smaller model answers differently; I always record which client answered.
- Retrying a stream halfway: after the first token the start of the answer would be duplicated.
- Chaos in production: Llm:ChaosEnabled must be false outside development.
📦 GitHub repo
The complete project — API, decorators, scripted test client, docker-compose.yml with Ollama and model download, .http file and xUnit suite — is public on GitHub. The README lists the same Docker Compose, dotnet test and dotnet run commands as this article.
Full code on GitHub: https://github.com/fscamuzzi/llm-resilience-dotnet-polly-fallback
git clone https://github.com/fscamuzzi/llm-resilience-dotnet-polly-fallback.git
cd llm-resilience-dotnet-polly-fallback
docker compose up -d
dotnet test
dotnet run --project src/LlmResilience.Api --urls http://localhost:5050fscamuzzi/llm-resilience-dotnet-polly-fallback · GitHub ↗
🏁 Conclusions
Retry, timeout, breaker and fallback are not four variants of the same idea: they answer four different failures and must be composed in the right order. With IChatClient and Polly they stay confined to two decorators and the app does not notice.
The natural next step is to measure them: count retries, open circuits and fallback answers with OpenTelemetry metrics, as in the article on LLM observability.
Frequently asked questions about LLM resilience .NET
Why not retry every LLM error?
Only transient errors (network, 5xx, 429, 408, timeouts) can go away on their own. An invalid prompt or a wrong key would fail again and cost time and, on the cloud, tokens.
What is the right order of retry, circuit breaker and timeout in Polly?
Retry outside, circuit breaker in the middle, timeout inside. The timeout applies per attempt and the breaker counts every single failure.
Does Microsoft.Extensions.AI have a fallback between models?
I do not know of a built-in one: IChatClient is easy to decorate, so I write a FallbackChatClient myself. It is a few dozen lines and stays provider-independent.
Can I retry a stream halfway?
No. After the first token the client has already received part of the answer and a retry would duplicate it. I only protect opening the stream, up to the first update.
Do I need a circuit breaker for each model?
Yes. With a shared breaker, primary failures would also open the circuit for the fallback, which is working fine. Every ResilientChatClient has its own pipeline.
Can I use OpenAI or Azure instead of Ollama?
Yes. The decorators only depend on IChatClient: I change the two registrations in AddResilientChat and the pipeline stays the same. Only the exceptions the provider throws change, so I re-check IsTransient.
Let's talk
If this topic is relevant to you, write to me: comparing notes on code and AI is always time well spent.



