Observability and Evaluation for Microsoft Agent Framework Agents in ASP.NET Core

Microsoft Agent Framework is now generally available and is the recommended path for new agent work in .NET. It sits on top of Microsoft.Extensions.AI, so the same IChatClient you already use with Ollama or Azure OpenAI becomes an AIAgent with one extension method. The practical problem that appears the moment you move beyond a console demo is the same one every production team hits: the agent calls tools, consumes tokens, and sometimes invents answers, yet you have almost no visibility into what actually happened.

This article shows a minimal, current pattern that adds OpenTelemetry instrumentation and a simple local evaluation loop around an Agent Framework agent hosted in ASP.NET Core. The agent runs against a local Ollama model and has one ordinary C# tool. The same registration works if you later swap the chat client for Azure OpenAI or OpenAI.

What the Agent Framework actually emits

Agent Framework builds on the Microsoft.Extensions.AI middleware pipeline. When you call UseOpenTelemetry on an IChatClientBuilder or on an AIAgent builder, the framework emits traces and metrics that follow the OpenTelemetry GenAI semantic conventions. You get spans for the overall agent invocation, for each chat completion, and for every tool execution. Token counts, latencies, and (optionally) sensitive content appear as attributes.

You do not have to write your own ActivitySource for the common path. You do have to subscribe to the right source names and decide whether sensitive data is allowed in your environment.

Packages and project setup

Create a minimal API project targeting .NET 8 or later (Agent Framework 1.x works on the supported .NET versions). Install the packages that are current as of late 2026:

dotnet add package Microsoft.Agents.AI
dotnet add package Microsoft.Extensions.AI
dotnet add package OllamaSharp
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

OllamaSharp is the recommended way to obtain an IChatClient for a local Ollama instance. Microsoft.Extensions.AI.Ollama is deprecated.

Register the instrumented agent

In Program.cs the registration is short. Add the OpenTelemetry middleware to the chat client and to the agent. You don't need to add function invocation yourself: the agent wraps the chat client in Microsoft.Extensions.AI's function-invocation middleware automatically, so each model call and each tool execution gets its own span under the agent run.

using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
using OllamaSharp;
using OpenTelemetry.Trace;
using System.ComponentModel;

var builder = WebApplication.CreateBuilder(args);

const string TelemetrySource = "MyApp.Agent";

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource(TelemetrySource)
        .AddConsoleExporter()
        // Uncomment for an OTLP collector or Aspire dashboard
        // .AddOtlpExporter(o => o.Endpoint = new Uri("http://localhost:4317"))
    );

builder.Services.AddSingleton<IChatClient>(sp =>
{
    var ollama = new OllamaApiClient(new Uri("http://localhost:11434"), "llama3.2");
    return ollama
        .AsBuilder()
        .UseOpenTelemetry(sourceName: TelemetrySource, configure: cfg => cfg.EnableSensitiveData = false)
        .Build();
});

builder.Services.AddSingleton<AIAgent>(sp =>
{
    var chatClient = sp.GetRequiredService<IChatClient>();
    return chatClient.AsAIAgent(
        instructions: "You are a helpful assistant. Use the GetWeather tool when the user asks about weather. Never invent weather data.",
        name: "WeatherAgent",
        tools: [AIFunctionFactory.Create(GetWeather)])
        .AsBuilder()
        .UseOpenTelemetry(sourceName: TelemetrySource, configure: cfg => cfg.EnableSensitiveData = false)
        .Build();
});

var app = builder.Build();

app.MapPost("/chat", async (ChatRequest request, AIAgent agent) =>
{
    var result = await agent.RunAsync(request.Message);
    return Results.Ok(new { answer = result.Text });
});

app.Run();

[Description("Get the current weather for a city.")]
static string GetWeather([Description("City name")] string city)
    => $"The weather in {city} is 18Â°C and partly cloudy.";

record ChatRequest(string Message);

EnableSensitiveData is deliberately false in the example. Turn it on only in development or behind a private collector; otherwise prompts and tool results land in your telemetry store.

What the traces look like

When a request hits /chat, you see a parent span for the agent run. Inside it sits a chat span, and if the model decides to call GetWeather you see a child execute_tool span. The attributes include the model name, token usage, duration, and the tool name. This is enough to answer the usual production questions: Did the agent call the tool it was supposed to call? How many tokens did this turn cost? Was the latency dominated by the model or by the tool?

If you point the OTLP exporter at the Aspire dashboard or at Jaeger, the same spans appear with the parent-child relationship intact. No extra custom instrumentation is required for the basic path.

Local evaluation before the agent leaves the machine

Traces tell you what happened. Evaluation tells you whether the behavior is acceptable, and you can start with plain assertions that run against your local model without a cloud bill.

A practical starting point is a small console harness (or an integration test) that exercises the same agent registration and checks two things: the final answer contains expected keywords, and the required tool was actually invoked.

using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;

// Assume the same AIAgent registration as above, resolved from a test host or built directly.

AIAgent agent = BuildWeatherAgent(); // same chat client + AsAIAgent registration as Program.cs

var evalResult = await agent.RunAsync("What is the weather in Seattle?");

// Simple local checks
bool containsExpected = evalResult.Text.Contains("Seattle", StringComparison.OrdinalIgnoreCase)
                        && evalResult.Text.Contains("18", StringComparison.OrdinalIgnoreCase);

// The response includes the intermediate tool-call messages, so check that GetWeather was really invoked
bool toolWasCalled = evalResult.Messages
    .SelectMany(m => m.Contents)
    .OfType<FunctionCallContent>()
    .Any(call => call.Name == "GetWeather");

Console.WriteLine($"Content check: {containsExpected}");
Console.WriteLine($"Tool check: {toolWasCalled}");

When keyword checks stop being enough, the Microsoft.Extensions.AI.Evaluation libraries (Microsoft.Extensions.AI.Evaluation.Quality and .Reporting) add LLM-graded evaluators such as relevance, coherence, and groundedness, plus cached, repeatable reports you can run in CI. The important design point stays the same: run the checks against the live agent (or a recorded fixture) before the endpoint is exposed, and every time you change the prompt, the model, or a package version.

Putting both pieces behind an ASP.NET Core endpoint

The pattern that works in production is to keep the agent registration exactly as shown above and to surface only a thin endpoint. Health checks can verify that Ollama (or the cloud endpoint) is reachable. A separate internal endpoint or a background job can run a small evaluation suite on a schedule and emit a custom metric if the checks start failing.

app.MapGet("/health/agent", async (AIAgent agent) =>
{
    try
    {
        var result = await agent.RunAsync("Say OK if you can use tools.");
        return Results.Ok(new { status = "healthy", preview = result.Text });
    }
    catch (Exception ex)
    {
        return Results.Problem(ex.Message);
    }
});

Keep this probe on a slow schedule or behind an internal route, because every call is a real model request. Because the OpenTelemetry middleware is already on the agent, every health-check call also appears in traces. You can alert on elevated error rates, missing tool spans, or sudden token spikes without writing extra logging code.

Practical constraints

Local models such as llama3.2 support tool calling reliably enough for this pattern, but smaller models sometimes skip the tool or hallucinate arguments. The evaluation checks catch that before users do. Sensitive data in traces is the other hard constraint: EnableSensitiveData = true is useful while you are debugging a single developer machine and dangerous once the collector is shared.

Agent Framework ships updates often, and the GenAI telemetry conventions are still evolving. Pin the packages you use and re-run the evaluation suite when you upgrade.

Closing

The difference between a working agent demo and an agent you can leave running is visibility and a repeatable check that the agent still does what you intended. OpenTelemetry on the Agent Framework builders gives you the traces and metrics. A short local evaluation loop gives you the quality gate. Both fit inside a normal ASP.NET Core host and stay provider-agnostic, so the same code works with Ollama today and with Azure OpenAI tomorrow.

Rahmat Afridi is an AI Integration Developer and Senior .NET Developer based in Islamabad, Pakistan, building production .NET and AI-integrated systems since 2016. See his work and AI integration services at rahmatafridi.com, or browse his code on GitHub.