MICROSOFT AGENT FRAMEWORK · 0.16.0-rc.1
Model routing
Qualify routes, count triage and compare each request with a direct baseline.
Objectives
- Routing policy
- Direct baseline and auxiliary calls
- Comparison and limits
An extension after Demo470
Routing distributes requests across profiles with different capabilities, latency and constraints. It also adds a dependency and a classification request. Starting with a direct baseline can therefore be reasonable. Define the task and success criteria before naming models: professional rewriting and incident analysis require different checks.
Qualify candidates before routing
Triage, Easy and Hard are configured roles rather than three guaranteed quality levels. For each role retain provider, endpoint, model, prompt version, limits and test set. Measure misroutes, refusals, factual responses and sensitive cases on intended requests. Rejected judge calibration cannot be bypassed by the name Hard. The course retains granite4:3b qualification rejection; each new combination needs its own evidence.
EP016 scripts proposed mistral/qwen, risk/confidence scores and a menu. Current RoutingExperiment accepts only Ollama, defaults all three roles to the same local model and parses only Easy or Hard after Trim. It provides no calibrated score, free-text sensitivity classifier or fallback. Historical model names illustrate roles rather than an installation list.
Keep policy in the host
var triage = await Call("triage", triageModel, "Classify the following request. Return exactly Easy or Hard.\n" + request);
string route = triage.Trim();
if (route is not ("Easy" or "Hard")) throw new InvalidDataException("Unrecognized triage result.");
if (request.Contains("503", StringComparison.Ordinal)) route = "Hard";
string prompt = "Answer briefly in English. Separate facts from hypotheses.\n" + request;
_ = await Call("routed-execution", route == "Easy" ? easyModel : hardModel, prompt);
_ = await Call("direct-hard-baseline", hardModel, prompt);
Source code · Maf.Observability · Folder
Two fixed requests are processed. The first rewrites a sentence; the second asks for hypotheses, risks and checks for HTTP 503 after migration. After triage C# forces the second to Hard by detecting 503. Any output other than Easy/Hard fails explicitly. --sensitive takes a host branch before model configuration and prints human escalation without inference: no authenticated human workflow is created.
Triage informs a resource decision; it grants no tool rights. An application must check permissions and confidentiality independently. Do not send a prohibited case to a larger model hoping for a better decision. Permission, budget and stopping policies remain the responsibility of the harness and host.
Compare the complete path
using var client = new ObservedChatClient(DemoConnection.DemoConnection.CreateChatClient(settings with
{ LocalChat = settings.LocalChat with { Model = model } }),
usage =>
{
(stage == "direct-hard-baseline" ? baselineUsage : routedUsage).Add(usage);
Console.WriteLine(JsonSerializer.Serialize(new { Stage = stage, Model = model, Usage = usage }));
});
var agent = Telemetry.Instrument(client.AsAIAgent(name: "RoutingExperiment", instructions: "Follow the host task. No tools or business writes."));
var timer = Stopwatch.StartNew();
var response = await agent.RunAsync(prompt, cancellationToken: stop.Token);
Console.WriteLine(JsonSerializer.Serialize(new { Stage = stage, Model = model, DurationMs = timer.Elapsed.TotalMilliseconds }));
return response.Text;
Source code · Maf.Observability · Folder
Each pair runs triage then routed-execution, followed by direct-hard-baseline, using fresh sessions and the same final prompt. Triage and execution requests feed routedUsage; the baseline has its own ledger. A complete successful run therefore expects four routed requests including triage against two direct requests. These counts follow code rather than a benchmark result. Read Stage, Model, DurationMs and counter coverage; no business answer is written.
Routed cost includes triage, execution, any auxiliary calls and operations. Unknown tokens remain unknown; treating them as zero would manufacture savings. The program configures no rate. Per-call durations are not a production latency percentile. For further work repeat on a representative corpus, alternate order, isolate warmup and cache, report errors and quality alongside usage, then apply attributed pricing assumptions.
Do not generalize the local comparison
Order is fixed, answers are not evaluated here and identical profiles demonstrate mechanics only. Distinct models may use different tokenizers. Apparent savings without equivalent quality do not meet the business need. Failure triggers no silent fallback; the message reports failure without a savings claim. This extension turns the course into neither a benchmark nor a production router.