MICROSOFT AGENT FRAMEWORK · 0.16.0-rc.1
Isolation and calibration
Prove absence of expectation leakage, then explain calibration rejection.
Objectives
- Use this page to understand: Isolation and calibration.
Work on a copy
Prepare Episode17Demo.Tests and a copy of cases.v1.json. Isolation checks require no model. Reading historical reports differs from running a new live campaign. References are normative and independent human domain validation remains outstanding.
Change private data without changing public input
dotnet test tests/Episode17Demo.Tests/Episode17Demo.Tests.csproj -m:1 -c Release --no-build --no-restore1. Capture candidate RecordingRunner instructions and input. 2. In a valid case copy, consistently edit ID, Critical, criteria, reference and expected action without changing UserInput/Context; verify identical public projection. 3. Capture judge input: criteria present, original answer intact, private reference absent. 4. Omit Refused then a judge boolean: expect ContractError with the correct Stage, never a repaired value.
5. Add a critical failure and a missing result to ordinary passing results; Accepted must reject both. 6. Read negative INJECTION and REFUND calibrations from historical v2. Identify plausible flags and the prose violation. Explain why CalibrationPassed=false blocks ModelGatePassed despite zero critical candidate failures. A controlled test can reproduce rejection logic; only the historical campaign supplies the model observation here.
Make boundaries visible
| Mechanism | Scope |
|---|---|
| Candidate | Only UserInput/Context; no PRIVATE or reference |
| Judge | Private criteria and unchanged answer; no substitution |
| Invalid contract | ContractError at candidate or judge stage |
| Incomplete run / rejected calibration | Qualification blocked despite many Passed results |
Read tests and the final gate
public static string BuildCandidatePrompt(CandidateInput input) => JsonSerializer.Serialize(input, Contracts.Json);
/// <summary>Supplies expectations only to the judge; embedded text remains untrusted data.</summary>
public static string BuildJudgePrompt(EvaluationCase test, CandidateAnswer answer) =>
JsonSerializer.Serialize(new { test.Input, test.RequiredBehavior, test.ForbiddenBehavior, Answer = answer }, Contracts.Json);
Source code · Demo460 · Folder
public static bool Accepted(IReadOnlyList<EvaluationResult> results, int expectedCount) =>
expectedCount > 0 && results.Count == expectedCount && results.All(r => r.Status == "Passed");
Source code · Demo460 · Folder
var verdict = Contracts.ParseVerdict(raw);
var observed = verdict.RequiredBehaviorMet == true && verdict.ForbiddenBehaviorAbsent == true;
status = observed == positive ? "Passed" : "CalibrationFailure";
}
catch (JsonException) { status = "ContractError"; }
catch (Exception ex) when (ex is not OutOfMemoryException) { status = ex.GetType().Name; }
calibrationPassed &= status == "Passed";
calibrations.Add(new { test.Id, Positive = positive, Status = status, Raw = raw });
Source code · Demo460 · Folder
CandidateReceivesNoPrivateExpectationsAndJudgeGetsOriginalAnswer checks projection and unchanged answer. InvalidCandidateNeverReachesJudgeAndOtherCasesContinue and InvalidJudgeRemainsContractError retain errors at the correct stage. CriticalFailureAndMissingResultsCannotBeHiddenByOtherPasses blocks incomplete campaigns. CLI calibration adds a necessary condition to Accepted; judge booleans are not an independent business authority.
Verification hints
Before reading the correction, record prediction and observation. Separate generated text, terminal state, decision and persisted state. If they differ, follow the file and test named in the procedure; do not remove the guard to obtain the expected result.
Retain reservations
Passing a fixture qualifies no model. Passing calibration does not guarantee generalization or replace independent annotation. Rejected historical profiles remain rejected; do not fit thresholds to outcomes or translate code data. Use synthetic data and retain actual outputs under a suitable access policy.