Demo460 — Candidate Evaluation

Prepare suite, prompts and directory

Demo460.App distributes evaluation/cases.v1.json, prompt_candidat_v1.txt, prompt_candidat_v2.txt and prompt_juge.txt. Defaults are fixture mode, v2 prompt and one repetition, as defined in Program.cs. Choose a new output directory for every run; an existing directory is rejected. SiteMAF runs no model for its reader.

dotnet build BaseTechnique.slnx -m:1 -c Release

• Demo190 — Retrieved Evidence

• Demo450 — Contract Regression

Exercise the pipeline without a model

$fixtureOutput = Join-Path $env:TEMP ("maf-evaluation-fixture-" + [guid]::NewGuid().ToString("N"))

dotnet run --project demos/Demo460_CandidateEvaluation/Demo460.App.csproj -c Release --no-build -- --mode fixture --prompt v2 --repeat 2 --output $fixtureOutput

Get-Content (Join-Path $fixtureOutput "report.json")

The fixture deliberately replays the reference matching public input and returns a scripted positive verdict. Its internal case lookup is not the live candidate inference path. Expect Measurement=ScriptedFixtureOnly, FixtureGatePassed=true, ModelGatePassed=false, CalibrationPassed=null. Two repetitions of eight cases expect sixteen results; this count follows parameters and corpus rather than measuring quality.

FixtureGatePassed = mode == "fixture" && accepted,
ModelGatePassed = mode == "live" && accepted,
CriticalFailures = results.Count(r => r.Critical && r.Status != "Passed"),
ContractErrors = results.Count(r => r.Status == "ContractError"),
CalibrationPassed = mode == "live" ? (bool?)calibrationPassed : null,
Calibrations = calibrations, Results = results

demos/Demo460_CandidateEvaluation/Program.cs · L102–L107 · 9a895c94534c

Source code · Demo460 · Folder

• demos/Demo460_CandidateEvaluation/Demo460.App.csproj

Run a separate live campaign

$liveOutput = Join-Path $env:TEMP ("maf-evaluation-live-" + [guid]::NewGuid().ToString("N"))

dotnet run --project demos/Demo460_CandidateEvaluation/Demo460.App.csproj -c Release --no-build -- --mode live --prompt v2 --repeat 2 --output $liveOutput

This command requires Ollama with the common model available, or an explicitly selected API connection. It calibrates on sixteen references/counterexamples before evaluating actual answers. --judge-model accepts another already installed model on the same endpoint; it does not select another provider. The runner’s global bound is twenty minutes, allowed repetitions are 1 to 10 and model parameters are temperature 0 / MaxOutputTokens 800, as defined in code.

To compare v1 and v2, retain identical suite, judge, models/digests, binaries, endpoint and parameters and use two new directories. Compare statuses by CaseId/Repetition and check qualification gates in each report. Historical Compare-Evaluation scripts are not distributed; do not present them as a current command.

Inspect calibration and isolation

dotnet test tests/Episode17Demo.Tests/Episode17Demo.Tests.csproj -m:1 -c Release --no-build --no-restore

public static string BuildCandidatePrompt(CandidateInput input) => JsonSerializer.Serialize(input, Contracts.Json);

/// <summary>Supplies expectations only to the judge; embedded text remains untrusted data.</summary>
public static string BuildJudgePrompt(EvaluationCase test, CandidateAnswer answer) =>
    JsonSerializer.Serialize(new { test.Input, test.RequiredBehavior, test.ForbiddenBehavior, Answer = answer }, Contracts.Json);

demos/Demo460_CandidateEvaluation/src/PromptEvaluator.cs · L10–L14 · 9a895c94534c

Source code · Demo460 · Folder

• demos/Demo460_CandidateEvaluation/Demo460.App.csproj

    var verdict = Contracts.ParseVerdict(raw);
    var observed = verdict.RequiredBehaviorMet == true && verdict.ForbiddenBehaviorAbsent == true;
    status = observed == positive ? "Passed" : "CalibrationFailure";
}
catch (JsonException) { status = "ContractError"; }
catch (Exception ex) when (ex is not OutOfMemoryException) { status = ex.GetType().Name; }
calibrationPassed &= status == "Passed";
calibrations.Add(new { test.Id, Positive = positive, Status = status, Raw = raw });

demos/Demo460_CandidateEvaluation/Program.cs · L76–L83 · 9a895c94534c

Source code · Demo460 · Folder

• demos/Demo460_CandidateEvaluation/Demo460.App.csproj

Open report.json, manifest.json and cases.json/candidate.txt/judge.txt snapshots. Check Mode, Measurement, SharedModel, Calibrations, Results, Stage and raw outputs. Exit status is 1 whenever a gate fails; retain failure artifacts. For historical granite4:3b, the qualification rejection documented in the chapter remains the dated reference result even after passing application tests.

• Candidate, judge and qualification

Retain the qualification reservation

No model is qualified by the fixture or this page. The dependent judge can accept harmful counterexamples; a campaign with no invalid contract can still fail. Reference review is internal to Codex; independent human domain approval has not been obtained. Raw reports require suitable access and retention when replacing synthetic data.

Exercise boundaries

• Isolation and calibration

• Candidate, judge and qualification

• Deterministic regression