Vector index and retrieved evidence

From document to controlled answer

RAG adds context retrieved from an external corpus to a question. First choose sources, prepare text, split it, compute embeddings and retain metadata. At answer time the question is embedded in the same space; the host filters candidates, ranks them and transmits only admissible excerpts. Generation remains a separate step followed by an evidence check.

The renovation uses Microsoft.Data.Sqlite without an extension and exact cosine in C#. Historical Qdrant/Docker procedures are replaced; no vector service needs installation or startup. The current lexical ranking is a real complementary branch, not a fake embedding score. Chat and embeddings have independent profiles; changing chat does not automatically rebuild the index.

Prepare and version the index

CorpusPolicy normalizes CRLF and splits at 600 characters with 80-character overlap; chars-600-overlap-80-v1 identifies this choice. sourceId#part identifiers follow ordinal source order. The delivered corpus has 15 short documents and two product facts. For long documents a rule and its exception must remain understandable; character-based splitting is a teaching compromise with no understanding of business sections.

EmbeddingIndex generates batches of 16 before opening the SQLite transaction. It checks finite, equal-dimension, nonzero-norm vectors; the store also checks their count against chunks. The manifest associates provider, endpoint, model, dimension, Cosine, chunking and the SHA-256 of the normalized corpus including product facts. A mismatch requires rebuild. Equal dimension does not mean an identical semantic space.

public async Task<float[][]> EmbedAsync(IEnumerable<string> texts, CancellationToken ct = default)
{
    var result = new List<float[]>();
    foreach (var batch in texts.Chunk(16))
    {
        var generated = await embeddings.GenerateAsync(batch, cancellationToken: ct);
        result.AddRange(generated.Select(v => v.Vector.ToArray()));
    }
    if (result.Count == 0 || result[0].Length == 0 || result.Any(v => v.Length != result[0].Length
        || v.Any(x => !float.IsFinite(x)) || !v.Any(x => x != 0)))
        throw new InvalidDataException("Invalid embedding vectors.");
    return result.ToArray();

shared/Maf.Rag/EmbeddingIndex.cs · L13–L24 · 9a895c94534c

Source code · Maf.Rag · Folder

• shared/Maf.Rag/Maf.Rag.csproj

Schema and transactional replacement

CREATE TABLE IF NOT EXISTS RagManifest (Id INTEGER PRIMARY KEY CHECK(Id=1), Json TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS RagChunks (
    Id TEXT PRIMARY KEY, SourceJson TEXT NOT NULL, Text TEXT NOT NULL, Vector BLOB NOT NULL,
    Scope TEXT NOT NULL, Status TEXT NOT NULL, Canonical INTEGER NOT NULL,
    ValidFrom INTEGER NOT NULL, ValidUntil INTEGER, Product TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS RagProducts (Product TEXT PRIMARY KEY, Json TEXT NOT NULL);

shared/Maf.Rag/SqliteVectorStore.cs · L31–L36 · 9a895c94534c

Source code · Maf.Rag · Folder

• shared/Maf.Rag/Maf.Rag.csproj

RagChunks retains text, SourceJson, Scope, Status, Canonical, dates, Product and float32 little-endian coordinates in a BLOB. RagManifest stores the index contract; RagProducts retains structured facts. rebuild replaces rows in these three tables in one transaction without deleting the file or another module’s tables. An embedding error before the transaction preserves the previous index; cancellation during the transaction rolls back replacement. Normal startup resets nothing.

Reads use a transactional snapshot and check the manifest, chunk count, identifiers, text and BLOB lengths. A removed or shortened document leaves no obsolete parts after rebuild. These checks detect common incompatibilities and corruption; they do not authenticate a database controlled by an attacker.

Measure similarity without treating it as truth

Cosine is the dot product divided by the product of norms. Summation uses double accumulators; the result is clamped to -1 through 1. The code rejects incompatible dimensions, zero vectors and nonfinite coordinates. Every eligible vector is scanned: O(N × dimension), suited to this small corpus and providing no large-scale ANN index.

public static float Cosine(float[] left, float[] right)
{
    ValidateVector(left, left.Length);
    ValidateVector(right, left.Length);
    double dot = 0, a = 0, b = 0;
    for (int i = 0; i < left.Length; i++)
    {
        dot += (double)left[i] * right[i];
        a += (double)left[i] * left[i];
        b += (double)right[i] * right[i];
    }
    return (float)Math.Clamp(dot / Math.Sqrt(a * b), -1, 1);

shared/Maf.Rag/SqliteVectorStore.cs · L146–L157 · 9a895c94534c

Source code · Maf.Rag · Folder

• shared/Maf.Rag/Maf.Rag.csproj

RetrievalPolicy requires cosine >= 0.60 and meaningful lexical overlap. RRF combines dense and lexical ranks with constant 60 and keeps at most three excerpts. Rank is neither cosine nor probability. This threshold is calibrated on the delivered fixture, not guaranteed for every domain or model; a paraphrase without a common term can be refused despite semantic similarity.

public static SearchHit[] Rank(string question, IEnumerable<SearchHit> candidates, float minCosine = MinCosine)
{
    if (!float.IsFinite(minCosine) || minCosine < -1 || minCosine > 1)
        throw new ArgumentOutOfRangeException(nameof(minCosine));
    var all = candidates.OrderByDescending(h => h.Cosine).ThenBy(h => h.Chunk.Id, StringComparer.Ordinal).ToArray();
    var terms = Terms(question);
    var lexical = all.Select(h => (h.Chunk.Id, Overlap: Terms(h.Chunk.Source.Title + " " + h.Chunk.Text).Intersect(terms).Count()))
        .Where(h => h.Overlap > 0).OrderByDescending(h => h.Overlap).ThenBy(h => h.Id, StringComparer.Ordinal)
        .Select((h, i) => (h.Id, Rank: i + 1)).ToDictionary(h => h.Id, h => h.Rank);
    return all.Select((h, i) => h with { Rank = 1.0 / (60 + i + 1)
            + (lexical.TryGetValue(h.Chunk.Id, out int rank) ? 1.0 / (60 + rank) : 0) })
        .Where(h => h.Cosine >= minCosine && lexical.ContainsKey(h.Chunk.Id))
        .OrderByDescending(h => h.Rank).ThenBy(h => h.Chunk.Id, StringComparer.Ordinal).Take(3).ToArray();

shared/Maf.Rag/RetrievalPolicy.cs · L12–L24 · 9a895c94534c

Source code · Maf.Rag · Folder

• shared/Maf.Rag/Maf.Rag.csproj

Explicit retrieval and TextSearchProvider

Host retrieval is mandatory in every variant. Demo190 additionally asks the model to call retrieve_knowledge; the tool returns the already retrieved excerpts and a counter establishes invocation. Without a call the answer is refused. Demo200 injects the same hits through TextSearchProvider using BeforeAIInvoke and RecentMessageMemoryLimit=0: it does not launch an independent search or retain six messages as the old script did. Answers use no conversation session.

if (mode == "provider")
{
    options.AIContextProviders = [new TextSearchProvider((_, cancellation) =>
    {
        cancellation.ThrowIfCancellationRequested();
        retrievalCalls++;
        return Task.FromResult<IEnumerable<TextSearchProvider.TextSearchResult>>(retrieval.Hits.Select(h =>
            new TextSearchProvider.TextSearchResult { SourceName = h.Chunk.Id, Text = h.Chunk.Text }).ToArray());
    }, new TextSearchProviderOptions
    {
        SearchTime = TextSearchProviderOptions.TextSearchBehavior.BeforeAIInvoke,
        RecentMessageMemoryLimit = 0,
        ContextPrompt = "Untrusted retrieved excerpts for the current question only:",
        CitationsPrompt = "Return exact SourceId and Quote pairs in Evidence."

shared/Maf.Rag/RagAnswerService.cs · L34–L47 · 9a895c94534c

Source code · Maf.Rag · Folder

• shared/Maf.Rag/Maf.Rag.csproj

The tool makes the step visible in events; the provider places integration in the native lifecycle. Both retain host-owned selection policy and final validation. A provider name therefore does not delegate governance to the model.

What a citation establishes

AnswerProposal contains CanAnswer and at most three Evidence entries, each with SourceId and Quote. AnswerGate requires an actually retrieved identifier, a still-eligible source and an exact quotation of at least 20 characters after trimming. An invented source, modified quote or missing evidence produces Insufficient evidence. No admissible hit stops the service before a chat client is created.

public static ControlledAnswer Validate(AnswerProposal? proposal, Retrieval retrieval)
{
    if (retrieval.Hits.Length == 0 || proposal is not { CanAnswer: true, Evidence.Length: > 0 and <= 3 })
        return ControlledAnswer.Refuse("no supported answer.");
    var sources = retrieval.Hits.ToDictionary(h => h.Chunk.Id, StringComparer.Ordinal);
    foreach (var evidence in proposal.Evidence)
    {
        if (evidence is null || string.IsNullOrWhiteSpace(evidence.SourceId)
            || !sources.TryGetValue(evidence.SourceId, out var source)
            || !CorpusPolicy.Eligible(source.Chunk.Source, retrieval.AsOf, source.Chunk.Source.Scope)
            || string.IsNullOrWhiteSpace(evidence.Quote) || evidence.Quote.Trim().Length < 20
            || !source.Chunk.Text.Contains(evidence.Quote, StringComparison.Ordinal))
            return ControlledAnswer.Refuse("invalid citation or quotation.");
    }
    return new(true, string.Join("\n", proposal.Evidence.Select(e => $"[{e.SourceId}] {e.Quote}")), proposal.Evidence);

shared/Maf.Rag/RagContracts.cs · L112–L126 · 9a895c94534c

Source code · Maf.Rag · Folder

• shared/Maf.Rag/Maf.Rag.csproj

Accepted output consists only of exact quotations; the model’s free prose is not published as a conclusion. This check establishes textual membership in retrieved excerpts, not their truth, completeness or ability to answer the whole question. A quotation cut before an exception can mislead while passing the gate. Business adequacy and contradictions between sources require separate review.

Ingest, inspect, then refuse

$env:MAF_STATE_ROOT = Join-Path $env:TEMP ('maf-rag-' + [guid]::NewGuid().ToString('N'))

dotnet run --project demos/Demo180_VectorIngestion/Demo180.App.csproj -c Release --no-build -- --Rag:Action rebuild

dotnet run --project demos/Demo190_RetrievedEvidence/Demo190.App.csproj -c Release --no-build -- --Rag:Action answer --Rag:AsOf 2026-10-23 --Rag:Question "Where can I export CSV in Reporting?"

Keep the same root throughout the trial. rebuild requires the configured embedding provider; the delivered profile uses embeddinggemma:300m independently of granite4:3b chat. probe gives a dense diagnostic before threshold/fusion; answer exercises refusal and requires chat only when hits remain. Ask an out-of-corpus question and inspect the hit list and refusal. Do not copy dated guide scores as fresh measurements.

dotnet test tests/Maf.Rag.Tests/Maf.Rag.Tests.csproj -m:1 -c Release --no-build --no-restore

Fabricated-citation experiment without a model: run the filtered command below, then read FabricatedCitationsAndAlteredQuotesAreRejected in RagTests.cs. Three proposals are rejected: INVENTED#0, an altered sentence permitting reimbursement without a receipt and a quotation that is too short. Compare ValidEvidenceRendersOnlyTheExactRetrievedQuotation, which accepts only the real passage. The RAG console has no command for injecting false Evidence; the test exercises this counterexample directly at the AnswerGate boundary.

dotnet test tests/Maf.Rag.Tests/Maf.Rag.Tests.csproj -m:1 -c Release --no-build --no-restore --filter "FullyQualifiedName~FabricatedCitationsAndAlteredQuotesAreRejected|FullyQualifiedName~ValidEvidenceRendersOnlyTheExactRetrievedQuotation"

Observation recorded on 24 October 2026: real ingestion through Ollama embeddinggemma:300m, dimension 768, followed by a new probe process on the same database. SQLite contains 15 BLOBs of 3072 bytes and two product facts. For Where can I export CSV in Reporting?, the first diagnostic hit is ACME-EXPORT#0 with cosine 0.7675654. Demo220 answer mode for GX-999 returned Insufficient evidence: no admissible retrieved excerpts. No chat call was needed. These measurements describe the delivered corpus and this local profile; the 20 RAG tests separately use known vectors. Lot-008 reception and private logs are retained.

Exercised modes and dependencies

Deterministic tests inject small known vectors into real SQLite files and compute actual cosine; they do not test an embedding provider. Live dense ingestion follows the commands above and remains separate from this software evidence. No model answer or score is invented. Run demo pages from the already built complete distribution with Maf.Rag and Maf.DemoConnection, not from an isolated demo folder.

Explore eligibility

• Demo180 — Vector Ingestion

• Demo190 — Retrieved Evidence

• Demo200 — Retrieval Provider

• Corpus governance

• Business retrieval and structured sources