Benchmarking a RAG pipeline: retrieval, reranking, and data quality

From a useful Playground to a repeatable benchmark

The RAG Gateway case study introduced a shared path from documents to cited answers. Its Playground made it easy to try vector search, hybrid search, reranking, and model reasoning. But one question and one request per setting could not show whether a change helped reliably, where it helped, or where it failed.

I built a benchmark client to run a blind question set against named retrieval profiles, repeat requests, and save the Gateway responses and conditions as structured JSON. That gave me a way to compare configurations across individual questions and categories, inspect failures, and revisit the results without relying on a handful of memorable examples.

The benchmark is an engineering study of one private corpus and one evolving system. Its numbers are observations, not universal rankings of Atlas features or RAG quality.

A blind question set and a separate answer key

The corpus is a curated set of Markdown documents derived from homelab notes. Questions are tagged with categories so results can be compared by the behavior being exercised. The category names are chosen by the benchmark author: this is an extensible working vocabulary, not a required or exhaustive taxonomy.

These labels can be adapted to the corpus and the evaluation goal. A benchmark can use only the categories it needs or define additional ones; consistent labels make category-level comparisons easier to interpret.

The benchmark input contains questions only. A separate answer key records expected facts and acceptable source documents. That key is never sent to the Gateway or the model under test; otherwise, the evaluation could leak the answer. An independent judge receives the answer key after a run and grades the saved answers against it.

This separation supports two different checks:

• Retrieval evidence: did the response identify an expected source document?
• Answer quality: did it give a complete, supported answer, handle conflicts, and abstain when the corpus lacked the requested fact?

They are related but not interchangeable. A retrieved source can be cited without supporting the answer, and a correct answer does not prove that it came from the expected evidence.

The examples below show the question file and its separate answer key. The run receives only homelab_benchmark_questions_only.json; homelab_benchmark.json stays offline for judging and is never sent to the Gateway or the model under test.

Question file excerpt:

{
  "name": "HomeLab RAG benchmark — questions only",
  "corpus_archive": null,
  "snapshot_date": "2026-11-11",
  "question_count": 46,
  "note": "Blind question set",
  "questions": [
    {
      "id": "HL-23",
      "category": "factuel",
      "question": "Combien de RK1 équipent la Turing Pi 2 et comment les rôles MongoDB sont-ils répartis ?"
    },
    {
      "id": "HL-32",
      "category": "multilingue",
      "question": "Which kernel release enabled Transparent Huge Pages on the RK1 nodes?"
    },
    {
      "id": "HL-46",
      "category": "information_manquante",
      "question": "Quelle est la température actuelle de la RTX 3060 du NAS ?"
    }
  ]
}

Matching answer-key excerpt:

{
  "name": "HomeLab RAG benchmark — answer key",
  "corpus_directory": "ingest_flat",
  "snapshot_date": "2026-11-11",
  "status": "Answers validated manually",
  "questions": [
    {
      "id": "HL-23",
      "expected_answer": "4 RK1: 3 data nodes and 1 management/monitoring node.",
      "expected_sources": [{
        "file": "13-rk1-cluster.md",
        "frontmatter_document_id": "homelab-13-rk1-cluster"
      }],
      "expected_document_ids": ["13-rk1-cluster.md"]
    },
    {
      "id": "HL-32",
      "expected_answer": "5.10.160-rockchip-thp1; do not claim deployment on every node is proven.",
      "expected_sources": [{
        "file": "14-rk1-kernel-thp.md",
        "frontmatter_document_id": "homelab-14-rk1-kernel-thp"
      }],
      "expected_document_ids": ["14-rk1-kernel-thp.md"]
    },
    {
      "id": "HL-46",
      "expected_answer": "The corpus contains no current measurement. Check a live source such as nvidia-smi or NAS monitoring.",
      "expected_sources": [
        {
          "file": "00-start-here.md",
          "frontmatter_document_id": "homelab-00-start-here"
        },
        {
          "file": "99-benchmark-specific.md",
          "frontmatter_document_id": "homelab-99-benchmark-specific"
        }
      ],
      "expected_document_ids": [
        "00-start-here.md",
        "99-benchmark-specific.md"
      ],
      "notes": "Do not invent a value or reuse an old measurement."
    }
  ]
}

The shared id is the join key between the blind question and its reference. The examples cover a factual lookup, a multilingual query with a qualification, and a question that should be answered by acknowledging missing live data. The full question set, answer key, and test-only adversarial fixture are not part of this article. The fixture was added to a later corpus revision to test misleading material; it is not part of the production knowledge collection.

Profiles and execution method

The matrix changes the retrieval path while keeping the requested generation settings consistent within each run:

The client supports a warmup request, repeated measured requests, a seeded question order, selection by question ID, timeouts, and optional capture of retrieval diagnostics. Each result records the requested profile, question and repetition IDs, answer, sources, timing, token counts when returned, and relevant run conditions. The benchmark keeps failures visible rather than silently retrying them.

Configuring matrix.json

matrix.json defines named profiles. The client sends the same blind questions through each profile so the resulting files can be compared by profile, question ID, and repetition. This is the matrix used for the four retrieval conditions:

{
  "profiles": [
    {
      "name": "vector",
      "description": "Baseline: vector only, no reranking, reasoning off",
      "options": {
        "hybrid_enabled": false,
        "rerank_enabled": false,
        "reasoning_effort": "none",
        "retrieval_min_score": 0.0,
        "temperature": 0.0,
        "retrieval_num_candidates": 50
      }
    },
    {
      "name": "hybrid",
      "description": "Baseline plus hybrid search; min-score cutoff is not applied",
      "options": {
        "hybrid_enabled": true,
        "rerank_enabled": false,
        "reasoning_effort": "none",
        "retrieval_min_score": 0.0,
        "temperature": 0.0,
        "retrieval_num_candidates": 50
      }
    },
    {
      "name": "vector_rerank",
      "description": "Baseline plus Atlas native reranking",
      "options": {
        "hybrid_enabled": false,
        "rerank_enabled": true,
        "reasoning_effort": "none",
        "retrieval_min_score": 0.0,
        "temperature": 0.0,
        "retrieval_num_candidates": 50
      }
    },
    {
      "name": "hybrid_rerank",
      "description": "Hybrid with vector branch reranked before rank fusion",
      "options": {
        "hybrid_enabled": true,
        "rerank_enabled": true,
        "reasoning_effort": "none",
        "retrieval_min_score": 0.0,
        "temperature": 0.0,
        "retrieval_num_candidates": 50
      }
    }
  ]
}

The shared settings hold reasoning off, temperature at zero, and the requested candidate count at 50 while the profiles vary hybrid search and reranking. The retrieval_min_score option is included for consistency, but this Gateway version does not apply that cutoff in hybrid mode. Hybrid scores use rank fusion and should not be compared numerically with vector similarity scores.

Running the benchmark from the command line

From the benchmark directory, first validate the plan without making HTTP requests:

python .\run_benchmark.py `
  --base-url https://rag.example.net `
  --tenant default `
  --questions homelab_benchmark_questions_only.json `
  --matrix matrix.json `
  --document-map document_id_mapping.json `
  --max-tokens 4096 `
  --top-p 1 `
  --seed 42 `
  --repetitions 5 `
  --warmup 1 `
  --shuffle `
  --order-seed 42 `
  --dry-run

Replace the example URL with the Gateway root URL, without /v1. Check the generated plan and document map, then run the same command without --dry-run to send requests. The client needs the actual ingested tenant. --repetitions 5 measures every selected question five times per profile; --warmup 1 adds one unscored request per profile; --shuffle --order-seed 42 gives each profile a reproducible question order. --seed 42 is passed to generation where supported, but does not guarantee identical retrieval or answers.

The optional --document-map is for interpreting results and is not sent to the Gateway. It must map filenames to arrays of verified Atlas document IDs. For the corpus snapshot used here, the persisted Atlas document ID is the filename, so for example "13-rk1-cluster.md": ["13-rk1-cluster.md"]. Keep that mapping aligned with the ingested corpus. The answer key is not a command-line input. Supply the API key through the client’s secure prompt or environment variable; do not put a secret in the command or shell history.

Use --ids HL-23,HL-46 to select specific questions or --limit 2 to make a small plan before a longer run. --output-dir keeps result files in a chosen subdirectory of benchmark. The output JSON records planned and completed requests, conditions, answers, source metadata, and timing; it can become large. Optional --capture-retrieval makes a separate post-chat retrieval request to capture passage text. That probe is diagnostic, adds work to the run, and does not prove which text the chat model saw. A warmup is excluded from measured quality and latency, and failed HTTP requests remain visible as failures.

I used one warmup and three measured repetitions for the first full run, then five measured repetitions on a focused set of six difficult questions. The focused run was designed to investigate cases that had looked surprising in the full run. The questions were shuffled consistently within each run; profiles were executed sequentially, which remains a possible source of cache or load bias.

The judge checks the run schema, matches records by question ID and question text, separates HTTP errors from scored answers, and reports complete, partial, or incorrect answers. It also checks chronology, abstention, source identity, and citation support where the output makes that possible. The judge is useful for reviewing many records, but it is not ground truth: close calls and root-cause explanations still need human inspection.

The first full run: little separation between profiles

The first analyzed run used the earlier 33-question corpus snapshot, with three repetitions per question. Each profile planned 99 measured requests. The table reports successful answers only for quality and latency; errors are listed separately.

The 502 error counts were 5, 5, 5, and 6 respectively; mean successful-response token counts were 2,283, 2,316, 2,301, and 2,283. Errors are excluded from the answer-score and latency averages.

Vector reranking had the highest mean score, but its lead over vector search was only 0.01 on this rubric. Hybrid search was nearly tied with vector search, and hybrid reranking scored lowest while recording the most errors. These observations do not establish that the search mode caused the differences: the run had one corpus snapshot, and the active model and server settings could not be verified.

The judge found recurring answer problems that mattered more than the small profile averages. Every profile mishandled one chronology-sensitive deployment-status question. Answers to a multi-source synthesis question often omitted required dependencies, and many otherwise useful answers left out a specific loopback binding. Citation support and exact faithfulness could not be verified because the chat endpoint did not expose source text and retrieval probes were disabled.

The filename matches in result metadata suggested that vector reranking returned an expected-source filename more often than plain vector search (91/94 versus 86/94 successful answers). This is only a diagnostic filename match. There was no verified Atlas ID mapping in that run, so it is not a verified Hit@K metric.

A focused run: the reranked vector profile led

The full run surfaced six difficult questions, so I repeated those five times per profile against the same earlier corpus snapshot. This run gave a clearer signal, but it is a small, deliberately difficult subset rather than a representative sample of all user questions.

HTTP 502 errors were 1, 2, 1, and 2 respectively. The answer breakdown is complete / partial / incorrect; error responses have no answer score.

Across the 28 paired successful answers shared by vector and vector reranking, reranking won 6, tied 22, and lost none. Hybrid reranking was weaker than hybrid search on the 26 shared successes: hybrid won 9, reranking won 5, and 12 were ties. These paired results help explain the averages, but they still come from one sequential run and do not justify a general performance claim.

Retrieval diagnostics told a similar story. Among successful answers, listed expected documents appeared in the top five 24/43 times for vector reranking and 18/42 times for hybrid reranking. These are hits against the answer key’s listed references, not corpus Recall@5: the source lists were not guaranteed to be exhaustive, and exact citation support remained unverified.

The most notable regression was temporal. Hybrid reranking repeatedly treated a deployment as unvalidated even though the answer key identified a later validation record as canonical. That record was absent from the top five for the affected question. This points to a retrieval and corpus-history problem worth investigating; it does not prove that reranking alone caused the incorrect answer. The same profile also answered another chronology question incorrectly in all five repetitions and gave incomplete answers on a dependency question.

What the monitoring view adds

The benchmark client reports progress and writes detailed result records, while the Gateway’s live monitoring view helps show what the service is doing during a run. It is useful operational context for spotting activity and failures, but it does not replace per-question result data or an answer-quality assessment.

The third run: first analysis of revised corpus V19

The full-corpus V19 run is complete. It tested 46 blind questions across four retrieval profiles, with five measured repetitions per question: 920 planned requests in total. The first analysis is available, but it remains preliminary and needs deeper review before supporting a firm configuration choice.

These expected-document hit rates count matches against the benchmark’s listed references; they are not corpus Recall@K, because the expected-source lists may be incomplete. On paired successful observations, reranking raised expected-document coverage by 11.2 percentage points for vector and 9.0 percentage points for hybrid, an average gain of about 10 percentage points. These are absolute gains in expected-document retrieval, not a 10% relative gain or evidence of better answer accuracy. The paired deltas use matched successful observations, so they differ slightly from subtracting the aggregate rates in the table. The judge analysis later corrected its initial answer-quality summary; those scores and paired correctness comparisons still need reconciliation against the question-level records.

Each profile also had 12 short HTTP 502 failures, spaced every 19 records, while the failed question IDs varied. That pattern is not tied to one question. NAS kernel logs confirmed a memory-cgroup OOM in the related Ollama incident, and the OOM issue has since been resolved by increasing the resource allocation in the TrueNAS Ollama application configuration; see the Gateway incident analysis. The benchmark files do not include per-failure kernel or cgroup logs, so they cannot independently attribute every recorded 502 to an OOM event.

V19 also exposed a French-language retrieval trap in the revised corpus. An older document used the conditional form SERAIT, while a more recent document used the imperative wording À NE PAS UTILISER. This is a reason to use Atlas Search’s lucene.french analyzer instead of the language-neutral lucene.standard analyzer for this French corpus: the Atlas Search analyzer example shows French-aware normalization of accented and inflected terms.

This exact example also needs a careful analyzer check. Lucene’s default French stop-word list includes ne, pas, and serait, so French analysis can remove the negation and conditional markers that distinguish these statements. Inspect the tokens produced for both passages and keep the dated source context visible; the analyzer does not resolve their different meanings or chronology by itself. The Lucene French stop-word list documents these exclusions.

Next steps, in order:

1. Investigate the counterintuitive hybrid-plus-rerank results. Compare candidate lists and ranks before and after fusion and reranking, and identify which query categories drive the gap. Reconcile the corrected judge summary and audit answer support as part of this analysis.
2. Then test reasoning off and on while holding the model, retrieval profile, corpus, and generation settings constant. Verify the model-specific thinking control rather than assuming the requested setting was applied.
3. After that, compare other LLMs using the same questions, retrieval conditions, and scoring rubric.

The OOM issue that interrupted V19 has been resolved through a higher resource allocation for the Ollama application in TrueNAS. Question-level examples, expected answers, and the adversarial fixture remain private.

Limits that shape the conclusions

These experiments are reproducible enough to find useful leads, but several gaps limit the claims I can make:

• Model and runtime settings: the Gateway returned a public alias, not a verified underlying model identity. Some requested sampling and reasoning settings could not be confirmed as active.
• Execution order: profiles ran sequentially, so caching and service load could affect timings and possibly answers.
• Failures: HTTP 502 responses were not scored. Excluding them from answer means is necessary, but error rates are part of the operational result.
• Citation evidence: the chat response did not include the exact chunk text. A separate retrieval probe would be diagnostic evidence, not proof of the exact context used for generation.
• Retrieval identity: filename matches are weaker than a verified mapping from source files to Atlas document IDs. Expected source lists are acceptable references, not a complete relevance judgment.
• Judge limitations: an LLM judge can apply a rubric consistently at scale, but it may miss nuance or overstate a likely cause. Scores should be read alongside examples and source evidence.
• Scope: this is one curated private corpus, one Gateway implementation, and a small number of run configurations. The results do not generalize automatically to other datasets, models, embedding choices, or workloads.

Conclusion: curate first, then compare carefully

The benchmark changed my view of what needed attention. In the first full run, all four profiles landed close together. On the focused set, vector reranking led, while hybrid reranking regressed on several chronology and synthesis questions. The evidence is useful for choosing what to investigate next, but it is not strong enough to declare one retrieval strategy universally best.

The most durable lesson so far is about the corpus. Adding more documents or a more elaborate retrieval path cannot resolve conflicting source history by itself. A small, well-curated collection with clear canonical updates gives embeddings, rank fusion, reranking, and the model a better chance to return useful evidence. When evidence conflicts or is missing, the system should expose uncertainty instead of turning an old statement into a confident answer.

The next phase will first explain the hybrid-plus-rerank results, then isolate the effect of reasoning, and finally compare other LLMs. With the Ollama resource allocation increased to resolve the OOM issue, reruns can also check whether the earlier 502 pattern disappears. The goal is not to find the fastest profile in isolation, but to understand which configuration produces the most reliable answers for this dataset and why.