The three techniques are cross-encoder reranking, disabling MMR, and document-awareness prefixes. Each one looks like a clear win in its own ablation. Together they deliver much less than their separate results add up to.

Why it matters

Most teams decide what stays in a RAG pipeline one ablation at a time. If a stage wins its ablation, it goes in, and the expected gain of the full stack is the sum of the wins. Every stage adds latency and cost to every query, so when the sum does not hold, the team pays for gains it never sees.

HyDE is a small example from the same paper. It raises NDCG@10 by 3.1% but doubles first-hop latency, which reduces the latency-adjusted gain to 0.9%.

The setup

The paper (arXiv 2610.29722) uses a framework called lasagna to study how RAG components compose. It runs 167 configurations chosen with a D-optimal design (Fedorov et al., 1972; Atkinson et al., 2007), across eight configurable pipeline stages. A D-optimal design picks a subset of the full grid of configurations that still lets the effect of each factor be estimated. The study covers eight benchmarks and corpus sizes from 5K to 200K documents. It adds 138 non-factorial ablation configurations across 24 targeted experiments.

Retrieval is scored with NDCG@10. Generation runs report faithfulness and answer relevance from RAGAS v0.4 (Es et al., 2024), judged by GPT-5.1, plus exact match, token-level F1, p50/p95 latency and cost per query. GPT-5.1 also generates the answers for every configuration. Faithfulness here measures whether the generated claims are supported by the retrieved context.

The complication

The natural reading of a set of winning ablations is that the gains stack. The paper finds that they partly overlap. Its ablations suggest that reranking already provides some of the filtering benefit of disabling MMR, which leaves less room for the two to help when combined.

The balanced pipeline is not an outlier. Including all five separately positive retrieval techniques raises the shortfall to 72%. Across 17 tested combinations, the shortfall ranges from 7% to 72%, with a median of 41%.

The same kind of overlap shows up between the pipeline and the generator. A stronger generator helps the lightest pipeline far more than the heaviest, so a heavier stack buys less when the generator is already strong.

There is a second complication. Embedding, enrichment and reranking dominate the retrieval-quality metric, while query augmentation and agentic gating dominate answer faithfulness. Across matched configurations, the two metrics are essentially unrelated, with a pooled Pearson r of -0.01 (p = 0.91). So in this study, an NDCG@10 gain says almost nothing about whether the answers became better supported.

The evidence

The chart puts the predicted lift next to the realized one, and the difference between the two bars is the 63% shortfall.

The factorial analysis shows the same split between the two metrics. It reports η² (the share of a metric’s variance associated with one factor) for each stage. On retrieval, the largest marginal associations are embedding (η² = 0.164), enrichment (η² = 0.144) and reranking (η² = 0.090). On faithfulness, query augmentation (η² = 0.224) and agentic gating (η² = 0.217) lead, while embedding and enrichment are negligible (η² < 0.02 each). The authors note that these values are unadjusted, meant for exploratory screening, and not additive, because the design is unbalanced and the effects share variance.

Several single components also behave differently than you might expect. MMR reduces NDCG@10 by 3.6% relative on average, opposite its usual benefit on broad, heterogeneous corpora. The 2019 MS-MARCO MiniLM-L6 cross-encoder (Nogueira & Cho, 2019) ranks above all four recent rerankers tested. The authors speculate that its training aligns especially well with their focused domains. Chunker choice spans 0.126 NDCG@10 across 20 configurations, a 36% relative difference larger than most factorial main effects.

Corpus size changes which pipeline wins. The standard profile leads at 5K and 15K documents, and balanced leads from 25K onward. At 200K, balanced exceeds standard by 13.7% relative NDCG@10. Section 5 reports p = 0.003 for that gap, and the appendix reports p = 0.004 under Welch’s t. The two values are consistent with a different test or reporting protocol, and the gap is significant under both. The 25K and 100K gaps are directionally consistent but underpowered.

Between the two agentic profiles, balanced has 4.5% higher NDCG@10 than max-reliable, with only a 0.005 faithfulness difference. Section 5 gives its response-time advantage as 13%, and the appendix gives 13.5% lower mean response time. The two figures are consistent with rounding.

The formal interaction test is more cautious than the headline numbers. The pre-declared test of the main interaction hypothesis, H1, was not met. A focused follow-up, pre-specified before it ran, confirms that the query augmentation × agentic gate interaction is real (p = 0.007), but far below that bar. Its partial η² is 0.033, and 10 of 20 combinations depart from additivity by at least two percentage points. Under the study’s own variance rule, 0.033 sits on the additivity-holds side of the 5% floor. So the combined pipelines fall well short of the summed gains, while the one interaction tested formally stays below the bar the authors set in advance.

Limits

The authors call their findings case-study evidence, and their derived profiles candidates for further validation rather than prescriptions for high-stakes deployments. The benchmarks sit in five focused domains: biomedical, financial, scientific, argumentative, and multi-hop QA over Wikipedia. The scaling distractors come from those same five domains, so broad, mixed corpora remain untested.

Faithfulness depends on one judge, GPT-5.1. A second judge agrees only moderately (Pearson r = 0.44, Cohen’s κ = 0.35), so relative trends hold up but exact faithfulness scores depend on the judge. Raw faithfulness is also partly driven by answer length.

The hypotheses were pre-declared but not filed as a third-party pre-registration. The design matrix, per-query scores and analysis scripts are not in the public release, and are available from the authors on request. The corpora run from 5K to 200K documents, so larger indexes are outside what was measured.

Takeaway

Before adding another stage to a RAG pipeline, measure the combined pipeline end to end on your own corpus. Track retrieval quality and answer faithfulness separately, because in this study one barely tracked the other.

If a stage only earned its place in a solo ablation, I would rerun the full pipeline without it before paying its latency on every query.

The study is a reproduction of five factuality methods that include Faith, FactGRPO and Flame. Its authors also found that none of the methods reproduces its published results in full.

Why it matters

If you choose a factuality method from the tables in its paper, you may end up with a model that answers more briefly, pads answers with off-topic claims, or gets worse on short questions. Faith and FactGRPO lose about 25 and 18 points of TruthfulQA accuracy against the base model. Of FactGRPO’s biography responses, 36% contain claims that are supported by evidence but irrelevant to the prompt.

The setup

The original papers are hard to compare with each other. None of them shares a model or dataset with all the others, three report precision-like metrics, one credits refusals and one excludes them.

So the authors trained every method on Llama-3.1-8B-Instruct (Grattafiori et al., 2024), with the training regime and data that its own paper specifies. Each alignment stage is a LoRA adapter of rank 32 (Hu et al., 2022), placed on top of the method’s own supervised stage and trained with its paper’s hyperparameters.

The common benchmark has two long-form generation tasks scored with VeriScore (Song et al., 2024) and two short-form question answering tasks graded by an LLM judge. VeriScore is a claim-level scorer. It splits a long answer into short factual claims, checks each claim against retrieved evidence, and counts how many are supported. Precision is the share of claims that are supported. F1@K combines that precision with how many supported claims the answer contains, measured against a target count K. A relevance-filtered score counts only the claims that address the prompt.

Each method is compared with the untrained base model on the same four datasets. The authors apply Holm’s correction for multiple comparisons. Holm’s correction raises the bar for significance when many tests run at once, so that a few lucky differences are not counted as wins.

The complication

Each method arrives with published results. Under each method’s own protocol, all methods except Faith and FactGRPO follow a similar trend to their original implementations, but none reproduces its published results in full. Faith and FactGRPO do not follow a similar trend.

On the common benchmark, the published gains mostly vanish. On closer inspection, the authors find that the rewards optimise for format, length or brevity rather than factuality and relevance. The same analysis finds that the choice between on-policy and offline training matters little.

The evidence

Start with the short-form tasks. Faith and FactGRPO lose about 25 and 18 points of TruthfulQA accuracy. The other three methods stay within two points of the base model, which is not statistically significant. The judge shapes the absolute numbers here, since regrading with gpt-oss-120b changes up to a third of the TruthfulQA labels, but the large drops of FactGRPO and Faith persist.

Accuracy on its own can hide how differently these models behave. On SimpleQA, Flame raises its accuracy over attempted questions by declining more often, but its overall accuracy stays unchanged.

Neither of the two long-form exceptions holds up well. Faith has the highest precision on the biography task. The authors show that this is an artifact of 0.4 supported claims in a one-sentence, very brief response, which is why its relevance-filtered precision is significantly above the base model. A model that says very little has little room to be wrong.

FactGRPO is the other exception. Its relevance-filtered biography F1@K is 0.342, against 0.325 for the base model, and the difference is marked significant. But its reward counts a claim as correct whenever evidence supports it, even if the claim is irrelevant to the prompt. Of its biography responses, 36% contain such claims.

The picture shows the TruthfulQA losses: Faith and FactGRPO got about 25 and 18 fewer answers right per 100 questions than the base model, while the other three stayed within two points.

The verifier matters as much as the judge. Re-verifying biography claims against the subject’s Wikipedia article lowers precision by about twenty points for methods that produce long responses.

A separate sensitivity check leaves the order of the methods almost unchanged, but FactGRPO’s one significant gain no longer survives the paired test.

Two further results are worth knowing. The authors report refusal rates of 0.000 on three datasets and 0.0003 on the fourth, even though 17% to 56% of questions are classified as unknown. In one reproduction, 400 steps of Dr. GRPO left the reward flat, and precision rose slightly while the supported-claim count fell, against the published gains in both.

Limits

The common evaluation uses one 8B model with LoRA adapters, four English datasets, and smaller training budgets than the original papers. Results may differ with larger models, longer training or full fine-tuning. Each method was trained once, so variance across random seeds was not measured.

The Dr. GRPO run used a tenth of the published optimisation, so the authors say it bounds a tenth of the budget rather than the method. The short-form judge decides the absolute numbers. The authors also note that the lack of improvement on these metrics does not rule out factuality gains measured in other ways.

Takeaway

Before adopting a factuality method, check its answer length, claims per answer, refusal rate and claim relevance. Then check whether its gain over the untrained base model survives a different verifier or judge.

The more useful lesson here is about evaluation: a factuality score means little until a different verifier or judge agrees with it.

Why it matters

If you switch thinking off, you probably expect short answers at lower latency and token cost. But if reasoning text shows up anyway, the short-answer half of that plan has already failed. The cost half is shaky too. The authors warn that a response that looks answer-only should not be read as cheaper inference, because hidden reasoning may still be running.

The setup

The authors split every response into a pre-answer trace and a final answer, and then score it at three levels.

The models tested include DeepSeek-V4-Flash and GPT-5.4. The study runs four models through APIs, plus Qwen3-4B and Qwen3-32B locally. For the four API models, each mode uses 1,000 sampled items from each of BoolQ, MMLU, MMLU-Pro, GSM8K and MATH, plus all 687 StrategyQA items. The local Qwen3 runs use the full benchmark splits.

Every mode gets the same 8,192-token response budget, so a shorter cap in no-think mode can’t explain the differences in output. The “minimal” setting is treated as near-off, and not as fully off.

Whether a response shows visible explicit inference is decided by a blinded AI judge, GPT-5.5 at temperature 0. Humans labelled 600 outputs to check that judge. Their labels agree with GPT-5.5 on 98.8% of them, and with a second judge, Claude Opus 4.8, on 96.7%.

Two of the paper’s measures come up below, EIR and ETR. Both are the paper’s own metrics, so take their exact definitions from the paper itself.

The complication

The obvious reading is that the switch decides whether a model reasons out loud. But the paper finds something else: visible inference is organized more by the task’s answer space than by the switch itself. Answer space here means the kind of answer a task asks for, from yes/no and multiple choice through to open-ended answers the model has to build.

The authors call this “thinking inertia”. They use the term for visible reasoning that persists under no-thinking controls and becomes more common as the answer space opens.

The open-ended end of that range is the clearest. There, answer-only responses collapse to nearly zero everywhere, while explicit inference stays high. The authors say this is consistent with open answers needing more intermediate computation than picking from narrow or visible options. They word it as a fit with what they observe, but stop short of proving the cause.

GPT-5.4 is the main exception to a pattern the paper reports on yes/no and multiple-choice tasks. On both, it stays answer-only most of the time, with correspondingly low explicit inference.

The evidence

The picture walks through one real reply and then the paper’s headline case: with thinking disabled, DeepSeek-V4-Flash still shows visible explicit inference in 99.9% of its open-ended responses.

The first fix most people would reach for is a stricter prompt. Across the five models with thinking fully disabled, strict answer-only prompting raises mean ETR from near zero to roughly 40%. But the same prompting also reduces mean accuracy by about 15 percentage points. Both are means across the five models, and the paper itself hedges them with “roughly” and “about”.

The paper’s worked example shows where that accuracy can go. Stronger direct-answer pressure can remove the computation needed to check a subtle constraint, in that case the word “smallest” in the question.

A prefilled answer template, the paper’s M6 setting, does no better. It neither reliably produces answer-only responses nor consistently suppresses visible explicit inference. In the authors’ words, it is “not a stronger version of Strict”.

The paper also rewrites closed questions as open-ended ones. On that rewrite, accuracy falls to 52.6%, ETR drops to 0%, and EIR rises to 90.14%. The numbers for the original format aren’t quoted here, so read these as levels after the rewrite and leave aside the size of the change.

That 0% can look like it clashes with the roughly 40% from strict prompting. But the two figures are consistent, because the two experiments change different things. Strict prompting keeps the tasks and changes the instruction, which pushes the model toward answer-only output. The rewrite changes the task itself into an open-ended one, which is exactly where answer-only output collapses.

The result also holds up with a different judge. Claude Opus 4.8 finds visible explicit inference in 61.02% of open-ended outputs under the paper’s M5 condition, close to the 62.00% that GPT-5.5 finds.

Limits

The biggest limit is what the paper measures. Its claims concern visible output only, and it does not treat provider signals as a description of internal reasoning. So a clean answer-only response does not show that the model did less work, or that the call cost less.

The authors leave multilingual, code, tool-use, multi-turn and stochastic settings open, along with how the results hold as proprietary endpoints change. If you build agents, treat this as a strong hint for your setup, since it was not tested there.

The API results come from 1,000-item samples per benchmark, and not from full test sets. Explicit inference is scored by an AI judge, but the human check on 600 outputs makes that less of a worry.

Takeaway

If your product turns thinking off, read a sample of real responses for each type of task you serve, and start with the open-ended ones. Check whether reasoning text still shows up in them. Before you add a stricter answer-only prompt, measure accuracy on your own tasks, because the paper saw mean accuracy fall by about 15 points. And don’t take answer-only output as proof that cost went down.

I’m wary of any answer-only prompt that makes output look tidy until someone measures what it does to accuracy.