Loop Engineering for RAG Generation: Iterate Top-K One at a Time

A user asks a question. Retrieval returns five candidate chunks and passes them all to the LLM in one prompt. The LLM reads all five to extract the same answer the first candidate already had. The other four chunks were signatures, footnotes, and irrelevant paragraphs. The pipeline paid for nothing.

This is the silent cost of naive RAG. It works, and on hard questions it is the right choice. But on easy factual ones where the top-1 already has the answer, batch feeding wastes tokens.

Two Regimes, One Decision

Retrieval hands generation an ordered top-K. There are two ways to feed it in[reference:50]:

Batch mode: One LLM call sees all K candidates. Token cost is K × chunk size. Latency is one round-trip. The LLM processes all chunks even when the top-1 was sufficient[reference:51].

Sequential mode: Treats K candidates as an ordered list. Send the top-1 first. Ask the LLM if that is enough. Stop when it says yes[reference:52].

In the example above: generation runs once on the top-1 candidate. The LLM returns that the answer is found and sufficient. Tokens spent: one chunk. That is 20% of the batch cost[reference:53].

When Batch Wins

Batch mode is the only correct mode for three question types[reference:54]:

• Exclusion questions: "Which policies do NOT apply?" Sequential would stop at top-1 and miss the rest.
• Comparison questions: "How does this year compare to last year?" The answer requires both candidates in the same call.
• Tied relevance: When top-K candidates' scores are within 5%, retrieval cannot reliably promote top-1. Batch lets the LLM arbitrate with full evidence.

Per-Question Dispatch

The clean architecture does not pick batch or sequential globally. It picks per question[reference:55]. Question shape, decomposition pattern, and intent drive the choice[reference:56].

The dispatcher reads the parsed question and routes. Four question shapes yield four routing decisions[reference:57]. Factual lookups (Amount, Date, Boolean) benefit from sequential. Lists and comparisons need batch[reference:58].

The Sufficiency Signal

Sequential mode only works if the generation brick can report whether the candidate it just saw contained the answer[reference:59]. The typed contract exposes two booleans: answer_found and answer_sufficient[reference:60].

The sequential loop reads these booleans, not a confidence float or custom heuristic. A confidence float would force a threshold that drifts model to model. Two booleans do not drift[reference:61].

Three exits drive the loop[reference:62]:

• answer_found and answer_sufficient: stop, return answer.
• answer_found but not answer_sufficient: loop to next candidate.
• Budget exhausted: stop, return best effort.

Cost Reality

For an enterprise insurance Q&A workload with K=5[reference:63]:

• Batch: 5 × chunk size per question.
• Sequential: 1 × chunk size for 80% of questions (easy factual lookups), 5 × chunk size for 20% (complex).

The ratio is significant. Sequential is the cheap default once the typed contract is in place. Batch is reserved for the question types that need it[reference:64].

Why Not Agentic Dispatch?

The series stops short of letting the LLM decide between batch and sequential[reference:65]. The dispatcher is deterministic, not agentic. The reason: audit. The same question on the same day must route the same way[reference:66]. An LLM that re-plans per call cannot give that guarantee[reference:67].

The decision is made by the parser, not by the LLM. That keeps the audit trail intact[reference:68].