A recurring assumption about retrieval-augmented generation systems is that when they hallucinate, the fix is a better prompt, a bigger model, or a larger retrieval window. A detailed comparison between a naive RAG pipeline and an upgraded four-stage version, run against real documents including a World Bank commodities report and NIST cybersecurity standards, argues that assumption is largely wrong. The wrong answer, in each documented case, came from a stage upstream of generation handing the model a distorted or incomplete version of the source material, which the model then answered honestly and confidently.
The baseline naive pipeline follows the pattern most RAG builds start with: parse a document into flat text, chunk it, keep the passages that best match keywords from the question, and ask a model to answer. That approach works fine on short, clean prose documents. It breaks down in four distinct, reproducible ways on denser enterprise documents.
The first failure sits in parsing. A price table in the World Bank's Commodity Markets Outlook, when flattened into plain text and cut into fixed-size chunks, separates a row's label from its numeric value, since a table is fundamentally a grid rather than a linear stream. Retrieval then hands the model chunks that never contain both pieces together, and the model correctly reports low confidence, since the alignment between label and cell was destroyed before it ever saw the question. Returning a structured, line-by-line representation with each row's spatial position intact fixed the issue entirely.
The second failure is a vocabulary mismatch rather than a ranking problem. When a NIST document referred to a concept using different terminology than the one used in the query, keyword search simply never found the right pages, since the query's own words never appeared in the source text. No amount of retrieving more passages could fix this, because the correct pages were never being searched for at all; the fix required expanding the query into domain synonyms before retrieval ran.
The third failure showed up in a 32-page NIST Cybersecurity Framework document, where a term appeared on many pages but was formally defined on only one. Keyword and similarity-based retrieval ranked every page as roughly equally relevant, so the one page that actually mattered fell below the cutoff for what got kept. Reading the document's own table of contents and routing retrieval toward the section that structurally should hold the definition fixed this, and the same routing approach reportedly held up even better on a much longer, 400-plus page NIST control catalog, where naive frequency-based ranking got worse as the document grew while structural routing did not degrade.
The fourth and final failure happened even when retrieval worked correctly. Asked for a 2026 figure in a report whose forecasts stopped at 2025, a free-text generation step grabbed the nearest available number and presented it as the answer, confidently and wrong, because nothing in an unconstrained text response forces a model to flag that the specific value it was asked for simply is not present. Requiring the model to return a structured, typed answer with a confidence field and a citable span, rather than open prose, made the missing-value case visible instead of silently filled in.
The piece is careful to note two caveats: naive RAG performs fine on short, clean, prose-only documents, and the four documented failures represent cases that held up consistently across repeated runs rather than one-off anomalies. The broader argument is that reliable RAG requires a matching contract at each of four stages, document parsing, question parsing, retrieval, and generation, rather than treating any single stage as the place where hallucinations originate.