The tension in enterprise document processing is straightforward: a fast, free parser like PyMuPDF can process a page in five milliseconds, but it often loses structure. A vision-language model can extract anything from a complex diagram, but at ten thousand times the cost and ten seconds per page. Running the heavy parser everywhere is financially untenable, while running the cheap parser everywhere produces confidently wrong answers from broken parses.
The solution is adaptive parsing: start cheap and escalate only when the pipeline detects that the cheap parse missed the answer. A new technical walkthrough from the ongoing "Enterprise Document Intelligence" series demonstrates exactly how this feedback loop works in practice, using two real-world examples from the seminal "Attention Is All You Need" paper.
The Cascade: Eight Checks Before You Pay
The architecture builds an escalation cascade across four core bricks: document parsing, question parsing, retrieval, and generation. The first eight checks are ordered from cheapest to most expensive. Checks one and two run on every page in milliseconds, using deterministic signals like character density and a flat-table fingerprint to flag likely parse failures for free.
Checks four through six operate during question parsing and retrieval, catching issues before generation even begins. Only if those upstream checks give the page a clean bill of health does the pipeline reach check seven: the LLM's self-evaluation flag, baked into the typed answer schema. Check eight, a separate groundedness check using a fresh LLM or NLI model, runs after the answer is produced to catch any fabrications the generating model was too confident to flag itself.
Case A: Escalating a Flat Table to Azure Layout
The first walkthrough addresses a common failure mode. The question asks for the value of h for the base model in Table 3 of the Attention paper. The correct answer is 8, located in a filled cell on page nine.
PyMuPDF parses the page, returning the 13 cells of the base row as 13 separate, unconnected lines. The column headers are similarly flattened, some spanning multiple lines. During the first generation pass, the LLM (GPT-4.1) correctly identifies the value as 8. However, its structural confidence flag triggers False. To map line 25 to the "h" column, the model had to count column positions in a flattened header. It got it right, but the risk was real.
This binary flag is the trigger. The orchestrator escalates the page to Azure Document Intelligence, which returns the table as pipe-anchored markdown with each row on a single line. On the second generation pass, the LLM returns the same value, 8, but this time with full structural trust. The cost of the escalation was about $0.003 and four seconds, a small price for an answer that is now auditable and structurally verified.
Case B: Escalating a Diagram to a Vision LLM
The second walkthrough addresses a different failure: Figure 1, the encoder-decoder architecture diagram. The question requires understanding the diagram itself, not just the surrounding prose.
PyMuPDF returns the prose and a placeholder row for the figure with no actual content. The LLM produces a partial answer with a structural caveat. Azure DI is of no help here because the diagram is free-form vector geometry, not a table. The pipeline reads the caveat, routes to a vision-language model, and receives a structured description of the architecture.
With the visual description appended to the context, the LLM produces a complete answer with high confidence, naming details like the residual connections, layer norm placement, and cross-attention bridge that the prose alone did not enumerate. The cost: five seconds for PyMuPDF, plus 10-30 seconds and a few cents for the vision call on a single image, while the other 14 pages of the paper never see a vision model.
The Stress Test: When the LLM Signal Isn't Enough
The walkthroughs represent the best-case scenario. A stress test across eighteen runs, using three questions of growing structural difficulty, three models, and two parsers reveals a critical weakness: the LLM's self-evaluation flag is unreliable on its own.
When a weaker model gave the wrong answer, it did not flag the structural issue. It fabricated an answer and asserted the structure was clean. Conversely, the only time the structural flag fired, it came from a model that did not need rescuing (GPT-4.1 with PyMuPDF on the simplest question). Replacing the binary flag with a continuous score didn't help either; scores clumped between 0.85 and 1.00 regardless of correctness, producing a 60% false-alarm rate at a useful threshold.
However, when asked for a one-sentence rationale alongside the flag, the model's description of the parse shape was concrete and consistent. The verdict on top of that rationale was unstable, swinging wildly across repeated runs on the same input. The takeaway is clear: a feedback loop that reads the rationale and applies a deterministic rule, such as escalating when the rationale contains phrases like "split across multiple lines," is stable. Reading the binary flag or score directly is not.
The Data Model: Caching and Auditability for Free
The pipeline's data model carries the cascade. A parser_name column on every parsing table allows mixed parses, escalation, audit, and caching to be reduced to filtering on a column. Over time, the cache fills with the enrichments that questions triggered. The most-queried regions of the most-queried documents get the deep-parsed version automatically, while rarely-touched regions stay on the cheap parse.
End-to-end auditability is preserved throughout. The annotation brick converts the evidence span into a rectangle drawn on the source PDF. A reviewer can scan the highlighted page and verify in seconds that the cited cell matches the LLM's claim, even if the feedback loop fired and the page was re-parsed with a heavier tool.
The next installment in the series will apply this self-correction framework to cross-references, demonstrating how the same skeleton supports any new failure shape.