Most RAG systems don’t understand sophisticated documents — they shred them
By now, many enterprises have deployed some form of RAG. The promise is seductive: index your PDFs, connect an LLM and instantly democratize your corporate knowledge.
But for industries dependent on heavy engineering, the reality has been underwhelming. Engineers ask specific questions about infrastructure, and the bot hallucinates.
The failure isn't in the LLM. The failure is in the preprocessing.
Standard RAG pipelines treat documents as flat strings of text. They use "fixed-size chunking" (cutting a document every 500 characters). This works for prose, but it destroys the logic of technical manuals. It slices tables in half, severs captions from images, and ignores the visual hierarchy of the page.
Improving RAG reliability isn't about buying a bigger model; it's about fixing the "dark data" problem through semantic chunking and multimodal textualization.
Here is the architectural framework for building a RAG system that can actually read a manual.
The fallacy of fixed-size chunking
In a standard Python RAG tutorial, you split text by character count. In an enterprise PDF, this is disastrous.
If a safety specification table spans 1,000 tokens, and your chunk size is 500, you have just split the "voltage limit" header from the "240V" value. The vector database stores them separately. When a user asks, "What is the voltage limit?", the retrieval system finds the header but not the value. The LLM, forced to answer, often guesses.
The solution: Semantic chunking
The first step to fixing production RAG is abandoning arbitrary character counts in favor of document intelligence.
Using layout-aware parsing tools (such as Azure Document Intelligence), we can segment data based on document structure such as chapters, sections and paragraphs, rather than token count.
Logical cohesion: A section describing a specific machine part is kept as a single vector, even if it varies in length.
Table preservation: The parser identifies a table boundary and forces the entire grid into a single chunk, preserving the row-column relationships that are vital for accurate retrieval.
In our internal qualitative benchmarks, moving from fixed to semantic chunking significantly improved the retrieval accuracy of tabular data, effectively stopping the fragmentation of technical specs.
Unlocking visual dark data
The second failure mode of enterprise RAG is blindness. A massive amount of corporate IP exists not in text, but in flowcharts, schematics and system architecture diagrams. Standard embedding models (like text-embedding-3-small) cannot "see" these images. They are skipped during indexing.
If your answer lies in a flowchart, your RAG system will say, "I don't know."
The solution: Multimodal textualization
To make diagrams searchable, we implemented a multimodal preprocessing step using vision-capable models (specifically GPT-4o) before the data ever hits the vector store.
OCR extraction: High-precision optical character recognition pulls text labels from within the image.
Generative captioning: The vision model analyzes the image and generates a detailed natural language description ("A flowchart showing that process A leads to process B if the temperature exceeds 50 degrees").
Hybrid embedding: This generated description is embedded and stored as metadata linked to the original image.
Now, when a user searches for "temperature process flow," the vector search matches the description, even though the original source was a PNG file.
The trust layer: Evidence-based UI
For enterprise adoption, accuracy is only half the battle. The other half is verifiability.
In a standard RAG interface, the chatbot gives a text answer and cites a filename. This forces the user to download the PDF and hunt for the page to verify the claim. For high-stakes queries ("Is this chemical flammable?"), users simply won't trust the bot.
The architecture should implement visual citation. Because we preserved the link between the text chunk and its parent image during the preprocessing phase, the UI can display the exact chart or table used to generate the answer alongside the text response.
This "show your work" mechanism allows humans to verify the AI's reasoning instantly, bridging the trust gap that kills so many internal AI projects.
Future-proofing: Native multimodal embeddings
While the "textualization" method (converting images to text descriptions) is the practical solution for today, the architecture is rapidly evolving.
We are already seeing the emergence of native multimodal embeddings (such as Cohere’s Embed 4). These models can map text and images into the same vector space without the intermediate step of captioning. While we currently use a multi-stage pipeline for maximum control, the future of data infrastructure will likely involve "end-to-end" vectorization where the layout of a page is embedded directly.
Furthermore, as long context LLMs become cost-effective, the need for chunking may diminish. We may soon pass entire manuals into the context window. However, until latency and cost for million-token calls drop significantly, semantic preprocessing remains the most economically viable strategy for real-time systems.
Conclusion
The difference between a RAG demo and a production system is how it handles the messy reality of enterprise data.
Stop treating your documents as simple strings of text. If you want your AI to understand your business, you must respect the structure of your documents. By implementing semantic chunking and unlocking the visual data within your charts, you transform your RAG system from a "keyword searcher" into a true "knowledge assistant."
Dippu Kumar Singh is an AI architect and data engineer.