Using Docugami to Add Metadata to Chunks for High Accuracy Document QA#
One issue with large documents is that the correct answer to your question may depend on chunks that are far apart in the document. Typical chunking techniques, even with overlap, will struggle with providing the LLM sufficent context to answer such questions. With upcoming very large context LLMs, it may be possible to stuff a lot of tokens, perhaps even entire documents, inside the context but this will still hit limits at some point with very long documents, or a lot of documents.
For example, if we ask a more complex question that requires the LLM to draw on chunks from different parts of the document, even OpenAI’s powerful LLM is unable to answer correctly.
chain_response = qa_chain("What is rentable area for the property owned by DHA Group?")
chain_response["result"]  # the correct answer should be 13,500
' 9,753 square feet'
At first glance the answer may seem reasonable, but if you review the source chunks carefully for this answer, you will see that the chunking of the document did not end up putting the Landlord name and the rentable area in the same context, since they are far apart in the document. The retriever therefore ends up finding unrelated chunks from other documents not even related to the Menlo Group landlord. That landlord happens to be mentioned on the first page of the file Shorebucks LLC_NJ.pdf file, and while one of the source chunks used by the chain is indeed from that doc that contains the correct answer (13,500), other source chunks from different docs are included, and the answer is therefore incorrect.
chain_response["source_documents"]
rtdocs_stable/api.python.langchain.com/en/stable/modules/indexes/document_loaders/examples/docugami.html