I want to extract “chunks” of documents, which is useful for many workflows.
For example:
How can I do this ?
This post makes some references to https://community.palantir.com/t/how-to-do-pdf-annotation-in-workshop/3289/2
There are many ways to compute “chunks” from PDFs:
For pictures, this would only be rectangles of “interest” on the images.
There are multiple solutions to source documents:
Here, we will take the easiest example by manually uploading documents in a mediaset
We will want to process those documents, to extract some element of interest.
We need to:
The processing can be divided in multiple stages: preprocessing, text extraction, chunking, entity or information extraction and post-processing. This can happen at multiple granularity: at a whole document level, per page, or per “element” (e.g. extracted by a layout model).
There is a whole diversity of strategies and options to perform this processing. See below picture to have a high level view of it.
Here we will take a simple example where we have PDF from which we want to extract specific pieces of information statically (e.g. date of the document, who signed the document, presence or absence of an element, etc.) and have a reference of where this is in the document.
We will see two options:
If the documents are produced digitally, the text can be extract “exactly” (aka, not via OCR). This allows for an exact match during searches in the document. This is the simplest setup.
Note: this approach might still work even on scanned documents PDFs, as you can still rely on the page number to locate the source of the extracted information. The user will have to locate the extract within the defined page which is sometimes good enough for some workflows.
The annotation are based on text extracts of the document.
Let’s setup the processing:
Split the text in semantically coherent sections, these sections semantically have a start and end, and speak about one topic.
These chunks MUST contain all of the text from the topic and can span multiple lines. Section headers are useful references for chunking.
When the topic changes, the chunk changes as well.
A corresponding question/answer pair should be included in the same chunk.
Different Question/answers pairs should be chunked in different chunks, as its semantically different.
Create these "chunks" by adding "<CHUNK_BREAK>" (no spaces) into the text as a chunk delimiter. Do not change anything else in the provided text. Do not remove the <page_number> delimiter.
If you see a new line in text, keep the \n character. Do not change a single character of the provided text.
We now split each pages into its different chunks, by splitting on the character we introduced, exploding the array and extracting both the content and the position of each chunk within the page. We as well filter out empty chunks as those would not be useful.
We can now process each chunk, for example to extract entities or information that is relevant for our use-case to generate annotations we can display.
Note: We can do much more with those chunks, like validating the LLM didn’t alter the text, generate titles, extract information and aggregate back at document level, first validate which chunks contain this information, etc.
Note: Those chunks are very useful and can be used for other workflows, like RAG pipelines, by creating embedding vectors etc. which can easily be done in Pipeline Builder but out of scope for this tutorial.
You are a tool for <industry> which extract information from <type of documents>
You are looking for <type of information>, these are <description>
Your goal is to parse the following page text into an array of JSON structs where every entry contains the following fields:
- <key>: <description of what to extract, format, type>
- <key>: <description of what to extract, format, type>
- <key>: <description of what to extract, format, type>
If the text doesn't contain an a <type of information> return null
The output should be a valid JSON (without \n or or other escaping)
At this stage, we obtained
If the document are not digital (aka, no text layer exists in the PDF), it is possible to use OCR or Layout models to extract the information.
Note: for documents containing parts that should be processed differently (e.g. charts, schematics, tables, …), it is possible to use a mix of Vision models and Layout extracted to extract granularly each piece of information (e.g. all data points of a chart, etc.). This won’t be covered here as it is more advanced but it is possible.
The first option is to do the same as for Solution A, but to use the “OCR” option in step 4. The processing will be the same, but you won’t be able to highlight the exact text segment once in Workshop, but only jump to the page from which it was extracted (and this is often enough !).
The second option is to use a Layout Model. As this model is experimental, it might not be available or enabled on your Foundry enrollment.
If available you will see the “Document Information Extraction” model in Model Catalog (navigate to workspace/model-catalog).
If you don’t see it, you might want to reach out to the administrator of your Foundry enrollment, who might be able to enable this model in Control Panel.
The chunks will have exact coordinates of “rectangles” on the page. where the structure of the page is used to extract blocks of information.
As of today, you can call it from Code Repository but this might be soon available in Pipeline Builder.
Let’s do the layout extraction in Code repository (see this other post):
https://community.palantir.com/t/how-to-use-a-layout-model-on-pdfs/3290/2
With this approach you will have
You might want to create a dataset containing elements we wanted to extract, with a reference to the chunk it was extracted from, but this step is similar Solution A - steps 7 and 8 in particular.
At this stage, we have a few datasets produced by our pipeline (via code repository or pipeline builder):
Objects types will be linked together like on the below picture.
You can create the Object Types like from any dataset, by clicking the top right “Actions > Create Object Type”
You can use the MediaItemRid (not stable between pipelines snapshot) or the “path” (stable between pipelines snapshots) of the document as a primary key.
You can then create links between the Object Types, using the foreign/primary keys:
We now have an Ontology of 3 Object Types. We can create our application to show the document, the chunks and the pieces of information we want.
Let’s build our application
Display the chunk on the document will depends on the information stored in your chunk, and so on your choices you made earlier in the tutorial.
If you have only the “page number” in your chunks
If you have only text in your chunks
Very similar to the page scroll, you can pass some text to highlight in the document.
Note: This is an actual “search” of text in the document. Some PDFs have special characters (bullets lists, etc.) which are not extracted during the Pipeline processing, and hence cannot be matched back on their original PDF as the actual text in the chunk and text in the PDF will not exactly match. In this case, you might need to pivot to the other approaches (page or chunk).
If you have the coordinates of the box of the chunk
We will this time configure the PDF Viewer widget by passing the variable containing all the chunks, to the widget, and specifying which property of the Object Type contains the bounding box.
Alternative: You can also display only the selected bounding box of your choice by passing a “select” variable containing the chunks you want to display (e.g. active or multi-selected chunks).
Note: You will also obtain a variable containing the “currently selected bounding box”, see arrow in purple on the below picture.
Note: You can use this “selected annotation” variable both way - to get the selection of the user in the PDF Viewer, but as well to select a particular bounding box. Hence you can pass all the chunks to the PDF Viewers, and highlight the one that is currently actively selected in the object list.
Hi Vincent,
Thanks a lot for this super informative post. I would love to use the “Document Information Extraction” model. I have set up the example given in the Model Catalog, but I am getting a 400 Error. Could you provide an example with code and input and output data and media sets? Would be super helpful!
Thank you,
Philipp
