Your organization runs on documents. Invoices. Applications. Contracts. Claims. Lab results. Each of which contains critical information. Each of which rarely arrives to you in a clean, structured, machine-readable way.

We’ve talked previously about the risks this can create for the business, and how Intelligent Document Processing (IDP) can be a solution to that problem. In this article we’ll dig deeper into what IDP is and how you can leverage it to finally unlock the value hidden in your unstructured data.

‍

What Is IDP?

IDP is a pipeline or process that takes your documents, identifies what they are, pulls the most important data out of them, validates that data based on rules you create, and then exposes that data in a structured way for other systems to use.

This is important, because you have hundreds or thousands of different document types, and they rarely follow a single format. They almost certainly don’t respect your business rules. As a result they’re not ready to be used by any of your BI or analytics teams, nor are they able to be leveraged easily by AI or automation workflows.

IDP uses a combination of technologies to accomplish this, including Optical Character Recognition (OCR), Artificial Intelligence (AI), Machine Learning (ML), and Natural Language Processing (NLP). But you can think of the end result like a bridge between your unstructured mess of documents across the organization and the clean and orderly world of your databases and workflows sitting on the other side.

How an IDP pipeline works

There are as many IDP implementations as there are companies. But the architecture underneath the hood tends to be pretty consistent, and usually has some form of the following six stages:

1. Ingestion. Collecting all of your documents from various places. This could include email inboxes, scanned documents, PDF in your EHR, etc.
2. Image Pre-processing. This is a critical step for image-based documents, as it makes every future step much more accurate. It will correct the alignment of the image, de-noise them to make them clearer, convert them to black and white, and crop them as needed.
3. Classification. AI models categorize your documents by type (this one is an invoice, this one is a contract, etc.) This makes sure each document is sent to the right workflow.
4. Data Extraction. OCR converts the text into machine readable format, while NLP and ML models try to understand and extract the most important data points.
5. Validation and Enrichment. The extracted data gets validated for accuracy. This might involve cross-referencing it with existing databases (matching a vendor name to a master list, for example). It could also involve a manual escalation process for human review when confidence is low.
6. Integration and delivery. You deliver this clean and validated data where it needs to go. Typically this would include places like your ERP, CRM, EHR, data warehouse, etc. Increasingly this now also means any of your AI-enabled workflows.

What IDP Unlocks for the Business

Once you have a pipeline like this in place, you can do some exciting things. IDP helps you:

• Reduce manual work and tedious data entry.
• Improve accuracy and compliance because business rules are encoded in the pipeline.
• Create consistency across vendors, turning everything into one standardized format.
• Speed up decision-making, by getting data into an environment where it can actually be used orders of magnitude more quickly.
• Generate insights previously hidden in documents - things like trend data, exception patterns, root causes, risk signals, customer themes, etc.
• Enable automation and AI.

How to Get Started with IDP

Step 1: Define Objectives.

As with any technology, the first step is to get clear on what you’re trying to solve for. In this step you will surface potential use cases, and prioritize the ones that have the highest potential ROI.

Step 2: Assess Current Capabilities and Develop a Business Case

You need to look at your current infrastructure to identify any technical gaps that might exist. Armed with the right use case and a good understanding of your current state, you can create a compelling business case to get buy-in.

Step 3: Find the Right Solution.

You’ll want to find a solution that gives you high accuracy with extraction, can support various document types (and languages), integrates with your existing systems through documented APIs, can scale without performance hits (usually via cloud-based solutions), and has the right security and compliance protocols in place (HIPAA, etc.) Make security a top priority from the beginning - it’s hard to layer this in later.

Step 4: Prepare Your Data

This is a time-consuming but critical step. Invest the time to prepare and label your dataset for training. High quality training data materially impacts the final result.

Step 5: Pilot and Refine

Start with a pilot test on a small subset of documents to assess its performance. Find any issues or edge cases, fine-tune the rules around extraction, etc. Take advantage of Human-In-The-Loop when needed here. This can help verify low-confidence data and further train the model.

Step 6: Deploy and Manage Change

Once you have a successful pilot, you can deploy across the relevant departments. Critical to this step is having a clear change management strategy in place. That typically will include user training, documentation, and support channels for handling questions.

Step 7: Monitor and Optimize

IDP is not a set-and-forget exercise. You’ll want to monitor processing time, error rates, and straight-through processing rates (STP). You’ll also want to talk to end users to find ways to further streamline or improve the process.

Start your IDP Journey

IDP is a highly practical and accessible solution to your unstructured data problem. It allows you to dramatically reduce operating costs, minimize errors, and free your team up to focus on more strategic work. If you’d like help standing up your first IDP pipeline, don’t hesitate to reach out.

‍