As much as 90% of enterprise data is unstructured according to some estimates. Invoices. Contracts. Slack chats. Medical records. Claims data. PDFs that were scanned years ago. Almost none of it able to be used or acted on.

The costs of this issue are immense. The obvious one is labor. But that’s the tip of the iceberg. There are other costs and risks underneath the surface, some of which you can’t see.

Donald Rumsfeld, who served as U.S. Secretary of Defense in the early 2000s, offered an observation that has since become widely noted for its clarity and insight

There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns - the ones we don't know we don't know.

In this article we’ll walk through the various costs of unstructured data through this lens, and propose a solution to dealing with them.
‍

‍

‍

The Known Knowns: Costs You Can See

The first category of costs are the ones you know you’re spending money on, even if you underestimate them. Chief among them is labor.

The answer to the unstructured data issue has historically been to throw people at the problem. Hiring teams to read, re-key, reconcile, validate, or route documents by hand. “That’s just the cost of doing business.”

The scale of manual document work can be hard to quantify. One reason is it’s distributed across roles, departments and teams. For any individual team member it feels like a mild annoyance - a few minutes here, an hour there.

But across a thousand person org? The math gets pretty ugly.

And that’s just the time. There are second and third-order effects from all this waste. High turnover due to the mind-numbing nature of this sort of work. Slow onboarding of new team members, because you have to train them them on all the weird nuances of the various documents. The productivity drag of having your highest-value team members doing low-value tasks.

‍

Known Unknowns: Costs You Know Exist… But Can’t Quantify

Unstructured data isn’t just inefficient. There are other risks and issues that you suspect are happening, but you can’t measure. well. Examples include:

• The human error problem. Manual processes introduce mistakes. Those mistakes can trigger overpayments, delayed reimbursements, incorrect pricing, or compliance violations.
• The standardization problem. If you can’t guarantee the accuracy of your source data, it’s very hard to guarantee the accuracy of anything that then flows from it. That means forecasting, reporting, revenue recognition or customer experience data might not reflect real life. And if you can’t trust the data, you can’t make confident decisions.
• The customer experience problem. Slow or inconsistent document handling creates an inferior customer experience. Accounts are delayed in getting set up. Claims take longer than they should. Order processing is slow. Follow-ups that could be automated are manual instead. When you’re competing against organizations that have already done the work of modernizing these processes, you’re set up to lose.

‍

Unknown Unknowns: Costs You Don’t Even Realize You’re Paying

This is the deepest, most expensive layer. It’s the cost of insights you never generate. By definition they are invisible. You don’t know they exist because you’ve never seen them. Examples might include:

• Lost strategic insight. You might not know why customers call. What patterns exist in claims data. How contract terms shift over time. What bottlenecks exist in fulfillment. What risk patterns might be emerging. Because this stuff is sitting in unstructured documents, they don’t exist in your BI environment.
• Lost automation opportunities. You don’t know which decisions could be automated. What approval paths might be simplified. What tasks could be replaced with rules or models.
• Lost predictive power. You can’t see what opportunities might exist to make more proactive decisions. In forecasting. Personalization. Churn analysis. Fraud detection

‍

Unknown Knowns: The Costs That Exist But Are Invisible To You

Rumsfeld didn’t mention this one, but like any good consultants we like to think in terms of 2x2s. This is the category people often forget. There’s knowledge inside your organization that leadership doesn’t see. Examples include:

• Institutional knowledge. There are individuals in your organization who know how to interpret certain types of documents, because they’ve been around forever. But that knowledge is tribal. It’s not written down anywhere. And so it’s resident in a tiny sliver of the organization. Many of whom are probably going to retire in the next few years.
• Hidden process pathways. Unstructured data often can create a governance or knowledge management problem. It can unknowingly create situations where only a few operators know how a document should actually be handled, because the SOP is out of date, or each department has their own version. Often this manifests itself in informal workarounds and duct-taped solutions.

‍

How OCR and IDP can address these issues.

Modern Intelligent Document Processing (IDP) solutions can turn your unstructured documents into reliable data you can act on.

OCR is perhaps the most obvious example. And it’s a critical tool in this toolkit. But ideal state you have a full document processing pipeline, which includes:

• Classification (What type of document is this?)
• Extraction (What specific information is inside?)
• Validation & Normalization (Is it clean, consistent, and correct?)
• Integration (Where does it flow next?)

With this pipeline in place, you can eliminate hundreds of hours of manual work and unlock new strategic assets in the process.

In order to make this happen, you need to stop thinking of your documents like static files and start thinking of them as important data sources. You need to create a robust data foundation that allows you to get these documents in the right place. And you need to incorporate smart AI-enabled tools to help make sense of this data and help your team act on it.

In our next article, we’ll break down what a modern Intelligent Document Processing pipeline really looks like.

‍