AI Text Extraction Eval Pipeline

The Problem

The company uses an AI service to extract indexing fields from recorded documents — grantors, grantees, consideration amounts, parcel IDs, legal descriptions. Validating whether that extractor was accurate enough for production was entirely manual: retrieve ground truth from the data lakehouse, download document images one by one through a web UI, submit each document to the extractor individually, manually compare extracted fields against ground truth, and somehow aggregate results into a defensible accuracy claim. The process took roughly eight days per batch. With schema versions shipping regularly, there was no scalable way to validate that a new version didn’t regress accuracy — or to prove that it improved it.

The Solution

A single CLI command — or a natural-language prompt via Claude Code — that fetches ground truth from the data platform, downloads document images, submits all documents to the extractor concurrently, compares every extracted field against ground truth using type-appropriate matching logic, and produces a multi-rubric accuracy report with confidence intervals and diagnostic recommendations. The output doesn’t just say whether the extractor is accurate; it says where it’s failing, why, and what to do about it.

Outcome & Impact

Eight days of manual work per evaluation batch became approximately 15 minutes per automated run. The initial phase validated across hundreds of documents spanning multiple document types and counties. Two significant findings — a role-semantics mismatch that appeared as catastrophically low accuracy, and a systematic business-logic divergence in consideration amounts — were identified and correctly classified by the pipeline. Neither would have been detectable at scale through manual spot-checking.

The pipeline is smoke-tested and functional. A larger-scale phase targeting statistically significant verdicts across broader document and county coverage is planned.

My Role & Contributions

Aspect	Detail
Role	Sole designer and builder
Team size	1
Timeline	~2 weeks (concept through validated smoke runs)
Scope	Pipeline architecture, all five stages, statistical methodology, diagnostic logic, Claude Code skill integration
Key decisions	What to automate and in what order; type-aware comparison strategy; schema-aware null handling; designing output to answer “what’s wrong and what do we do” rather than just “what’s the number”

Technical Overview

┌──────────────────────────────────────────────────────────┐
│                  Evaluation Pipeline                     │
├──────────────────────────────────────────────────────────┤
│                                                          │
│  Stage 1: Ground Truth Fetch                             │
│  Databricks SQL → lakehouse join → XML parse → packages  │
│                                                          │
│  Stage 2: Document Retrieval                             │
│  OAuth2 token → document API → PDFs                      │
│  (Playwright browser automation as fallback)             │
│                                                          │
│  Stage 3: Extraction + Comparison                        │
│  PDFs → extractor API (async, throttled) → results       │
│                                                          │
│  Stage 4: Verdict + Diagnostics                          │
│  Results → multi-rubric scoring → diagnostic tree        │
│                                                          │
│  Stage 5: Report                                         │
│  Claude Code synthesizes artifacts into final report     │
└──────────────────────────────────────────────────────────┘

Ground truth extraction joins multiple tables across the data lakehouse and parses XML blobs from the source system to recover human-indexed field values. Document retrieval supports two modes: an OAuth2 API path with auto-refreshing tokens and configurable rate limiting, and a Playwright browser automation fallback for environments where the API auth path is blocked.

The comparison layer uses 18 field comparators with type-specific matching strategies. Party names use substring matching with role awareness. Consideration amounts use numeric tolerance. Dates use format-agnostic parsing that handles multiple conventions. Legal descriptions use fuzzy token-set matching. A naive exact-match approach would have reported dramatically lower accuracy than the extractor actually achieves; the multi-rubric design makes the gap between normalization issues and true extraction errors immediately visible.

The verdict engine produces four scoring rubrics — strict, fuzzy, semantic, and subset — with Wilson 95% confidence intervals on all accuracy claims. A six-branch diagnostic tree maps coverage, accuracy, and consistency signals to specific action recommendations. Every numeric value in the output is traceable to its source documents via a claim registry.

Schema-aware null handling loads county-specific field definitions to distinguish legitimately optional fields from data quality anomalies in the ground truth itself — a distinction that prevented hundreds of false negatives in the initial runs.

Challenges & Key Decisions

Automation surfaced patterns that manual spot-checking never could

The first substantive finding wasn’t about extraction accuracy at all — it was about ground truth assumptions. Documents in one category appeared to have roughly 11% party name accuracy, which would have killed the extractor’s credibility. Automated comparison across hundreds of documents made the pattern immediately visible: the source system indexes one party role while the extractor correctly identifies a different role that is semantically accurate for the document type. A human checking ten documents would have concluded the extractor was broken and moved on. Programmatic reclassification resolved the category correctly.

Systematic divergence vs. true extraction failure

In another category, consideration amounts appeared systematically wrong. The pipeline made it clear this was a business logic difference rather than an extraction error: the extractor returns gross sale price while the source system records taxable selling price after statutory exemptions. Both values are correct — they answer different questions. The pipeline flagged these as pending audit rather than marking hundreds of documents as failures.

Ground truth is never as clean as assumed

The source system data has its own quality issues — fields left blank that should be filled, inconsistent indexing conventions across counties. Schema-aware null handling and anomaly detection on the ground truth itself prevented weeks of chasing false negatives from data quality problems in the reference data.

Tech Stack

Layer	Technologies
Language	Python
Data	Databricks SQL, data lakehouse
Auth	Azure AD OAuth2 (two service principals)
Libraries	RapidFuzz, Playwright
Orchestration	Claude Code skill

Lessons Learned

• Automation surfaces patterns that spot-checking never will. Both major findings only became visible because the pipeline compared hundreds of documents simultaneously. Scale is what makes systematic patterns legible.
• Ground truth is never as clean as assumed. Building anomaly detection for the reference data itself — not just the extractor output — prevented weeks of chasing false negatives.
• The diagnostic signal matters more than the accuracy number. Knowing the extractor is 80% accurate is far less useful than knowing the 20% gap is entirely normalization differences that a post-processing layer would fix.