Building Production-Ready
Document Processing Pipelines

Document processing demos are easy. You drop a PDF into a notebook, call an extraction API, print structured JSON to the console, and it works every time — because the demo PDF was chosen for exactly that purpose.

Production is different. PDFs arrive rotated, split across fax pages, photographed with a mobile phone in bad light, or exported from a 1998 accounting system that considers whitespace optional. The extraction API returns a confidence score of 0.4 on a field your downstream system considers mandatory. Three records arrive at exactly the same second. One document is 340 pages long.

The pipeline that handles these cases reliably is not the same pipeline as the demo. This is a walkthrough of the five layers every production document processing system needs, the failure modes in each, and what we've found works at scale.

Why demos fail in production

The gap between demo and production in document processing is usually not the extraction model. Modern OCR and LLM-based extraction is genuinely good. The gap is everything around the model: how documents arrive, how they're prepared, how outputs are validated, and how errors are handled when they inevitably occur.

The three failure patterns we see most often:

• No pre-processing layer. Documents go directly to extraction. On a clean scan this works. On a low-resolution mobile photo, a multi-page merge, or a scanned document with a coffee stain, extraction returns garbage — and nothing upstream caught it first.
• No validation layer. Extraction outputs are passed directly to downstream systems. When a field is missing or incorrectly classified, the error propagates silently. You find it three weeks later during a reconciliation.
• No exception path. The pipeline works until it doesn't. When it hits a document it can't process, it fails or drops the document. There's no queue for human review, no notification, and no audit trail of what happened.

The 5 pipeline layers

Layer 1: Ingestion

Ingestion is how documents enter your system. In production, this is rarely a single clean channel. Typical sources include email attachments, SFTP drops, portal uploads, webhook callbacks from third-party systems, and — still, in many industries — fax-to-email forwarding.

A robust ingestion layer handles:

• Deduplication. The same invoice arriving twice (once by email, once by portal re-upload) should not generate two processing jobs.
• Format normalisation. PDF, DOCX, TIFF, PNG, JPG — all normalised to a consistent internal format before any further processing. We typically normalise to PDF with embedded text where possible.
• Metadata capture. Source channel, arrival timestamp, sender, original filename. This becomes the start of your audit trail.
• Virus/malware check. For any pipeline accepting external uploads, mandatory.

Failure mode: ingestion that doesn't acknowledge receipt. The sender assumes the document arrived. It didn't. Nobody knows until something downstream fails to appear.

Layer 2: Pre-processing

Pre-processing prepares each document for extraction. This is the layer most demos skip — and the layer that makes the biggest difference to extraction quality on real-world documents.

Key pre-processing operations:

• Orientation detection and correction. Documents arrive rotated. Upside-down invoices are more common than you'd expect from a company that should know better.
• Quality assessment. DPI check, blur detection, contrast measurement. A document below quality threshold goes to an exception queue rather than extraction — because sending a low-quality document to extraction wastes an API call and gives you unreliable output.
• Page segmentation. Multi-page documents often contain multiple logical records — a 12-page ZIP file of invoices, one per page. Segmentation splits these before extraction runs separately on each.
• Classification. Invoice, purchase order, delivery note, contract, ID document — the document type determines which extraction model and template to apply.

Failure mode: treating all documents as equal quality going into extraction. A pipeline with good pre-processing routes uncertain documents to human review before extraction. A pipeline without it extracts confidently from poor inputs and returns bad data with high confidence scores.

Layer 3: Extraction

Extraction is the layer that gets all the attention in demos. It's important, but it's not where most production issues originate.

A few principles that matter in production:

• Use structured extraction, not free-form prompting. Asking an LLM to "extract the invoice data" and return JSON works until the model decides to interpret "invoice date" as the date it was processed, not the date printed on the document. Structured extraction with field-level prompts, examples, and explicit constraints is more reliable.
• Capture confidence scores per field. Not all fields are equally important. A missing "reference number" may be tolerable; a missing "total amount" is not. Field-level confidence lets the validation layer make intelligent routing decisions.
• Handle multi-model routing. For some document types, a specialised template-based extractor (Azure Document Intelligence, for example) outperforms a general LLM. For others — freeform text fields, complex tables, degraded scans — an LLM with a targeted prompt performs better. A good pipeline routes to the right model by document type.
• Log extraction inputs and outputs. When extraction produces wrong output, you need to be able to replay the extraction against the original document with a revised prompt or model. This requires logging the exact input — not just the output.

Layer 4: Validation

Validation is where you catch extraction errors before they reach downstream systems. It's the layer that most distinguishes a production pipeline from a demo.

Validation rules divide into three categories:

• Field-level rules. Is the date parseable? Is the total amount a positive number? Does the IBAN checksum validate? Is the mandatory field present? These are cheap to run and catch the majority of extraction errors.
• Cross-field rules. Does line-item subtotal sum to the stated total? Does the invoice date predate the due date? Are the referenced order numbers consistent with your supplier register? Cross-field rules catch errors that are valid in isolation but wrong together.
• Business logic rules. Does the total exceed your approval threshold? Is the supplier on the approved vendor list? Is the VAT rate consistent with the supplier's country? These are your business constraints, not just data format constraints.

The output of validation is a confidence score and a routing decision: auto-approve, flag for review, or reject. The thresholds for each category are configurable and should be reviewed as the pipeline processes more volume.

Failure mode: binary pass/fail validation that rejects anything with a missing field. This generates an exception queue that the operations team ignores because it's too noisy. Tiered, configurable validation with sensible defaults processes the easy cases automatically and routes the genuinely ambiguous ones to review.

Layer 5: Integration

Integration is the final step: delivering extracted, validated data to the downstream system it's destined for — ERP, accounting platform, CRM, database, or workflow engine.

What matters in production:

• Idempotent delivery. If the pipeline retries on failure, the same record should not appear twice in the downstream system. Every delivery should be idempotent — producing the same result whether called once or ten times.
• Error handling and retry logic. The downstream system is unavailable. The API rate limit is hit. The record fails validation at the destination. Each failure type needs a different response: retry with backoff, pause queue, alert operator, dead-letter the record.
• Delivery confirmation and reconciliation. For every document ingested, you need to be able to answer: was it delivered, when, and to what system? A daily reconciliation check — documents ingested vs documents delivered — catches silent failures before they accumulate.

Monitoring and drift detection

A document processing pipeline in production needs monitoring beyond uptime. The failure modes are often silent: extraction accuracy degrades gradually as document formats change, validation rules that were right last quarter don't cover a new supplier's format, confidence scores drift downward without crossing a hard threshold.

Metrics worth tracking:

• Auto-approval rate over time (declining = more documents failing validation)
• Confidence score distribution per document type and per field
• Exception queue volume and age (growing queue = backlog building, rules too strict)
• Human correction rate (how often do reviewers change the extracted value)
• Processing time per stage (pre-processing latency spike = quality issues at source)

The human correction rate is the most valuable signal. When reviewers regularly correct a specific field for a specific supplier, the extraction prompt for that field and supplier needs updating. A correction feedback loop — where reviewer edits feed back into improved extraction — is the difference between a pipeline that holds at 95% accuracy and one that drifts to 80% after a year.

Recommended stack for new pipelines

For a new production document processing pipeline, our current default stack:

• Ingestion: Python (FastAPI), Celery for async job queuing, Redis for task state
• Pre-processing: pypdfium2 or pdfplumber for PDF handling; OpenCV for image quality assessment; a classification model for document typing
• Extraction: Azure Document Intelligence for structured, known-format documents; GPT-4o with structured output mode for freeform and degraded documents
• Validation: Pydantic models for field-level rules; custom validation functions for cross-field and business logic; configurable routing thresholds
• Integration: Idempotent delivery functions per destination type; dead-letter queue in PostgreSQL; daily reconciliation job
• Monitoring: Structured logging to a central store; Prometheus metrics for pipeline stages; alerting on exception queue growth and accuracy drift

What this looks like in 6 weeks

A well-scoped document processing pipeline — one document type, one integration target, a defined volume range — can be in production in 5–7 weeks. The timeline breaks down roughly as:

• Week 1–2: document sample collection, pipeline architecture, extraction model selection and prompt iteration
• Week 3: ingestion, pre-processing, and classification layer
• Week 4: extraction, validation rules, and exception queue
• Week 5: integration, monitoring, and load testing
• Week 6: parallel run (pipeline + manual), sign-off on accuracy thresholds, production handover

The parallel run in week 6 is non-optional. It's the only way to validate that the pipeline's outputs match what your team would have produced manually — and it's the point where you discover the edge cases your sample collection missed.