Category: Technical / AI Verification Reading Time: 8 minutes

Building Confidence Scores for Mission-Critical OCR in Regulated Environments

Most AI safety research focuses on frontier problems: alignment of superintelligent systems, robustness against adversarial attacks, fairness in high-stakes decisions. These are important questions.

But there's a different class of AI safety challenge that gets less attention: how do you verify that deployed AI systems produce accurate output when mistakes have legal and safety consequences?

This isn't about preventing AI from becoming dangerous. It's about preventing already-deployed AI from producing dangerous outputs.

Menu digitisation provides a perfect case study. The AI task - extract text and allergen information from photos of physical menus - seems trivial. Any commercial OCR API can do this. But the regulatory environment demands 100% accuracy on allergen information. A single mistake can cause anaphylaxis, trigger criminal prosecution, and destroy a business.

The technical challenge: how do you achieve regulatory-grade accuracy with consumer-grade economics?

The answer: systematic verification architecture.

The Baseline Problem: Why Single-Model OCR Isn't Enough

Most menu digitisation attempts start with a simple pipeline:

Naive approach (DO NOT USE IN PRODUCTION)

def digitise_menu(menu_image): text = ocr_api.extract_text(menu_image) allergens = nlp_model.extract_allergens(text) return format_menu(text, allergens)

This works for demonstrations. It fails in production.

Problem 1: OCR Accuracy Variance

Commercial OCR services (Google Cloud Vision, AWS Textract, Azure Computer Vision) perform well on clean printed text. But restaurant menus aren't clean printed text:

Real-world menu characteristics:

• Handwritten specials boards, which are much harder for OCR to read reliably

• Multi-column layouts with unclear reading order

• Decorative fonts with poor character distinction

• Stains, creases, and physical damage

• Poor lighting in restaurant photos

• Reflections from laminated menus

• Mixed languages on a single menu

On actual restaurant menu images, single-model OCR accuracy drops well short of what regulatory compliance on allergen information requires.

Problem 2: Cascading Errors

Errors compound through the pipeline. Image quality issues feed into OCR extraction errors, which feed into text normalisation errors, which feed into allergen entity-recognition errors. Even if each individual stage has a high accuracy rate, the errors multiply through the chain rather than average out. A pipeline built from several "highly accurate" stages can still produce a cumulative error rate that is catastrophically insufficient for allergen safety.

Problem 3: Unknown Unknowns

The biggest problem isn't errors you can measure - it's errors you don't know exist. Single-model pipelines have no mechanism to estimate confidence. They'll confidently produce incorrect allergen information with no warning.

From a safety perspective, this is unacceptable. You can't fix errors you don't know about.

The Verification Architecture: Five Layers of Redundancy

Working with EasyMenus, we designed a verification pipeline aimed at regulatory-grade accuracy while maintaining economic viability. The architecture has five layers, each addressing specific failure modes.

Layer 1: Ensemble OCR with Voting

Rather than trusting a single OCR model, run three different engines in parallel:

def ensemble_ocr(menu`_image): """ Run multiple OCR engines and compare results. Flag discrepancies for human review. """ results = { 'google': google_vision_ocr(menu_image), 'aws': aws_textract_ocr(menu_image), 'azure': azure_ocr(menu_image) }

# Extract allergen-relevant sections
allergen_sections = {}
for engine, text in results.items():
    allergen_sections[engine] = extract_allergen_text(text)

# Compare results
if all_agree(allergen_sections):
    return merge_results(allergen_sections), confidence=0.95
elif majority_agree(allergen_sections):
    return majority_result(allergen_sections), confidence=0.85
else:
    return flag_for_review(allergen_sections), confidence=0.50

`

Key principle: Agreement between independent models indicates reliability. Disagreement indicates uncertainty requiring human review.

Cost implications: Running three OCR APIs multiplies the API cost, but this remains far cheaper than manual data entry.

Accuracy improvement: Ensemble OCR meaningfully outperforms a single-model baseline. Just as importantly, it identifies the subset of menus where further verification is needed, which a single-model pipeline has no way of flagging.

Layer 2: Allergen-Specific Named Entity Recognition

General-purpose NER models aren't trained on food allergen terminology across languages. We need specialised models.

Training approach:

• FSA's 14 major allergens as entity classes

• Regional terminology variations (e.g., "cacahuète" vs "peanut")

• Contextual phrases ("contains nuts" vs "nut-free" vs "may contain nuts")

• Multi-lingual corpus (English, French, Dutch, German for European markets)

Model architecture:

def extract_allergens(text, language='en'): ` """ Extract allergen entities with confidence scores. Flag ambiguous cases for review. """ # Load language-specific allergen NER model model = load_allergen_ner_model(language)

# Extract entities with confidence scores
entities = model.extract_entities(text)

# Filter for allergen-specific entities
allergens = [
    e for e in entities 
    if e.type in FSA_ALLERGEN_CLASSES
]

# Calculate per-allergen confidence
for allergen in allergens:
    allergen.confidence = calculate_confidence(
        context=text,
        entity=allergen,
        model_score=allergen.score
    )

return allergens`

Key innovation: Rather than binary classification (allergen present/absent), produce confidence scores for each allergen. This allows threshold-based routing to human review.

Accuracy improvement: Allergen-specific NER meaningfully outperforms general-purpose NER models, which aren't trained on the specific terminology and multilingual variants that allergen labelling requires.

Layer 3: Confidence Thresholding and Review Routing

Every allergen extraction gets a composite confidence score based on:

• OCR agreement across models

• NER model confidence

• Contextual validation (does the extraction make sense given dish type?)

• Historical accuracy on similar menus

`def calculate_composite_confidence(allergen_data): """ Combine multiple confidence signals. Route to appropriate processing pipeline. """ confidence_factors = { 'ocr_agreement': check_ocr_consensus(allergen_data), 'ner_confidence': allergen_data.model_score, 'context_validation': validate_allergen_context(allergen_data), 'image_quality': assess_image_quality(allergen_data.source_image) }

# Weighted combination
composite_score = (
    0.35 * confidence_factors['ocr_agreement'] +
    0.30 * confidence_factors['ner_confidence'] +
    0.25 * confidence_factors['context_validation'] +
    0.10 * confidence_factors['image_quality']
)

return composite_score

def route_menu(menu_data): """ Route menus based on confidence threshold. """ confidence = calculate_composite_confidence(menu_data)

if confidence >= 0.90:
    return auto_approve(menu_data)
elif confidence >= 0.70:
    return light_review(menu_data)  # Spot-check allergens only
else:
    return full_review(menu_data)   # Complete verification

`

Threshold selection: The confidence threshold was chosen based on cost-benefit analysis:

• Above the threshold: false positive rate low enough that auto-approval is economically viable

• Below the threshold: error rate too high for regulatory risk, human review justified

Review statistics: In production, the large majority of menus clear the confidence threshold and are auto-approved, with a minority routed to light or full human review.

Economic impact: Human review costs meaningfully more per menu than automated processing, but blending the two keeps average cost per menu low while preserving accuracy on the cases that need scrutiny.

Layer 4: Cross-Reference Validation (Where Available)

For restaurants using major suppliers with API access, validate extracted allergen claims against source ingredient data:

`def validate_against_supplier_data(menu_item, supplier_api): """ Cross-reference menu allergen claims with supplier ingredient data. Flag discrepancies for investigation. """ # Extract ingredients from menu item claimed_ingredients = extract_ingredients(menu_item)

# Query supplier database
supplier_data = supplier_api.get_ingredient_data(
    ingredients=claimed_ingredients
)

# Compare allergen profiles
menu_allergens = set(menu_item.allergens)
supplier_allergens = set(flatten_allergens(supplier_data))

# Flag discrepancies
missing_in_menu = supplier_allergens - menu_allergens
extra_in_menu = menu_allergens - supplier_allergens

if missing_in_menu or extra_in_menu:
    return flag_discrepancy(
        menu_item=menu_item,
        missing=missing_in_menu,
        extra=extra_in_menu,
        supplier_data=supplier_data
    )

return validation_passed(menu_item)

`

Current coverage: A minority of restaurants use suppliers with API access. For these restaurants, supplier validation catches a further slice of errors that the OCR/NER pipeline missed.

Future opportunity: As supplier API adoption increases, cross-reference validation becomes more valuable. This represents a scalable accuracy improvement with minimal marginal cost.

Layer 5: Audit Trail and Version Control

Every menu change logged with complete context:

`class MenuAuditEntry: """ Immutable audit log entry for menu changes. """ timestamp: datetime user_id: str change_type: str # 'ocr_extraction', 'human_review', 'supplier_update' source_data: dict # Original OCR output, supplier data, etc. confidence_scores: dict review_status: str # 'auto_approved', 'human_reviewed', 'flagged' changes_made: list regulatory_compliance_check: bool

def __str__(self):
    return f"[{self.timestamp}] {self.change_type} by {self.user_id}: {self.changes_made}"`

Regulatory value: In the event of an allergen incident, complete audit trail demonstrates due diligence:

• What was the source menu image?

• What did each OCR model extract?

• What were the confidence scores?

• Was it human-reviewed?

• When was the allergen information last updated?

Legal protection: Restaurants using verified systems can demonstrate reasonable care. This may reduce liability in allergen incidents caused by supplier errors rather than menu inaccuracy.

Performance Metrics: Real-World Results

After deploying the verification pipeline across a substantial volume of restaurant menus, the direction of the improvement is clear and consistent with what the architecture is designed to do:

Accuracy Metrics

The verification pipeline delivers a material uplift over a single-model OCR baseline across every metric that matters for allergen safety: overall accuracy, allergen detection recall, and allergen detection precision all improve, while false negatives and false positives both fall substantially.

Key insight: The reduction in false negatives is the critical outcome for safety. This is the metric that matters when a menu claims "nut-free" and nuts are actually present.

Processing Metrics

The large majority of menus are automated end to end, with only a minority requiring light or full human review, and average review time per flagged menu is short. The blended cost per menu, combining automated processing with the review layer, is designed to stay low enough to support an affordable per-restaurant subscription.

Operational Metrics

Safety record: Across the menus processed to date, VerityAI's advisory work with EasyMenus has not identified any allergen incidents attributable to the verification pipeline. The system has flagged and prevented a meaningful share of potential errors that would otherwise have reached production in an unverified pipeline.

Technical Lessons for Verification Systems

Building this verification pipeline revealed several principles applicable to other regulated AI deployments:

1. Ensemble Methods Aren't Just for Accuracy

We initially expected ensemble OCR to improve accuracy through voting. The unexpected benefit: disagreement signals uncertainty.

When three OCR models agree, confidence is high. When they disagree, something's wrong - poor image quality, unusual formatting, ambiguous text. This uncertainty signal is more valuable than marginal accuracy improvement.

Generalizable principle: In safety-critical AI, knowing when you don't know is more important than being slightly more accurate.

2. Confidence Scores Enable Economic Viability

Without confidence thresholding, only two options exist:

• Fully automated (cheap but inaccurate)

• Fully manual (accurate but expensive)

Confidence-based routing creates a third option: automated processing for high-confidence cases, which form the large majority of menus, with human review reserved for the smaller share that fall below the confidence threshold.

This hybrid approach achieves regulatory-grade accuracy at a fraction of the cost of manual processing.

Generalizable principle: AI safety doesn't mean replacing humans - it means knowing when to involve humans.

3. Domain-Specific Models Matter

General-purpose NLP models underperform on allergen extraction compared with allergen-specific models. The gap is the difference between regulatory compliance and catastrophic failure, not a marginal tuning improvement.

Training domain-specific models requires:

• Curated training data (FSA allergen lists and regional terminology variations)

• Domain expert annotation (food safety professionals, not ML engineers)

• A multi-lingual corpus, since European markets require several languages

Cost: Initial model development plus ongoing annual maintenance. The specific figures vary by market and language coverage.

ROI: Allergen incidents carry substantial regulatory penalties under EU food information law, so a domain-specific model that meaningfully reduces false negatives can pay for itself with a single prevented incident.

Generalizable principle: In regulated domains, generic models are insufficient. Domain-specific training is necessary, not optional.

4. Audit Trails Are Regulatory Insurance

The audit trail seems like technical overhead with no immediate value. But it provides critical protection:

In allergen incidents:

• Restaurant can demonstrate due diligence

• Insurance companies may reduce liability

• Regulatory authorities see systematic verification process

• Criminal prosecution less likely

In platform litigation:

• EasyMenus can demonstrate reasonable care

• Platform liability limited to negligence, not strict liability

• Defence against "should have known" arguments

Cost: Storage cost per menu is negligible against typical cloud storage pricing.

Value: Materially reduced liability exposure in the event of an incident or dispute.

Generalizable principle: In regulated AI, documentation costs are negligible compared to liability exposure.

5. Iterative Refinement Beats Initial Perfection

The verification pipeline didn't reach its current accuracy on first deployment. We got there through a sequence of iterations: identifying common error patterns in the human review queue, retraining models on the edge cases those patterns revealed, adjusting confidence thresholds based on observed false positive and false negative rates, adding supplier validation for integrated restaurants, and fine-tuning ensemble voting logic for specific menu types. Each iteration delivered an incremental accuracy gain, and the cumulative effect over the deployment period was substantial.

Generalizable principle: Perfect verification systems don't exist. Deployable verification systems improve iteratively based on production feedback.

Beyond Menu Digitisation: Applicable Domains

The verification architecture designed for EasyMenus generalises to other regulated AI deployments:

Medical Transcription:

• OCR on handwritten prescriptions

• NER for medication names and dosages

• Confidence thresholding for pharmacist review

• Audit trail for liability protection

Legal Contract Analysis:

• Multi-model extraction of key terms

• Entity recognition for parties, dates, obligations

• Confidence-based routing to paralegal review

• Version control for compliance auditing

Financial Compliance:

• Transaction categorisation for regulatory reporting

• Ensemble classification to reduce misclassification risk

• Human review queue for edge cases

• Audit trail for regulatory examination

Content Moderation:

• Multi-model toxicity detection

• Confidence thresholding for human moderator escalation

• Audit trail for appeals and legal compliance

The common pattern: AI handles high-confidence cases, humans review low-confidence cases, audit trail protects against liability.

This hybrid approach makes regulatory-grade AI economically viable for SMB markets. Restaurants can now access regulatory-compliant allergen tracking for menu digitisation at price points previously available only to enterprise buyers.

The Path Forward

Verification systems for AI-generated content represent an underexplored area of AI safety research. Most academic work focuses on:

• Adversarial robustness (defending against attacks)

• Fairness and bias (demographic parity)

• Explainability (understanding model decisions)

These are important. But they don't address the practical question: how do we know if this specific AI output is correct?

Verification systems need:

• Confidence estimation methods that are well-calibrated

• Efficient ensemble architectures that balance accuracy and cost

• Human-in-loop designs that scale economically

• Audit trail standards for regulated industries

• Domain-specific accuracy benchmarks

As AI deploys into regulated environments - healthcare, finance, legal, food safety - verification architecture will become as important as model architecture.

The question isn't whether AI can achieve 99% accuracy. The question is whether we can build systems that know when they've achieved it.

About the Author

Sotiris Spyrou is CEO of VerityAI, an AI compliance and risk management consultancy, and advisor to EasyMenus.xyz. He specialises in applying AI safety frameworks to regulated industries and has advised companies on verification systems for food safety, healthcare, and financial services applications. Sotiris writes about practical AI safety at verityai.co/blog.

Disclosure: Sotiris is CEO of VerityAI. This article describes technical verification architecture designed during a genuine advisory engagement for allergen compliance in menu digitisation systems.

Further Reading:

• When AI Safety Is a Matter of Life or Death (companion article on stakes and failure modes)

• EasyMenus Case Study: AI Safety Frameworks for Menu Digitisation (complete implementation details)

• EU Food Information Regulation 1169/2011

• FSA Allergen Guidance for Food Businesses

Technical Resources:

• Ensemble OCR implementation guide

• Allergen NER training datasets

• Confidence calibration methods for production ML

Published: 02 November 2025 Category: Technical / AI Verification Tags: #AIVerification #OCRAccuracy #RegulatedAI #AllergenCompliance #PracticalAISafety

For hands-on help, see VerityAI's AI governance and compliance.