Back to Work
AI-Assisted Document Verification
Caribou Financial
Designing the review architecture for assisted verification and conditional automation
Executive Summary
As Caribou's refinance volume grew, the verification workflow stopped scaling. Agents were reconciling document uploads, application data, and partner requirements across two disconnected systems: the primary application platform and a separate internal operations tool, while tracking review progress through manual notes. The result was slower throughput, inconsistent decisions, and applications stalling at the verification step.
The project manager and I focused on defining the verification architecture before touching interface design. We mapped the full review lifecycle across both systems, interviewed loan officers to understand where review decisions broke down in practice, and worked with operations leads and agents to translate partner rule documentation into validation logic. The architecture combines queue-based review, rule-driven checklist validation, and AI-assisted document extraction in a single workflow. The system supports assisted verification today while enabling conditional automation as extraction reliability and rule validation thresholds are met.
Measured Impact
Enabled verified applications to progress without manual interventionConfidence thresholds and rule validation gated conditional automation, reducing review overhead for clean applications.
Eliminated cross-system reconciliation delaysUnified workflow state replaced manual comparison across two tools with independent review records.
Reduced inconsistent verification outcomesPartner and document rules embedded into a generated checklist removed agent reliance on recalled logic.
Lowered processing cost and prevented review interruptionsBatch extraction before queue load eliminated repeated service requests and decoupled review from extraction latency.
Overview
Each refinance application requires verification of core documents: driver's license, vehicle registration, and insurance, plus additional supporting documents depending on applicant attributes, partner policies, and state regulations. Caribou's partner network introduced enough variation in acceptance rules that agents could not apply a single fixed checklist across applications.
Customer data and uploaded files lived in the application platform. Verification progress and assignment were managed in a separate internal operations tool. Because these systems maintained independent workflow states, agents compared information across both tools and tracked decisions through manual notes, creating delays, dropped context during reassignment, and inconsistent outcomes across the team.
An AI-assisted extraction system was being introduced to identify fields from uploaded documents, but reliability varied by document type and image quality. Because extraction accuracy was not yet sufficient to support full automation, I designed the system to support assisted verification first, with a path to conditional automation once reliability thresholds were met.
Verification Architecture Evolution
- Supported Platform: Internal operations tool (web)
- Timeline: Jul - Dec 2025 (MVP Phase 1: Jan 2026)
- Primary Audience: Operations agents, operations leads
Discovery
Mapping the Verification Workflow
The project manager and I started by tracing the full document review lifecycle across the application system and the operations tool, following how agents accessed uploads, validated application data, and tracked review status across both tools. We interviewed loan officers to understand where verification decisions broke down under real operating conditions, then catalogued required document types, field comparisons, and approval conditions. These were mapped against outputs from the extraction system to identify where automated results aligned with agent decisions and where manual verification was still required.
This mapping surfaced four structural problems:
- Agents manually reconciled data across systems with no shared state
- Review progress was tracked inconsistently, causing context loss during reassignment
- Partner rule logic was documented but not embedded in the workflow, so agents applied it from experience
- Extraction outputs were not integrated into the review interface, requiring agents to cross-reference results separately
Early workflow mapping and review layout sketch used to define verification controls and agent workflow
Checkpoint Audit
I reviewed each validation step against documented partner conditions and regulatory requirements. Several checks existed not because any rule required them, but because the workflow had accreted informally over time. Stripping those out was as important as formalizing what remained.
The project manager's rule documentation became the source of truth for validation logic. We confirmed each rule through proof-of-concept reviews with agents. This surfaced interpretation gaps between how rules were written and how agents actually applied them in edge cases, which shaped the final checklist model.
Design
Verification Architecture
Verification decisions depended on document type, partner requirements, and state rules. Because of this variability, a fixed screen sequence would have required separate flows for every partner configuration. We designed a rule-driven checklist model instead, so each application generates the appropriate validations dynamically.
The architecture addressed four structural issues from the prior workflow:
- Unified workflow state: Customer data, document status, and verification progress are handled in one system. Agents no longer reconcile records across tools or lose context during reassignment.
- Queue-based review model: Applications move through a shared queue where agents can prioritize by status, assignment, or urgency, replacing the manual lists and notes agents previously maintained.
- Rule-driven validation: Each document generates a checklist derived from partner rule mappings. Agents complete verification without referencing external documentation or applying logic from memory.
- Staged automation model: I designed for assisted verification first, with automation enabled only when both confidence thresholds and rule validation conditions are satisfied. Embedding automation before the underlying system was consistent would have introduced errors that were harder to catch and correct than manual review failures.
Agent Review Workflow
The prior workflow had no structured way to surface rule failures during review. Agents caught them by memory or missed them entirely. When a document enters review, the interface now displays extracted fields, validation checks, and rule failures together. Passing checks remain visible for reference but require no action. Failed rules move the document into manual review with the specific failure surfaced. Agents do not have to identify what is wrong, only resolve it. Agents can correct extracted values, apply allowed exceptions, or override rules. Required notes ensure exceptions are recorded before approval becomes available, creating an audit trail for compliance review.
The verification flow keeps document preview, rule validation, and exception handling in a single panel so agents can resolve issues without leaving the review step. System checks run automatically, AI suggestions are shown with confidence states, and manual review is required only when rules cannot be satisfied automatically. The workflow moves from review required, to assisted validation, to verified state, ensuring partner rules and audit requirements are satisfied before the application advances.
Review required
The interface shows detected fields, rule checks, and exception requirements together. Passing checks remain visible but require no action, while failed rules move the document into manual review. Agents can override rules, apply an allowed exception, and add a required note before approval becomes available.
Detection and validation
Detected values are highlighted on the document and linked to rule checks in the validation panel. Confidence states and rule status are visible at the same time so the agent can accept the suggestion, edit the value, or apply an exception without leaving the review view.
Verified state
After all checks pass or are resolved, the document moves to a verified state. All validations show as passed or resolved, manual review is cleared, and the primary action advances the workflow to the next required document, preventing incomplete applications from progressing.
Production
Phase 1 - January 2026
Launched as a controlled evaluation under real operating conditions. Automation was restricted by reliability thresholds - documents could only progress automatically when both rule validation and extraction confidence requirements were satisfied. The evaluation confirmed that the checklist model reduced reliance on agent experience for partner-specific rules and that queue-based assignment improved throughput visibility for operations leads. It also surfaced edge cases in multi-document applications where rule dependencies between documents created sequencing gaps the checklist model did not yet handle, which informed the scoping of Phase 2 gating logic.
Phase 2
Introduced conditional automation, allowing documents to bypass manual review once extraction accuracy and validation logic consistently met partner requirements. The project manager and I reviewed the workflow and rule logic with product, engineering, and operations to confirm that approval states matched actual review behavior before enabling automation at scale.
Decisions & Trade-offs
Automation vs. reliability: Approvals occur only when checklist validation and confidence thresholds pass. A flagged approval is still an approval. In a regulated lending workflow, the cost of an incorrect decision outweighs the cost of a manual review step. Allowing low-confidence results to proceed with a flagging mechanism was considered and rejected on that basis.
Unified workflow state vs. cross-system sync: Consolidating into one workflow required migrating verification state that operations had managed in the separate tool for years. The alternative of syncing state between systems introduced a new failure mode where sync delays would recreate the same reconciliation problems the project was designed to eliminate.
Batch extraction vs. real-time processing: Real-time extraction was the initial plan. Batch processing was adopted after latency failures stalled agents mid-review in early testing. Batch added a pre-processing step but removed extraction latency from the critical path entirely.
Rule-driven checklists vs. per-partner fixed flows: A fixed flow per partner configuration was faster to build initially. Dynamic checklist generation from rule mappings was chosen because the partner network was growing. Fixed flows would have required new design and engineering work for every new partner condition.
Manual override vs. full automation: Override remains available when extracted values or rule checks do not meet reliability thresholds. Removing it entirely was considered for Phase 2, but operations leads needed the ability to approve exceptions for edge cases that rule logic could not fully anticipate.
Synthesis
The hardest part of this project was not designing the interface. It was establishing the conditions under which automation could be trusted. That meant mapping the existing workflow precisely enough to know where rule logic diverged from how agents actually applied it in edge cases, validating each check against documented requirements rather than observed behavior, and designing staged review so that automation expanded only as the system earned it.
In regulated workflows, AI reliability is not just a model problem. It is a system design problem. Extraction accuracy matters, but so does whether the rules are correct, whether workflow state is consistent, and whether agents have enough visibility to catch what automation misses. The architecture has to be trustworthy before the automation can be. That principle shaped every decision in this project, and it is the one I would carry into any AI-assisted workflow operating under compliance constraints.
Next Project
Back to Work
AI-Assisted Document Verification
Caribou Financial
Designing the review architecture for assisted verification and conditional automation
Executive Summary
As Caribou's refinance volume grew, the verification workflow stopped scaling. Agents were reconciling document uploads, application data, and partner requirements across two disconnected systems: the primary application platform and a separate internal operations tool, while tracking review progress through manual notes. The result was slower throughput, inconsistent decisions, and applications stalling at the verification step.
The project manager and I focused on defining the verification architecture before touching interface design. We mapped the full review lifecycle across both systems, interviewed loan officers to understand where review decisions broke down in practice, and worked with operations leads and agents to translate partner rule documentation into validation logic. The architecture combines queue-based review, rule-driven checklist validation, and AI-assisted document extraction in a single workflow. The system supports assisted verification today while enabling conditional automation as extraction reliability and rule validation thresholds are met.
Overview
Each refinance application requires verification of core documents: driver's license, vehicle registration, and insurance, plus additional supporting documents depending on applicant attributes, partner policies, and state regulations. Caribou's partner network introduced enough variation in acceptance rules that agents could not apply a single fixed checklist across applications.
Customer data and uploaded files lived in the application platform. Verification progress and assignment were managed in a separate internal operations tool. Because these systems maintained independent workflow states, agents compared information across both tools and tracked decisions through manual notes, creating delays, dropped context during reassignment, and inconsistent outcomes across the team.
An AI-assisted extraction system was being introduced to identify fields from uploaded documents, but reliability varied by document type and image quality. Because extraction accuracy was not yet sufficient to support full automation, I designed the system to support assisted verification first, with a path to conditional automation once reliability thresholds were met.
Verification Architecture Evolution
- Supported Platform: Internal operations tool (web)
- Timeline: Jul - Dec 2025 (MVP Phase 1: Jan 2026)
- Primary Audience: Operations agents, operations leads
Discovery
Mapping the Verification Workflow
The project manager and I started by tracing the full document review lifecycle across the application system and the operations tool, following how agents accessed uploads, validated application data, and tracked review status across both tools. We interviewed loan officers to understand where verification decisions broke down under real operating conditions, then catalogued required document types, field comparisons, and approval conditions. These were mapped against outputs from the extraction system to identify where automated results aligned with agent decisions and where manual verification was still required.
This mapping surfaced four structural problems:
- Agents manually reconciled data across systems with no shared state
- Review progress was tracked inconsistently, causing context loss during reassignment
- Partner rule logic was documented but not embedded in the workflow, so agents applied it from experience
- Extraction outputs were not integrated into the review interface, requiring agents to cross-reference results separately
Early workflow mapping and review layout sketch used to define verification controls and agent workflow
Checkpoint Audit
I reviewed each validation step against documented partner conditions and regulatory requirements. Several checks existed not because any rule required them, but because the workflow had accreted informally over time. Stripping those out was as important as formalizing what remained.
The project manager's rule documentation became the source of truth for validation logic. We confirmed each rule through proof-of-concept reviews with agents. This surfaced interpretation gaps between how rules were written and how agents actually applied them in edge cases, which shaped the final checklist model.
Design
Verification Architecture
Verification decisions depended on document type, partner requirements, and state rules. Because of this variability, a fixed screen sequence would have required separate flows for every partner configuration. We designed a rule-driven checklist model instead, so each application generates the appropriate validations dynamically.
The architecture addressed four structural issues from the prior workflow:
- Unified workflow state: Customer data, document status, and verification progress are handled in one system. Agents no longer reconcile records across tools or lose context during reassignment.
- Queue-based review model: Applications move through a shared queue where agents can prioritize by status, assignment, or urgency, replacing the manual lists and notes agents previously maintained.
- Rule-driven validation: Each document generates a checklist derived from partner rule mappings. Agents complete verification without referencing external documentation or applying logic from memory.
- Staged automation model: I designed for assisted verification first, with automation enabled only when both confidence thresholds and rule validation conditions are satisfied. Embedding automation before the underlying system was consistent would have introduced errors that were harder to catch and correct than manual review failures.
Agent Review Workflow
The prior workflow had no structured way to surface rule failures during review. Agents caught them by memory or missed them entirely. When a document enters review, the interface now displays extracted fields, validation checks, and rule failures together. Passing checks remain visible for reference but require no action. Failed rules move the document into manual review with the specific failure surfaced. Agents do not have to identify what is wrong, only resolve it. Agents can correct extracted values, apply allowed exceptions, or override rules. Required notes ensure exceptions are recorded before approval becomes available, creating an audit trail for compliance review.
The verification flow keeps document preview, rule validation, and exception handling in a single panel so agents can resolve issues without leaving the review step. System checks run automatically, AI suggestions are shown with confidence states, and manual review is required only when rules cannot be satisfied automatically. The workflow moves from review required, to assisted validation, to verified state, ensuring partner rules and audit requirements are satisfied before the application advances.
Review required
The interface shows detected fields, rule checks, and exception requirements together. Passing checks remain visible but require no action, while failed rules move the document into manual review. Agents can override rules, apply an allowed exception, and add a required note before approval becomes available.
Detection and validation
Detected values are highlighted on the document and linked to rule checks in the validation panel. Confidence states and rule status are visible at the same time so the agent can accept the suggestion, edit the value, or apply an exception without leaving the review view.
Verified state
After all checks pass or are resolved, the document moves to a verified state. All validations show as passed or resolved, manual review is cleared, and the primary action advances the workflow to the next required document, preventing incomplete applications from progressing.
Production
Phase 1 - January 2026
Launched as a controlled evaluation under real operating conditions. Automation was restricted by reliability thresholds - documents could only progress automatically when both rule validation and extraction confidence requirements were satisfied. The evaluation confirmed that the checklist model reduced reliance on agent experience for partner-specific rules and that queue-based assignment improved throughput visibility for operations leads. It also surfaced edge cases in multi-document applications where rule dependencies between documents created sequencing gaps the checklist model did not yet handle, which informed the scoping of Phase 2 gating logic.
Phase 2
Introduced conditional automation, allowing documents to bypass manual review once extraction accuracy and validation logic consistently met partner requirements. The project manager and I reviewed the workflow and rule logic with product, engineering, and operations to confirm that approval states matched actual review behavior before enabling automation at scale.
Decisions & Trade-offs
Automation vs. reliability: Approvals occur only when checklist validation and confidence thresholds pass. A flagged approval is still an approval. In a regulated lending workflow, the cost of an incorrect decision outweighs the cost of a manual review step. Allowing low-confidence results to proceed with a flagging mechanism was considered and rejected on that basis.
Unified workflow state vs. cross-system sync: Consolidating into one workflow required migrating verification state that operations had managed in the separate tool for years. The alternative of syncing state between systems introduced a new failure mode where sync delays would recreate the same reconciliation problems the project was designed to eliminate.
Batch extraction vs. real-time processing: Real-time extraction was the initial plan. Batch processing was adopted after latency failures stalled agents mid-review in early testing. Batch added a pre-processing step but removed extraction latency from the critical path entirely.
Rule-driven checklists vs. per-partner fixed flows: A fixed flow per partner configuration was faster to build initially. Dynamic checklist generation from rule mappings was chosen because the partner network was growing. Fixed flows would have required new design and engineering work for every new partner condition.
Manual override vs. full automation: Override remains available when extracted values or rule checks do not meet reliability thresholds. Removing it entirely was considered for Phase 2, but operations leads needed the ability to approve exceptions for edge cases that rule logic could not fully anticipate.
Synthesis
The hardest part of this project was not designing the interface. It was establishing the conditions under which automation could be trusted. That meant mapping the existing workflow precisely enough to know where rule logic diverged from how agents actually applied it in edge cases, validating each check against documented requirements rather than observed behavior, and designing staged review so that automation expanded only as the system earned it.
In regulated workflows, AI reliability is not just a model problem. It is a system design problem. Extraction accuracy matters, but so does whether the rules are correct, whether workflow state is consistent, and whether agents have enough visibility to catch what automation misses. The architecture has to be trustworthy before the automation can be. That principle shaped every decision in this project, and it is the one I would carry into any AI-assisted workflow operating under compliance constraints.
Next Project
Measured Impact
Enabled verified applications to progress without manual interventionConfidence thresholds and rule validation gated conditional automation, reducing review overhead for clean applications.
Eliminated cross-system reconciliation delaysUnified workflow state replaced manual comparison across two tools with independent review records.
Reduced inconsistent verification outcomesPartner and document rules embedded into a generated checklist removed agent reliance on recalled logic.
Lowered processing cost and prevented review interruptionsBatch extraction before queue load eliminated repeated service requests and decoupled review from extraction latency.