AI technical due diligence
AI due diligence starts with the data
How to test data lineage, governance, and deployment risk before funding, acquiring, or launching an AI system.
By Kesho Partners
11 minute read
AI due diligence should trace every material claim back through five things: the data source, the right to use it, the transformations applied, the evidence produced, and the controls operating in production. Then test who owns each decision and what happens when data, models, or suppliers change. If that chain cannot be reconstructed, the system is not yet ready for investment, acquisition, or consequential deployment.
Start with the claim the data must support
Do not begin with a generic request for the data room. Write down the commercial and technical claims that must be true. The depth of review should follow those claims.
If the investment case depends on proprietary data, prove exclusivity and performance contribution. If the deployment case depends on accuracy, prove that evaluation data represents the real operating context. If the product depends on a supplier model, prove what evidence and controls remain available when that supplier changes.
| Claim | Evidence required | What can break the case |
|---|---|---|
| The data is proprietary | Origin, contracts, collection rights, permitted uses, renewal and deletion terms | The same data is widely available or cannot lawfully support the intended use |
| The system performs better | Representative evaluation set, baseline, error analysis and production results | Selected demos, test leakage or a benchmark unlike the deployment setting |
| Performance improves with use | Feedback capture, review, labelling, versioning and measured learning effect | Customer activity is collected but does not improve the system |
| The product is deployable | Architecture, security tests, telemetry, limits, rollback and incident evidence | The result depends on manual work or controls that exist only in a pilot |
Follow every material dataset from source to decision
A dataset inventory is not lineage. Lineage connects origin, rights, preparation, versions, evaluation, model or retrieval use, production outputs, retention, and deletion. It should let an independent reviewer reproduce which data and system version produced a material result.
NIST asks organizations to document intended purpose, deployment context, data suitability, test sets, limitations, and production monitoring. For high-risk AI systems, EU AI Act Article 10 goes further by specifying controls over data origin, collection purpose, preparation, assumptions, bias, gaps, representativeness, and deployment context.
| Lineage stage | Evidence to inspect | Material warning |
|---|---|---|
| Source and rights | Source register, dates, licences, consent, contracts, collection purpose and territory | 'Publicly available' is treated as permission |
| Preparation | Cleaning, filtering, deduplication, labels, exclusions, synthetic data and transformation code | Critical preparation exists only in notebooks or individual knowledge |
| Evaluation | Train, validation and test separation; versioned cases; subgroup and error analysis | Test leakage, unexplained exclusions or aggregate accuracy only |
| System use | Model, prompt, retrieval index, feature and policy versions linked to results | Performance cannot be tied to a reproducible configuration |
| Production | Input and output logs, user context, overrides, feedback, incidents and drift history | The deployed system is materially different from the evaluated system |
| Retention and exit | Deletion propagation, legal hold, customer export, supplier deletion and decommissioning process | The company cannot remove data or reconstruct where it travelled |
Provenance can show origin and recorded change. It does not by itself prove that data is accurate, representative, lawful, or fit for the intended purpose.
Governance must leave a decision trail
Policies are weak evidence unless they produce named ownership, recorded approvals, risk acceptance, change control, and incident decisions. The reviewer should be able to identify who may approve a new use, accept a known limitation, change a dataset, release a model, or stop the system.
The ICO places accountability with senior management and expects data flows, sources, outcomes, human involvement, alternatives, errors, and residual risk to be examined. EU AI Act Article 17 requires documented quality-management processes for providers of high-risk systems. Those legal duties are scope-specific, but the operating evidence is useful in any consequential review.
- Accountability
- A named executive owns the use case, while data, model, security, product, and operational responsibilities are explicit.
- Use approval
- The organization records intended use, prohibited use, affected people, jurisdictions, human review, and the non-AI alternative.
- Data control
- Owners approve sources, access, quality thresholds, retention, sharing, material transformations, and deletion.
- Supplier control
- Contracts allocate evidence, notification, audit, security, model-change, deletion, portability, and incident responsibilities.
- Risk acceptance
- Known limits and residual risks have an accountable approver, review date, monitoring condition, and escalation path.
NIST AI RMF and its Generative AI Profile are voluntary frameworks, not certifications. EU AI Act Articles 10, 12, 15, and 17 apply to defined high-risk systems and roles, not universally to every AI application.
Deployment changes the risk
A benchmark describes performance under test conditions. Deployment adds live users, uneven inputs, changing data, external tools, access rights, model updates, supplier failures, and irreversible actions. Diligence must therefore test the system around the model, not only the model score.
NIST calls for performance evidence under conditions similar to deployment and continuing production monitoring. EU AI Act Articles 12 and 15 require logging and lifecycle accuracy, robustness, and cybersecurity for covered high-risk systems. Joint government guidance recommends artifact versioning, adversarial testing, drift monitoring, incident response, rollback, and reevaluation after updates.
Rollback evidence must also account for state the new version has already written. The p99log engineering note Ship Fast. Recover Faster. provides a practical treatment of bounded rollouts, stop conditions, production telemetry, and state-compatible recovery.
| Control | Evidence to inspect | Failure signal |
|---|---|---|
| Release | Representative pre-release evaluation, security testing, approval and bounded rollout | The first production release carries the full user or transaction load |
| Observability | Input, output, quality, latency, cost, override, error and subgroup telemetry | A dashboard reports uptime but not whether decisions remain acceptable |
| Drift and change | Thresholds, model and data versions, supplier notices, retraining triggers and regression results | Models or datasets can change without a new decision |
| Containment | Permission limits, human approval, rate limits, isolation, kill switch and tested rollback | Stopping the system requires a new code release or supplier action |
| Recovery | Incident exercises, state reconciliation, customer communication and recovery objectives | Rollback restores code but not data, actions, or downstream state |
| Exit | Alternative supplier, portability test, deletion evidence and decommissioning plan | A critical dependency cannot be inspected, replaced, or exited |
Run the review in five passes
A useful review moves from management claims to evidence and then to direct challenge. Each finding should state the business consequence, not stop at a technical observation.
| Pass | Work | Output |
|---|---|---|
| 1. Scope | Identify the three to five data and deployment claims supporting the decision | Material claims and review depth |
| 2. Trace | Map sources, rights, transformations, versions, suppliers, evaluations and production flows | End-to-end lineage and evidence gaps |
| 3. Verify | Sample contracts, records, data, code, evaluation results, logs and approvals | Confirmed facts separated from assertions |
| 4. Challenge | Test edge cases, security, drift response, supplier change, rollback and deletion | Observed control effectiveness and failure limits |
| 5. Decide | Connect each material gap to value, timing, liability, delivery cost and deal or launch terms | Proceed, condition, reprice, stage or stop |
Turn evidence gaps into decision terms
Do not convert missing evidence into a medium score because no failure has been reported. Evidence absent is risk unpriced.
- Proceed
- Lineage is reproducible, governance decisions are owned, and deployment controls have operated under relevant conditions.
- Proceed with conditions
- The case remains credible, but defined rights, controls, tests, hires, or remediation must be completed before funding, closing, or launch.
- Reprice or stage
- Data access, supplier dependence, production work, or control gaps materially change cost, timing, margin, or risk.
- Stop
- A foundational data right, performance claim, accountability requirement, or failure boundary cannot be established or controlled.
This is a technical and operating diligence framework, not legal advice. Applicable data protection, AI, intellectual-property, contract, and sector rules require qualified counsel in the relevant jurisdictions.
Related service
AI and data technical due diligence
Kesho tests the data, architecture, AI evidence, governance, security, deployment controls, economics, and engineering capability behind an investment or enterprise decision.
Explore the service