The third-party AI evidence pack for financial services

The Bank of England and FCA's 2024 survey found that 75% of responding financial-services firms already used AI and one-third of reported use cases were third-party implementations. Forty-six percent described only a partial understanding of the AI technologies they used, with third-party models a major reason for the gap.

A conventional SaaS review remains necessary, but it may not establish how an AI system was evaluated, where its outputs are unreliable, which upstream models and datasets it depends on, how behavior changes, or when a person can intervene. Those questions matter more as AI enters customer, risk, compliance, and operational decisions.

Use the table as a starting request, then deepen the review according to customer impact, decision authority, data sensitivity, operational criticality, and substitutability. Ask for dated artifacts tied to the proposed product version and deployment, not a broad description of the vendor's general practice.

The same vendor can require different assurance for different uses. Summarizing internal policy has a different failure impact from recommending a fraud response, influencing credit, communicating with a vulnerable customer, or operating a critical control.

Classify the proposed use before reviewing the supplier. Consider the scale and reversibility of harm, customer and market impact, data sensitivity, degree of automation, ability to detect errors, operational criticality, and dependence on the service. Document who approved the classification.

Four of the five highest current AI risks in the Bank/FCA survey were data-related: privacy and protection, quality, security, and bias or representativeness. Trace customer and firm data through prompts, logs, retrieval stores, support systems, model providers, analytics, and subprocessors. Establish whether any party can use it for training or service improvement.

Map model, cloud, data, and tool dependencies. The survey found concentration among named providers and respondents expected critical third-party dependency to show the largest increase in systemic risk. The firm needs notice of material dependency changes and a tested response when a provider, model, region, or commercial term changes.

A vendor benchmark is not automatically representative of the firm's customers, data, language, products, channels, or controls. Agree the evaluation population, critical failure types, thresholds, and monitoring before approval. Where testing requires firm data, establish an appropriately controlled evaluation environment.

Human oversight should identify what reviewers see, what they can change, how much time they have, and when the system must escalate. Measure overrides, disagreements, complaints, and downstream outcomes. A nominal reviewer who routinely accepts an opaque output does not provide meaningful control.

Approval should identify which changes require notice, reassessment, or consent: foundation model changes, new subprocessors, different data use, material capability additions, new regions, revised safety controls, or changed service limits. Contractual rights matter only if named owners monitor and exercise them.

Maintain a current system owner, vendor owner, risk classification, evidence set, issue log, review date, and exit plan. Track performance and incidents against agreed thresholds. When evidence is unavailable, record the uncertainty and decide whether compensating controls reduce it sufficiently; do not convert absence of evidence into an assumed pass.