Consent
Customers should understand when conversations or account information are analysed to infer support needs. Where consent is relied upon, refusing or withdrawing it should not become a barrier to obtaining appropriate assistance.
Consumer & child protection IWIB4AI session Needs review
AI & Vulnerable Customers in Financial Services
Using AI to identify and support vulnerable financial customers without undermining privacy, fairness or human judgment.
Draft – legal and case law references not yet verified
Summary
Financial services and insurance firms can use predictive analytics and real-time conversation analysis to identify customers who may need additional support. These tools could help staff recognise difficulties and adapt assistance, but may also infer sensitive circumstances, misclassify customers or enable discriminatory treatment. The central challenge is ensuring that identification leads to appropriate support rather than intrusive profiling or disadvantage.
The Problem
Identifying vulnerability may require analysing information that customers do not expect firms to use or infer. Firms must distinguish supportive intervention from profiling that restricts access, changes pricing or exploits a customer's circumstances, while preserving meaningful choice and human review.
Ethical Grounds
Fairness
Vulnerability indicators may misinterpret language, disability or communication style. Firms should test for unequal error rates and prevent support-related classifications from becoming grounds for unjustified exclusion or worse terms.
Transparency
Customers should receive accessible explanations of how AI helps identify support needs and how resulting flags are used. Staff should understand that a prediction is an uncertain indicator, not a definitive assessment of the customer.
Human oversight
Trained staff should assess flagged needs with the customer rather than automatically acting on a vulnerability label. Customers need a practical way to correct inaccurate inferences and obtain human assistance.
Legal Compliance
| Framework | Provision | Relevance |
|---|---|---|
| GDPR | Arts. 5, 6 and 9 | Vulnerability profiling requires a lawful basis, purpose limitation, data minimisation and accuracy safeguards, with an additional Article 9 condition where processing involves special-category data such as health information. |
| GDPR | Arts. 13–14 | Firms must provide applicable information about the processing, including its purposes and legal basis, and additional information about automated decision-making where required. |
| GDPR | Art. 22 | If vulnerability profiling feeds solely automated decisions with legal or similarly significant effects, Article 22 restricts that processing and requires applicable exceptions and safeguards rather than merely nominal human involvement. |
| EU AI Act | Art. 6 and Annex III, points 5(b)–(c) | Customer-support profiling is not automatically high-risk, but systems used to evaluate natural persons' creditworthiness or for risk assessment and pricing in life and health insurance may fall within the Act's high-risk classification rules. |
| EU consumer law | Directive 2005/29/EC, Art. 5(3) | Commercial practices foreseeably affecting clearly identifiable groups particularly vulnerable because of mental or physical infirmity, age or credulity must be assessed from the perspective of the average member of that group. |
Sources
Court & Regulator Decisions
CK v Dun & Bradstreet Austria (opens in a new tab)
Customers affected by automated financial decisions need understandable explanations of the profiling procedure, notwithstanding firms’ claims of commercial secrecy.
OQ v Land Hessen (SCHUFA) (opens in a new tab)
Vulnerability scores that determine credit outcomes may trigger automated-decision safeguards even when a separate institution formally makes the final decision.
UI v Österreichische Post (opens in a new tab)
The dispute over inferred political affinities illustrates potential compensation claims when unlawful vulnerability profiling causes demonstrable non-material harm.
Key Takeaways
- Define support purposes clearly and prevent vulnerability flags from being repurposed for exploitation or unjustified disadvantage.
- Explain AI analysis accessibly and establish lawful grounds for both source data and sensitive inferences.
- Test identification tools for missed needs, false flags and unequal outcomes across customer groups.
- Give trained staff authority to override outputs and customers an accessible route to correction and human support.
Related Use Cases
From the Session
AI & Vulnerable Customers in Financial Services