Responsible AI practice IWIB4AI session Needs review
AI companions and emotion-responsive interfaces test whether consent remains meaningful when systems subtly influence users’ choices.
Draft – legal and case law references not yet verified
Emotional AI includes companions, affective chatbots and interfaces that adapt to users’ apparent feelings. These systems raise questions about whether users can meaningfully consent when personalised interactions also shape their preferences and decisions. The use case examines how GDPR consent requirements and EU AI Act safeguards address—and leave gaps around—subtle emotional influence.
Agreeing to personal-data processing does not necessarily mean understanding or accepting a system’s persuasive influence. The core tension is how to distinguish supportive personalisation from manipulation, particularly when emotional attachment or vulnerability weakens users’ ability to refuse or disengage.
Emotion-responsive interactions should support users’ own goals rather than covertly redirect decisions toward commercial or other external objectives. Users need practical ways to disable personalisation or leave without emotional pressure.
Consent should distinguish between operating a chatbot, analysing emotional signals and using those signals to personalise influence. Emotional attachment must not become a mechanism for securing or retaining agreement.
Users should understand that they are interacting with AI, what emotional signals it processes and why its responses adapt. Disclosure should not imply that inferred emotions are reliable facts or that simulated empathy represents genuine feelings.
Design and evaluation should address foreseeable risks such as emotional dependency, distress and exploitation of vulnerability. Safeguards should account for how repeated interactions can create harms that are not apparent in a single exchange.
| Framework | Provision | Relevance |
|---|---|---|
| GDPR | Arts. 4(11) and 7 | Where consent is the lawful basis for processing personal data, it must be freely given, specific, informed and unambiguous, and withdrawal must be as easy as giving consent. |
| GDPR | Arts. 5(1)(a)–(c), 12 and 13 | Processing personal data to infer or respond to emotions must meet fairness, transparency, purpose-limitation and data-minimisation requirements, including clear information about processing purposes. |
| EU AI Act | Art. 5(1)(a)–(b) | Emotional influence may fall within prohibitions on manipulative or deceptive techniques or exploitation of specified vulnerabilities when the statutory conditions, including significant-harm thresholds, are met; emotional persuasion is not categorically prohibited. |
| EU AI Act | Arts. 3(39) and 50(1), (3) | Article 50 sets disclosure duties for direct AI interaction and qualifying emotion-recognition systems, but the latter category requires biometric-data-based inference and does not automatically encompass text-only sentiment analysis. |
Garante per la protezione dei dati personali – Replika (Luka Inc.) (opens in a new tab)
The intervention directly addresses emotional companion chatbots, highlighting safeguards for minors and emotionally vulnerable users beyond merely obtaining agreement to data processing.
Power Coffee Paris – Emotional AI, Consent and Autonomy
