Privacy
Agents should access only the personal data needed for an authorised task. Privacy protections must cover connected tools, external providers, memory and activity logs, not just the agent's conversational interface.
Responsible AI practice IWIB4AI session Needs review
Power Coffee – Building Trustworthy AI Agents
Building public-sector AI agents that protect personal data, preserve institutional control and keep consequential decisions under meaningful human oversight.
Draft – legal and case law references not yet verified
Summary
This use case concerns designing and deploying AI agents for public-sector tasks while retaining control over systems, data and decisions. Agents that act with greater autonomy need clear limits on their permissions, access to personal data and authority to make consequential decisions. Trustworthiness depends on privacy-first architecture and human oversight that can genuinely intervene, rather than simply approve automated outputs.
The Problem
How can public bodies benefit from agents that act independently without surrendering accountability or exposing personal data? The central tension is between operational autonomy and effective institutional control, particularly when an agent's actions may affect people's rights or access to services.
Ethical Grounds
Human oversight
Human reviewers need sufficient information, authority and time to challenge an agent's actions. Consequential actions should have clear approval requirements, escalation routes and mechanisms to stop execution.
Accountability
Responsibility must remain identifiable across the public body, system provider and connected services. Delegating tasks to an agent should not obscure who authorised an action or who must investigate and remedy errors.
Autonomy
People affected by public-sector agents should understand the agent's role and have meaningful routes to challenge consequential outcomes. Efficiency should not come at the expense of their ability to engage with a responsible human decision-maker.
Legal Compliance
| Framework | Provision | Relevance |
|---|---|---|
| GDPR | Arts. 5, 6 and 25 | Agents processing personal data require a lawful basis, purpose limitation, data minimisation and data protection by design and by default, including restrictions on tool access and retention. |
| GDPR | Art. 22 | Solely automated decisions producing legal or similarly significant effects are generally prohibited unless an applicable exception and required safeguards are satisfied; nominal human approval does not necessarily remove a decision from this rule. |
| GDPR | Arts. 44–49 | Where agent infrastructure or connected services involve transfers of personal data outside the EEA, the GDPR's international-transfer requirements apply; data sovereignty is not a substitute for a valid transfer mechanism. |
| EU AI Act | Arts. 14 and 26 | For agents classified as high-risk AI systems, the applicable provider and deployer obligations require effective human oversight, including assignment of oversight to people with appropriate competence, training and authority. |
| EU AI Act | Art. 27 | Public bodies deploying covered high-risk AI systems must assess their impact on fundamental rights before deployment, subject to the provision's scope and applicable implementation timetable. |
Sources
Court & Regulator Decisions
NJCM et al. v State of the Netherlands (SyRI) (opens in a new tab)
The welfare-fraud ruling shows why public-sector agents affecting service access require transparent, proportionate data use and effective safeguards against intrusive profiling.
Ligue des droits humains v Conseil des ministres (PNR) (opens in a new tab)
Limits on automated passenger assessment illustrate the need to constrain adaptive systems and provide meaningful human review of consequential public-sector outputs.
Data Protection Commissioner v Facebook Ireland and Schrems (Schrems II) (opens in a new tab)
Public bodies using overseas agent infrastructure must assess international transfer safeguards and surveillance access rather than assuming contractual arrangements adequately protect personal data.
Key Takeaways
- Define each agent's permitted tasks, data access and action limits before deployment.
- Map hosting, external tools and international data transfers; do not equate local hosting with full compliance.
- Require meaningful human approval for consequential actions and provide reliable stop and escalation mechanisms.
- Document responsibilities, assess applicable legal duties and maintain proportionate audit trails.
Related Use Cases
From the Session
Power Coffee – Building Trustworthy AI Agents
Speakers
