Explain at Your Own Risk: Security and Privacy Challenges of Explainable AI in the Cloud

Led By: Omran Ayoub

Cloud-based Machine Learning as a Service (MLaaS) has made powerful models accessible through simple APIs, and regulatory and societal pressure increasingly requires these services to explain their decisions. Explainability is widely seen as a cornerstone of trustworthy AI, but it comes with a hidden cost. Every explanation reveals something about the model and the data it was trained on. In this talk, I will show how explanations exposed by MLaaS platforms can be exploited by adversaries to extract models and infer whether specific individuals were part of the training data, turning transparency into an attack surface. Drawing on our recent research, I will discuss how these risks can be measured, how explanations leak information, and to what extent defences can preserve useful explanations while limiting privacy exposure. I will conclude by reflecting on what this means for the design of trustworthy AI systems, from regulated applications to emerging agentic AI that increasingly relies on models offered as services and outline open challenges for building AI that is explainable, private, and secure at the same time.