Project Description
The collaborative research project AGORA-KI investigates how social AI agents are transforming automated content moderation on digital communication platforms. These systems can evaluate content, make decisions about its visibility or removal, and interact directly with users. Public communication is therefore increasingly shaped by AI systems that influence which content remains visible, which forms of expression are considered appropriate, and how communication rules are enforced.
AGORA-KI is part of the BMFTR funding measure “INSIGHT II – Interdisciplinary Perspectives on Disruptive Innovations” and its thematic area “The New Generation of Artificial Intelligence – Implications for Social Interaction, Art, and Culture.” Projects in this area examine the multidimensional impacts and interactions of AI and their implications for possible futures.
Automated Content Moderation and Democratic Oversight
Generative AI systems produce and amplify texts, images, and videos at a scale that makes comprehensive human review increasingly difficult. At the same time, agentic systems automatically evaluate, downrank, flag, restrict, or remove this content. As a result, this dual automation shifts decision-making power over visibility and communication norms toward algorithmic systems embedded in platform infrastructures.
Authorities and regulatory bodies, however, continue to rely largely on human review, case-by-case assessment, and transparent justification. This asymmetry therefore raises questions about responsibility, accountability, trust, and democratic control.
An Interdisciplinary Approach
AGORA-KI brings together empirical research, participatory reflection, legal and policy analysis, and technical expertise. The project examines public discourses and moderation practices while incorporating the perspectives of platform actors, civil society, regulatory authorities, and people affected by online hate, discrimination, or wrongful content removal.
In particular, the project considers the experiences of marginalized communities and tensions between protection from harm, freedom of expression, non-discrimination, procedural fairness, and transparency.
From Research to Governance
The EU AI Act and the Digital Services Act establish new requirements for transparency, accountability, and human oversight. Against this background, AGORA-KI develops empirically grounded and verifiable governance criteria that connect these requirements to the practice of automated content moderation. The resulting instruments are intended to support policymakers, regulatory authorities, civil society organizations, platform providers, and other actors involved in governing digital communication.
Project Phases
Phase 1 – Conceptual Foundations and Context Analysis
Theoretical framework, discourse analysis, and regulatory mapping
Phase 2 – Empirical Research and Participatory Evaluation
Interviews, participatory research, and foresight
Phase 3 – Governance Development, Auditing, and Transfer
Governance integration, practical testing, and policy transfer
Project Members
- Prof. Dr. Philipp Kellmeyer, University of Mannheim
- Patricia Gleim, University of Mannheim
- Dr. Dana Mahr, Karlsruhe Institute of Technology, Institute for Technology Assessment and Systems Analysis (ITAS)
- Nora Weinberger, Karlsruhe Institute of Technology, ITAS
- Lahari Yaddanapudi, Karlsruhe Institute of Technology, ITAS
- Jérémy Lefint, Karlsruhe Institute of Technology, ITAS
- Dr. Matthieu Binder, Think Tank iRights.Lab GmbH, Berlin
- Yannick Haan, Think Tank iRights.Lab GmbH, Berlin
- Lukas Glaser, Think Tank iRights.Lab GmbH, Berlin
- Jonas Navid M. Al-Nemri, PENEMUE Technologies GmbH, Freiburg
- Sara Egetemeyr, PENEMUE Technologies GmbH, Freiburg
- Lino Zurmühl, PENEMUE Technologies GmbH, Freiburg
Advisory Board & Associated Partners
- HateAid gGmbH
- Landesanstalt für Kommunikation Baden-Württemberg (LFK)
- Kathrin Morasch, LfK
- Yannic Frey, LfK
- Landeskriminalamt Baden-Württemberg, Stabsbereich Zentrum für Sicherheitsforschung
