AI in education — privacy risks, algorithmic bias and safeguarding
Adaptive learning platforms, automated assessment and tutoring AI — where minor data privacy and surveillance concerns are most acute.
AGC
AI Generated, Human Reviewed
AI in education covers adaptive learning platforms, AI tutoring systems, automated essay marking, plagiarism detection tools, and learning management systems with behavioural analytics. These tools are increasingly embedded in school and university contexts — often without meaningful consent from students or parents.
The governance concerns are acute because educational AI systems typically involve minors, collect detailed behavioural data, and influence high-stakes outcomes — grades, progression decisions, university admissions. Algorithmic bias in assessment has been documented; the A-levels algorithm controversy in the UK (2020) is the most widely cited example of automated assessment producing demonstrably unfair outcomes at scale.
Sensitive learning data on children collected at scale. Behavioural analytics, attention tracking, and learning pattern data are collected by ed-tech platforms with limited parental visibility or meaningful consent.
Automated grading encoding historical inequalities. Models trained on historical assessment data can reproduce and entrench existing inequalities — disadvantaging students from lower-income or minority backgrounds.
Monitoring children as a default. Proctoring software, engagement tracking, and behavioural analytics normalise continuous surveillance in educational contexts, with long-term privacy implications.
Gamification designed to maximise time-on-platform. Ed-tech platforms using engagement optimisation mechanics may prioritise platform usage over actual learning outcomes.
EU AI Act classification: High-risk. AI systems used in education and vocational training — including those determining access, evaluating learning outcomes, or assessing students — are explicitly high-risk under Annex III. These systems require conformity assessments, human oversight, transparency to affected students, and audit trails. Special protections apply for systems involving minors.
QUESTIONS
What is AI in education?
AI in education refers to systems that personalise learning, automate assessment, detect plagiarism, provide tutoring, or analyse student behaviour. Examples include Khanmigo, Turnitin’s AI detection, and adaptive learning platforms like Duolingo and Carnegie Learning.
Why is AI in education high-risk under the EU AI Act?
Educational AI systems make or inform high-stakes decisions about students — including access to courses, assessment outcomes, and progression. Because these decisions affect life opportunities, the EU AI Act classifies them as high-risk, requiring human oversight and transparency.
What are the privacy risks of AI in education?
Ed-tech platforms collect detailed behavioural data — keystrokes, attention patterns, learning speed — often on minors. Consent mechanisms are frequently inadequate. Data may be retained, sold to third parties, or used for purposes beyond the original educational context.