I find it useful to occasionally look back at where the conversation about AI in education began. Before ChatGPT, before the panic about academic integrity, before every conference had an AI track, researchers were already studying how chatbots work in educational settings. Okonkwo and Ade-Ibijola (2021) published a systematic review of 53 peer-reviewed studies from 2015 to May 2021, and reading it now, in early 2026, offers a fascinating window into both how much has changed and how much hasn’t.
Their paper, “Chatbots Applications in Education: A Systematic Review,” published in Computers and Education: Artificial Intelligence, maps the field across five application areas, identifies consistent benefits and recurring challenges, and outlines where future research should focus. What strikes me most is how many of the problems they flagged in 2021 remain unresolved today, and how many of the benefits they described have only intensified with the arrival of generative AI.
What Did the Systematic Review Examine?
Okonkwo and Ade-Ibijola reviewed 53 peer-reviewed studies published between 2015 and May 2021. They examined how chatbots were being used across five areas: teaching and learning, administration, assessment, student advising, and research and development. About two-thirds of the studies focused on teaching and learning.
The review found that educational chatbots could provide immediate responses, make information easier to access, support personalized learning, and serve many students at the same time. It also identified recurring concerns involving privacy, transparency, trust, supervision, maintenance, user attitudes, and the limited ways chatbot effectiveness was being evaluated.
These studies examined an earlier generation of chatbots. Most were rule-based or retrieval-based systems with far less conversational ability than current generative AI tools. The review still gives us a useful baseline. Many of the educational benefits have expanded since ChatGPT arrived, and many of the unresolved problems have grown with them.
Teaching and Learning Dominated Even Then
The review found that roughly two-thirds of chatbot studies focused on teaching and learning. Chatbots delivered course content, answered student questions, guided practice, and provided immediate feedback. Some systems adapted to individual learner responses and simulated tutoring interactions. Others supported specific skills like programming or writing. As the authors report: “The majority of Chatbot system applications in Education are focused on teaching and learning, administration, assessment, advisory, and research and development” (p. 8).
It feels familiar. Even now, with tools like ChatGPT, Claude, and Gemini widely available, the primary conversation about AI in education still centers on instruction. How do students use it to learn? How do teachers use it to teach? The focus has scaled up dramatically, but the fundamental questions haven’t shifted as much as we might think.
What has changed is the sophistication of the interaction. The chatbots in this review were largely rule-based or retrieval-based systems with limited conversational ability. Today’s large language models can hold extended conversations, generate ‘original’ content, and respond to nuanced prompts. The gap between those early systems and what students have access to now is enormous, and it has implications for every finding in this review.
The Benefits Were Real, and They’ve Only Grown
Okonkwo and Ade-Ibijola found consistent benefits across the literature. Chatbots centralized course materials, provided instant responses, handled multiple users at once, and improved engagement, particularly on mobile devices. The authors note:
the usage of Chatbots allows for the gathering of various forms of in-formation and storage in a unit (Information unit) for rapid and easy access by authorised users. Furthermore, Chatbots encourage personalised learning, provide instant support to users, and allow multiple users to access the same information at the same time. (p. 8)
Every one of those benefits is amplified in the current generation of AI tools. A student in 2021 could ask a chatbot a pre-programmed question and get a scripted answer. A student in 2026 can ask ChatGPT to explain a concept three different ways, generate practice problems, and give feedback on a draft, all in one conversation. The accessibility advantage that early chatbots offered has expanded into something qualitatively different.
But I think this is also where we need to be careful. Accessibility and engagement are good things. They’re also insufficient on their own. Guo et al. (2025) found in their year-long classroom study that the pedagogical design around ChatGPT determined whether students actually learned from it. Students valued AI as a supplement but rejected it as a replacement for human instruction. The tool’s availability didn’t guarantee learning. The task design did. And the lesson applies directly to the chatbot literature Okonkwo and Ade-Ibijola reviewed: availability is a starting point, not an outcome.
The Challenges They Identified Haven’t Gone Away
The most valuable part of this review, from my perspective, is its treatment of challenges. The authors identified ethical concerns (privacy, transparency, trust, agent persona), evaluation gaps (small samples, limited testing), user attitudes that shape adoption, and technical limitations in natural language processing. They write:
It is clear from the re-view that some factors, such as ethical, evaluation, user attitude, supervision, and maintenance issues, may have an impact on the adoption and use of Chatbots in education. This implies that these factors may skew users’ perceptions, limiting the applications of Chatbots systems in educational settings. To improve the penetration of Chatbot technology in education, researchers and stakeholders must define adequate solutions that can mitigate the negative effects of these challenges. (p. 8)
Four years later, every one of those challenges persists, and several have intensified. Privacy concerns are more urgent now that students share personal writing, emotional reflections, and academic work with commercial AI systems. Transparency matters more when the AI can produce convincing but inaccurate text. Trust is more complicated when tools feel authoritative but lack understanding.
Shaw and Nave (2026) identified cognitive surrender, the tendency for students to defer to AI outputs without critical evaluation, as a growing risk. Cognitive surrender wasn’t visible in the pre-ChatGPT literature, but the seeds were there in the trust and user attitude challenges this review documented.
Evaluation remains a weak spot too. Gerlich (2025) showed that heavy AI reliance weakens critical thinking through cognitive offloading. Fan et al. (2025) found that AI improved essay quality but not knowledge transfer, because students skipped the metacognitive work. These findings echo what Okonkwo and Ade-Ibijola flagged about uneven evaluation practices. We’re still not measuring the right things consistently enough.
What This Means for Teachers Using AI Chatbots
Reading this review now, I am struck by how familiar its findings feel. The technology has changed dramatically, but educators are still dealing with many of the same pedagogical questions. Immediate answers and easy access can support learning. They can also encourage students to move through a task without doing enough of the thinking themselves.
For teachers, the first question should concern the purpose of the chatbot interaction. A chatbot used to generate practice questions serves a different purpose from one used to explain a difficult concept or provide feedback on a draft. Defining that purpose helps determine what the tool should do, what the student should do, and where teacher guidance is still needed.
Before introducing an AI chatbot, teachers can ask:
- What specific learning outcome will this interaction support?
- What thinking must students complete before consulting the chatbot?
- How will students check the accuracy of its responses?
- Does the activity protect students’ personal and academic information?
- How will I determine whether students learned from the interaction?
- Where will students still receive feedback and guidance from a person?
The final question is especially important. Students may value the speed and availability of AI while still needing human explanation, encouragement, and judgment. An AI chatbot can provide examples, questions, feedback, and additional practice. The teacher remains responsible for deciding how those functions fit within a larger learning experience.
This review also reminds us that engagement is an incomplete measure of success. Students may enjoy using a chatbot and produce stronger-looking work without developing stronger understanding. We need to examine what students retain, whether they can transfer their learning, and how the tool affects their capacity to think independently.
For me, the lesson from these 53 studies is clear: access to an AI chatbot creates an opportunity. The quality of the learning still depends on the activity, the guidance students receive, and the intellectual work they are expected to complete.
References
- Celik, I. (2023). Towards Intelligent-TPACK: An empirical study on teachers’ professional knowledge to ethically integrate artificial intelligence (AI)-based tools into education. Computers in Human Behavior, 138, 107468. https://doi.org/10.1016/j.chb.2022.107468
- Guo, F., Li, T., & Cunningham, C. J. L. (2025). One year in the classroom with ChatGPT: Empirical insights and transformative impacts. Frontiers in Education, 10, 1574477. https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2025.1574477/full
- Mishra, P., Warr, M., & Islam, R. (2023). TPACK in the age of ChatGPT and generative AI. Journal of Digital Learning in Teacher Education, 39(4), 235–251. https://doi.org/10.1080/21532974.2023.2247480
- Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033
- Shaw, S. D., & Nave, G. (2026). Thinking fast, slow, and artificial: How AI is reshaping human reasoning and the rise of cognitive surrender. Working paper, The Wharton School, University of Pennsylvania. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6097646
