Generative AI in language assessment is moving quickly. The evidence behind it is much less settled.
Ping-Lin Chuang and Xun Yan examined this gap through a systematic review of 77 publications on the use of generative AI in language testing and assessment. Their review covers automated scoring, feedback, test-item generation, classroom assessment, and academic integrity. It also asks whether these applications meet the established demands of assessment: reliability, validity, fairness, and practicality.
The authors followed PRISMA guidelines and searched databases and selected journals up to October 28, 2024. Of the 77 publications, 63 were empirical studies and 14 were conceptual papers. The empirical research was divided fairly evenly among qualitative, quantitative, and mixed-methods designs.
At first glance, this looks like a substantial evidence base. Its distribution tells a more complicated story.
University settings accounted for 56 percent of the reviewed studies, and 60 percent examined classroom-based assessment. Only 9 percent involved K–12 education. Local testing programs accounted for 6 percent of the publications, while large-scale testing represented just 4 percent.
Writing dominated the research at 61 percent. ChatGPT appeared in 71 percent of the studies. Nearly half of the research was conducted in Asia.
These numbers show where experimentation is happening. They also reveal how much remains unknown about other language skills, younger learners, local placement testing, and high-stakes assessment.
AI can score and generate, but quality control remains human
Twenty-four studies examined automated scoring and feedback. Generative AI could reduce the time teachers spend evaluating writing and provide students with faster, more individualized responses.
However, the quality of the results depended heavily on preparation. AI systems needed contextual information, clear rubrics, examples from each performance level, and repeated calibration. Researchers also recommended dividing rating into several rounds so that reliability could be checked.
Even under these conditions, human raters generally produced more accurate and consistent scores.
This finding fits with the broader question of how teachers should recognize the value and limitations of AI-generated feedback in education. Speed is useful. A rapid response still needs to be checked for relevance, accuracy, and suitability for a particular learner.
Eight studies examined AI-generated test items. AI could produce many questions quickly and reduce development costs. Yet useful items required detailed prompting, expert review, pilot testing, and, in formal testing contexts, psychometric analysis. Generating an item is easy. Establishing that it measures the intended ability fairly is the difficult part.
One finding that caught my attention is that only two publications addressed assessment planning. Forty-eight focused on validation and use, while 22 examined administration and scoring.
This imbalance matters. Decisions about what should be assessed and how the target ability is defined come before item generation or automated scoring. If the construct is unclear, technological efficiency can simply make an unclear assessment operate faster.
This is why assessment validity matters more than catching AI use. Before asking whether an AI system can score a response, we need to ask what that response is meant to demonstrate.
The three assessment contexts are very different
For me, one of the strongest contributions of this review is its separation of large-scale, local, and classroom-based assessment.
Large testing organizations can develop proprietary systems, train specialized models, and build extensive quality-control procedures. Classroom teachers have more freedom to experiment with inexpensive public tools in low-stakes activities.
Local testing programs sit in a difficult middle position. They may need secure and standardized assessments but lack the funding, data, and technical expertise available to large organizations. Their tests must also reflect particular curricula and learner populations. Reliance on commercial AI systems could weaken that local alignment and expose student data to outside platforms.
The review therefore raises an equity issue that deserves more attention: AI may widen the resource gap between well-funded testing companies and smaller institutional programs.
Academic integrity presents another problem. The reviewed research found that AI detectors became less reliable when texts combined human and AI-generated writing. The authors advise educators against making integrity decisions solely from detector scores. That recommendation is consistent with the growing evidence on why AI detection in education is unreliable.
The review has limitations. It includes some non-peer-reviewed work, which helped capture an emerging field but introduced differences in research quality. Proprietary studies from testing companies may be missing. The search ended in 2024, and the technology has already changed considerably. The review maps the field well, though its findings cannot tell us whether AI improves language learning over time.
The clearest message is one of restraint. Generative AI can support feedback, scoring, and item development. We still need stronger evidence that these systems produce valid, fair, and sustainable language assessments across different learners and contexts.
Reference
Chuang, P.-L., & Yan, X. (2025). Language assessment in the era of generative artificial intelligence: Opportunities, challenges, and future directions. System, 134, Article 103846. https://doi.org/10.1016/j.system.2025.103846
