Measuring AI Literacy of Secondary Students

By /

I keep arguing on this blog that AI literacy has to be more than a vocabulary list. Students can recite what a neural network is and still have no idea how to question an AI output, refuse it, or use it with intent. That gap between knowing and thinking is exactly what Zhong and Liu (2025) set out to measure in their paper on AI literacy of secondary students.

Their proposal is the KAT framework: AI Knowledge, AI Affectivity, AI Thinking. Three dimensions, grounded in Piaget’s genetic epistemology and Bloom’s taxonomy, operationalized into a 48-item AI Literacy Scale validated with Rasch modeling, exploratory factor analysis, and confirmatory factor analysis on 1,392 Chinese secondary students. It’s one of the most psychometrically careful attempts at this I’ve seen, and it deserves a serious reading.

What KAT Actually Covers

Zhong and Liu argue that most existing AI literacy frameworks overweight the cognitive side. Students learn what AI is, how it works, maybe a bit about machine learning, and that gets counted as literacy. The authors push against that by treating affect and higher-order thinking as equal pillars.

Measuring AI Literacy of Secondary Students

Their AI Knowledge dimension covers foundational concepts, applications, and ethical awareness. AI Affectivity pulls in emotional responses, attitudes, willingness to engage, and perceived self-efficacy around AI tools. AI Thinking is the ceiling piece: computational thinking, critical evaluation of AI outputs, creative problem-solving with AI, and what the authors call “algorithmic thinking,” which is the capacity to reason about how a system arrives at a decision.

The Piaget and Bloom Grounding

What sets this paper apart from the crowded AI literacy scale space is the theoretical scaffolding. Zhong and Liu build KAT on Piaget’s stages of cognitive development and map the dimensions onto Bloom’s revised taxonomy. AI Knowledge corresponds to remember and understand. AI Affectivity pulls from the affective domain that Bloom’s collaborators developed separately. AI Thinking climbs into analyze, evaluate, and create.

I find this grounding useful because most AI literacy scales read like a checklist someone assembled after reading five EU policy documents. The KAT framework has a developmental logic. A 12-year-old and a 17-year-old are not the same cognitive animal, and a scale that treats them as interchangeable will produce noisy results. Zhong and Liu take that seriously, and the paper includes a developmental curriculum map stretching from kindergarten through high school, which is ambitious in a field that rarely thinks past the semester it’s piloting.

This matters because we already have thoughtful frameworks at the early-childhood end, like the work Su, Ng and Chu (2023) did on AI in early childhood, and frameworks at the university end, like the competency framework from Chee, Ahn and Lee (2025). The secondary gap has been underspecified. Zhong and Liu move us closer to closing it.

The single strongest argument in the paper is that AI Thinking should be the apex of the construct. The authors write that critical thinking, algorithmic reasoning, and creative application are “essential for navigating an AI-integrated society.” I agree with that framing and I’ve argued the same point in posts on cognitive surrender (Shaw and Nave, 2026) and on what Kalantzis and Cope (2025) describe as literacy becoming a design practice rather than a decoding practice.

If students can only describe AI, they can’t interrogate it. The whole educational project collapses into a slightly more advanced version of “here’s how the internet works.” Zhong and Liu’s instrument explicitly asks students to evaluate outputs, reason about system behavior, and generate solutions with AI assistance. That’s closer to what literacy has always meant in other domains: judgment.

Where I Push Back: The Gender Finding Needs More Care

One result in the paper deserves careful reading. Zhong and Liu report that boys scored higher on AI Knowledge while girls scored higher on AI Affectivity. The authors note this as a finding worth flagging but don’t push the interpretation very far. I’d push further.

Gender gaps in technology self-concept are old news. The concern is not that the gap exists in this sample. The concern is that an instrument designed to measure literacy may be partially measuring prior exposure and cultural positioning. Chinese secondary boys may have had more informal tinkering time with AI tools, more peer-group encouragement, and more identity-level permission to claim technical confidence. None of that is “literacy” in the sense Zhong and Liu want to measure. It’s background variance leaking into the construct.

The paper would be stronger if it treated the gender result as a validity question rather than a descriptive one. What does it mean that a 48-item scale produces this gap? Is the Knowledge subscale picking up familiarity with specific tools that boys encounter more often? If so, the instrument needs item-level review, not just aggregate reporting.

I find KAT comprehensive on cognitive and affective fronts and thinner on a dimension I’d argue is non-negotiable now: AI as a site of power. The framework treats ethics as a component of Knowledge, which reduces ethics to something students “know about” rather than something they practice. Roe, Furze and Perkins (2025) argue that AI literacy has to include a critical stance toward the material conditions of the technology: who owns it, who trains it, whose labor annotated the data, whose language gets under-represented in outputs. Zhong and Liu’s instrument doesn’t really reach that layer.

This is not a fatal gap. A scale can only do so much, and adding a critical-political factor would require a separate validation effort. But readers of the paper should not assume KAT captures the full construct. It captures a serious subset.

Reading This in 2026

The paper was published in 2025 and the data was collected in late 2023 through late 2024, which in AI time is already a different era. Generative AI tools have become routine in classrooms, and the items on any literacy scale will age quickly. Zhong and Liu acknowledge this and they frame AILS as a starting instrument that will need periodic re-anchoring.

My concern is that instrument development always lags the technology by two to three years. By the time a scale is fully validated, the world it measured is gone. This isn’t Zhong and Liu’s fault. It’s a structural problem for the whole field. One response is to publish more modular frameworks that can swap items without re-validating the full scale. Another is to prioritize frameworks like the one UNESCO (2024) released for students, which are more principles-based and less tied to specific tools.

What I’d Take Back to a Classroom

Two practical things. First, the KAT structure is worth borrowing even if you don’t adopt the scale. Design AI lessons that hit all three dimensions rather than the first two. Students who can describe AI and feel comfortable with it but can’t evaluate its outputs are only two-thirds literate. Second, use the developmental arc Zhong and Liu sketch. A ninth grader and a twelfth grader should not be doing the same AI assignment. The paper gives us a map for thinking about that progression, and maps are rare in this space.

AI literacy is still a construct under construction. Zhong and Liu give us a careful, theory-grounded, empirically tested draft. The next move is for teachers and researchers to push it into real classrooms and see what breaks.

References

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top