Humans in the Loop: Advancing Metaliteracy for Generative AI Learning Environments #ECIL2025

Sheila here, liveblogging a panel session Humans in the Loop: Advancing Metaliteracy for Generative AI Learning Environments with material created by Thomas P. Mackey, Brenda Van Wyk, Megan Eberhardt-Alstot, Kristine N. Stewart, Kristen Schuster and Matt Moyo. Mackey, Stewart and Moyo were there to present. [added 8 October - the slides are here]
Mackey introduced it with images created by Eberhardt-Alstot showing that pre-GenAI programmers were directing AI behaviors with the lay user having a more passive experience (e.g. auto correction on text). Post-genAI blurs the line between expert and non expert, so you have non-experts with natural language agency with the AI. Factors include peer acceptance, faculty and institutional policy, learning integrity (do you value learning and the learning process), cognitive understanding, awareness of expert Humans in The Loop (HITL) and self-exploration.
Mackey went on to talk about metaliteracy. I've blogged about his metaliteracy work numerous times and the website is here https://metaliteracy.org/ Mackey presented the metaliteracy framework highlighting various aspects including metcognition, affect, and the social / collaborative aspects. The associated learner roles are producer, publisher, researcher, participant, communicator, translator (not just of language), author, teacher, collaborator. Then there are the learning domains: affective, behavioural, cognitive, metacognitive and finally the learner characteristics: adaptable, open, productive, collaborative, informed, reflective, participatory, civic minded. Hopefully, then, someone will end up with a metaliteracy mindset. Mackey identified that the latest iteration of learning objectives included AI.
Mackey presented some selected results from Eberhardt-Alstot's mixed-methods PhD study, which identified the persona of the AI learner (who values learning and works with AI as a tool to support that goal) and the AI user (using AI for efficiency, using it to replace human cognitive effort). This struck me as being similar to the approaches to learning research (the deep and surface learner). Eberhardt-Alstot has also connected the characteristics to related metaliteracy characteristics.

Moyo presented on AI and postgraduate research, work with Van Wyck. Spervisors need to be able to "help students critically and ethically engage with information, knowledge production, and now also AI technologies". From that point of view, supervisors should become the "human in the loop". In other words they will be be critically ebvaluating, validating and refining: this aligns with a metaliterate approach. AI Literacy, Metaliteracy and Research literacy overlap in a synergistic fashion.  Mishra (2025) was cited - the was the false confidence trap, dual expertise challenge, novice's dilemma, expert's advantage. The four elements can be on both the supervisor and student's side in relation both to subject domain knowledge and knowledge of AI.
Moyo identified benefits of extending the Human in The Loop definition to include use of AI in research and supervision e.g. bias mitigation, improved trust in findings. The challenge was seen in the supervisor being able to keep up, and that the research landscape (and its bureaucracy) can be slow to adapt. In conclusion, students may feel tempted to take shortcuts, which can be countered by supervisor support. 

Stewart then talked about culturally sustaining pedagogy (work with Schuster). A key element is inclusive curriculum design. Schuster has taught cataloguing and data mangement to mostly international students, and she encourages students to "play with language" "using stories that poke fun at my own English language skills" e.g. practical differences in American and British usage. Thinking about ambiguity and interpretation is useful when creating metadata. Underpinning this is a desire to recognise diversity and promote accessibility. This is connected to AI as you can think of AI in different ways, AI depends on language, and also it can be used to explore and critique linguistic bias.
The next aspect that Stewart addressed was representation in AI. Stewart give some information about her context, working in a federal university in the UAE (United Arab Emirates), who are predominantly female, Emirati and ESL (English as Second Language) students. Stewart had been asked to develop training for ethical use of AI in research. As students didn't seem very engaged, she asked them to generate an image "of this class" with only 4 attempts allowed, and Stewart showed an example of 4 images, which increasingly looked like the class (though never perfectly). This exercise naturally raised discussion about issues such as gender bias (the first image was men who all looked the same). It helped the students to think about how they are represented in AI. This highlights learners' responsibilities in critical evaluation of AI output and experience in the development of knowledge.
Stewart talked about  how this work connects with metaliteracy goals, for example developing ethical roles as producers and consumers. Also Stewart talked about development of an AI assistant for research (called Aish) with students contributing to Aisha's development. Again this involvement can help them think about their identity and relationship with AI.

Mackey then returned to talk about a Digital Media Arts programme. This has given opportunity to discuss many AI-related issues, and also to collaborate in production. One example was a discussion of the work Theatre d'opera Spatial (an AI work that was submitted to, and won, an art competition) with questions such as - is it art? what are ethical concerns? what is at stake for creativity? Another activity consists of creating a digital object and then doing a written reflection (on tools used, ethical issues, what they learned about the role of AI in creativity). Digital Storytelling is another area where they can explore issues around use of AI. Also Mackey explained that at the end of the course students have to reflect on metaliteracy characteristics (e.g. which one best describes their growth on the course)
Mackey finished by identifying some strategies to empower users with AI including supporting ethical production of AI-generated content, creating scenarios to reflect on metaliterate roles.
Photo by Sheila Webber: statue of ETA Hoffman, featuring (I imagine) Kater Murr, Bamberg, September 2025