Beyond the Hype: Towards a Critical Debate About AI Chatbots in Swedish Higher Education

AI chatbots are computationally trained on what language looks like, not what it means

Focusing on the materiality of the language models on which AI chatbots are built on, we note that “LMs [language models] are trained only on linguistic form and don’t have access to meaning (Bender & Koller, 2020, p. 5185). More specifically, they are “trained on string prediction tasks: that is, predicting the likelihood of a token (character, word or string) given either its preceding context or (in bidirectional and masked language models) its surrounding context” (Bender et al., 2021, p. 611). When deployed, such systems are unsupervised and take a text as input, commonly outputting scores or string predictions (Bender et al., 2021). In this sense, the generated outputs are not based on a model capable of understanding words, characters, or meaning. Instead, the model operates statistically, independently of the semantics involved, at the level of the linguistic form of the human language. In other words, it focuses on what the words look like and the probability of them occurring in relation to each other instead of what they mean for someone.

Moreover, while AI chatbots generate text, they are not additive, creating original content; instead, they re-produce existing text based on the training data (Saunders, 2023).

In this context, some questions of relevance relate to: If AI chatbots can generate syntactically coherent human-like text, what are the implications for how teachers engage with teaching and student assessment practices? What are the consequences for student learning processes?

Such questions require higher education institutions to discuss the difference between “linguistic form” and “meaning” (Bender & Koller, 2020) openly and to put effort into designing program activities that cultivate students’ criticality and ability to discern nonsense text from knowledge in the AI era.

AI chatbots compute responses based on text available on the Internet

Following the material constitution of AI chatbots and how it may impact student construction of meaning, we ask where the data used to feed these large language models comes from and whether they are reliable. AI chatbot developers use a corpus of text data from the Internet to train their language model (Bender et al., 2021). While OpenAI, for example, states the company uses information that is publicly available on the Internet, information that they license from third parties, and information that their users or human trainers provide (OpenAI, 2024), the company has not disclosed the exact sources of this data for giving their model a broad understanding of language. As generated responses to users’ prompts are based on the corpus available to the large language model (Bender et al., 2021), the bias already existing in the corpus used to train the model will result in bias in responses given by AI chatbots. On this note, Noble (2018), who studied the problem of relying on the information accessed on the Internet by search engines like Google, has a point when she argues that “information organization is a matter of sociopolitical and historical processes that serve particular interests” (p. 138) and that “search does not merely present pages, but structures knowledge, and the results retrieved in a commercial engine creates their particular material reality” (p. 148). Considering this observation, we ask how students’ knowledge will be configured by interacting with AI chatbots like ChatGPT.

Relevant questions for higher education institutions and university teachers include: How deeply are students engaging with the material that matters when they use AI chatbots? Will the potential bias in the text-generated responses become an incentive to boost the development of academic argumentative skills or result in reproducing biases? How can higher education institutions inform students on how to use tools that are opaque with their sources when source criticism is so quintessential to the higher education experience?

AI chatbots are developed for purposes other than education

AI chatbots are not only an example of a recent technological development or the result of the operation of computational techniques and machine learning. Broadly, they are general purpose tools not developed for use in education contexts specifically. They are also products designed and developed by people, that is, groups of developers, designers, programmers, data, and computer scientists who are led by private companies’ commercial interests which may have great ambitions for technological and scientific progress but a limited understanding of the implications of their developments for critical sectors in society such as higher education. On this note, Gilliard and Rorabaugh (2023) observe that developers of chatbots may perhaps not primarily be “concerned with educators very much. It has come, disrupted, and walked away with no afterthought about what schools should do with the program. Now is a time for educators to understand how, if at all, such innovations fit in the education sector” (Gilliard & Rorabaugh, 2023, p. 1).

These observations echo Rahm’s (2019) critical piece on the need to change teaching practices every time a new digital technology emerges in society. Accordingly, the education sector is allocated the task of providing citizens’ development of competencies, in this case, AI literacy, to indirectly address, among others, the needs and ambitions of international companies to make their artifacts available. In this context, what task should university programs assume every time a novel technology becomes available on the market?

As discussed earlier in our work (Farazouli et al., 2023), AI chatbots have a potentially unsettling impact on how teachers view students’ work, throwing into sharp relief the potential breakdown in trust between university teachers and students. On this note, universities must engage in dialogue with students and colleagues about what kind of authentic problems AI chatbots solve in their teaching context. Bergviken-Rensfeldt and Rahm (2023) “explain that while AI may promise to replace ‘routine and tedious’ educational work with automation, research demonstrates that automation may paradoxically lead to the opposite result: an increased workload for humans as well as a restructuring of how the work is performed” (p. 117). Here, we need to consider important questions, such as: What problems do AI chatbots solve in the context of higher education practice? Why are AI chatbots viewed as the “next thing” that universities must embrace? What would happen if universities critically researched these technologies before adopting and introducing them into everyday practice?

AI chatbots redefine current understandings of student writing competence

The phenomena of copy-pasting a response generated by an AI chatbot may be problematic not only from an academic integrity point of view but also concerning students’ development and mobilization of various literacies and competencies. What do students learn when they ask AI chatbots to answer questions given in assignments? Especially when students may, at times, be more concerned about passing exams than meeting aimed learning objectives, there is a risk that chatbots will be used for performing through getting good grades rather than developing literacies, abilities, and competence (Byrnes, 2023). In this regard, it is not farfetched to ask if student-AI chatbot mediation will increase the gap between the students already writing critically and feeling self-confident in writing their responses and those struggling with writing and falling behind. While one group may be able to distinguish between passing an assignment and learning from an assignment, there is a risk that others might not be able to consider such a distinction. Amid performance pressures, AI chatbots may seem attractive despite the risk of lost literacy development opportunities. On this note, Lindgren (2023b) observes that while technology development has historically developed to liberate us from physical work, large language models and associated AI chatbots propose to “liberate” us from the creative tasks demanding idea creation, planning, argumentation, and other critical literary activities.

We ask educators how AI chatbots might change our understanding of student writing competence standards. What do we gain in higher education with novel interpretations of literacy? What do we lose? What kind of tasks will AI chatbots do now and in the future, and what are the implications for human creativity and knowledge? Will engaging with text generated by chatbots generate a depth and passion for learning?

AI chatbots produce disembodied text without a point of view

While spectacular, AI chatbot’s responses come from standard data science practices that most often attempt to “see everything from nowhere,” giving the impression of providing universal truths that are pure and objective (Haraway, 1988, p. 581). In this sense, understanding the voices and positionality involved in especially contested societal topics for students becomes difficult as responses provided by the chatbot are not grounded in human experience and existence. As such, the response cannot belong to a specific geographical place or specific felt time or enact a subjective “partial perspective” (Haraway, 1988) because computational linguistic techniques know nothing about belonging, feelings, or experience. As neither the content nor the structure is designed by a particular someone, the question is: What competencies do students need to develop to be able to argue with AI chatbots and develop critical literacy?

If accessing information today is about forming and developing one’s own critical point of view to act on the complex dilemmas affecting our societies, how are AI chatbots transforming students’ strategies to think critically and construct meaning? (Sporrong & Tikkanen, 2016).

Here, we identify several core questions, including: Who answers to whom in the text generated by AI chatbots? What does it entail to read information from unknown sources? How can students distinguish critical voices and identify where discourses on pressing social issues have emerged? How could the interests, ideologies, and values behind AI chatbots’ answers be deconstructed as part of a student’s learning journey?

AI chatbots may be developed through unethical data science practices

Practices in higher education are fundamentally relational and ethical concerning the very nature of engagement between teachers and students. Teachers should introduce subjects truthfully, treat students respectfully, and examine without prejudice (SULF, 2019).

Previous work suggests that university teachers want to engage ethically with AI (McGrath et al., 2023) and that learning technologies, some may include AI, could be deployed with equity ambitions in mind and be used by groups of students in the margins of the higher education system (Cerratto Pargman et al., 2023). However, we note that AI tools and systems may have been developed unethically (Bender et al., 2021) in terms of the resources they use (Luccioni et al., 2023), the data they are sourced on (Bender et al., 2021), and their apparent contempt for intellectual property (Grynbaum & Mac, 2023), and this gives cause for concern.

AI chatbots come with a massive environmental cost (Li et al., 2023). In particular, Strubell et al. (2019) show that neural network models for natural language processing “are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware” (p. 1). Based on the Strubell’s et al., (2019), evidence, Bender et al., (2021) emphasize the “environmental cost” of such models by arguing that “increasing the environmental and financial costs of these models doubly punishes marginalized communities that are least likely to benefit from the progress achieved by large language models and most likely to be harmed by negative environmental consequences of its resource consumption” (p. 610).

Furthermore, part of the training and fine-tuning involved in developing AI chatbots requires human intervention in the form of reinforcement learning from human feedback, where humans assess the degree of accuracy or the desired fairness in AI chatbot output. The model adjusts its output to human responses and learns over time (Frey & Osborne, 2023). This involves engaging humans to shift through the material to remove violent, sexist, and racist material.

As mentioned, AI chatbots do not inform us where they source their data. This has led to several ongoing cases of litigation where publishing houses, creative professionals, and authors are suing OpenAI and other companies involved in developing large language model-based AI chatbots over the theft of intellectual material (Creamer, 2023).

Against this background, we ask the reader to consider: What does it mean to learn at such a high cost to the environment? What does it mean to use tools that were developed where humans must siphon through material that is violent, racist, and discriminatory? And how can university teachers, in good faith, ask students to engage with material that has been potentially unlawfully and unethically sourced?