Conversational Agents in Education – A Systematic Literature Review

Conversational Agents in Education – A Systematic Literature Review Khosrawi-Rad Rinn Schlimbach Gebbing Yang Lattemann Markgraf Robra-Bissantz

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Conversational Agents in Education – A Systematic Literature Review 2022 ECIS 2022 Research Papers khosrawi-rad2022-conversational-agents-in-education khosrawi-rad2022-conversational-agents-in-education.txt, the project's staged copy 03c1cd3dd6fb98981edddeecc3cc6efe7b341b86 open-access information-systems

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them critique Quotation withheld because this record is not cleared for redistribution. pedagogical conversational agents in education Quotation withheld because this record is not cleared for redistribution. systematic-review systematic-review Quotation withheld because this record is not cleared for redistribution. mechanisms Research projects design pedagogical CAs for short-run demonstration rather than sustained use, so the agents are not maintained past the prototype and fail soon after deployment. Quotation withheld because this record is not cleared for redistribution. mechanisms Designers draw on only isolated fragments of the psychological/learning knowledge base, so the resulting agents realise only part of their potential. Quotation withheld because this record is not cleared for redistribution. Selection and analysis rested on the authors' subjective decisions, mitigated but not removed by peer-reviewed screening and two coding waves. what this limits: every inclusion and coding decision: INESCAPABLE LIMITATIONS REGARDING SUBJECTIVELY DRIVEN DECISIONS DURING THE SLR, reduced by peer review rather than removed Quotation withheld because this record is not cleared for redistribution. Relevant publications may have been missed during screening despite the breadth of databases and search terms. what this limits: the completeness of the survey: IT IS STILL POSSIBLE THAT RELEVANT PUBLICATIONS MIGHT HAVE BEEN OVERLOOKED DURING THE SCREENING PROCESS Quotation withheld because this record is not cleared for redistribution. The review captures only a fast-moving field's current state; later searches may give different results. what this limits: the shelf life of the findings: AS THE FIELD IS EVOLVING QUICKLY, WE CAN ONLY PROVIDE THE CURRENT STATE OF THE FIELD AND FUTURE SEARCHES ... MIGHT YIELD DIFFERENT RESULTS Quotation withheld because this record is not cleared for redistribution. The overview of the field excludes every publication not in English or German, so the reported state of the art is bounded by the reviewers' languages. what this limits: this review as a holistic overview of conversational agents in education: publications in any language other than English or German were excluded by the stated criteria Quotation withheld because this record is not cleared for redistribution. The review declares its own selection subjective, and states that this is normal for the method rather than particular to this team. what this limits: the coverage of any systematic literature review including this one: the authors state the selection and analysis process rests on subjective decisions, as is common for SLRs Quotation withheld because this record is not cleared for redistribution. The claim about the field's maturity is an inference from how often software names recur, not a measurement of the systems. what this limits: the claim that pedagogical conversational agents remain prototypes: it is an assumption drawn from software names being referenced only once and not developed in later publications Quotation withheld because this record is not cleared for redistribution. The review's own object is defined by publication facts as much as by content: papers in any language other than English or German are excluded, as is everything published before 2016. A reader takes the resulting picture as the state of the field, and the record already carries that reading as a sense. Quotation withheld because this record is not cleared for redistribution. 186 publications from before 2016 were excluded, leaving 386 for analysis eliminates: that the review's picture of the field is dominated by work predating the technology it is about · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. exclusion criteria exclusion criteria Quotation withheld because this record is not cleared for redistribution. pedagogical CAs pedagogical CAs Quotation withheld because this record is not cleared for redistribution. systematic literature systematic literature Quotation withheld because this record is not cleared for redistribution. prototype development prototype development Quotation withheld because this record is not cleared for redistribution. state of the art in pedagogical conversational agents - AS VISIBLE THROUGH THIS REVIEW: what seven databases return, filtered to English and German, and screened by subjective decisions Quotation withheld because this record is not cleared for redistribution. maturity of a pedagogical conversational agent - inferred from BIBLIOGRAPHIC RECURRENCE: a system named once and not developed further is taken as evidence of a prototype stage Quotation withheld because this record is not cleared for redistribution. research method - in this field mostly PROTOTYPE DEVELOPMENT (95 of 252 papers), evaluated mainly quantitatively; so 'method' here names what was BUILT more often than what was tested Quotation withheld because this record is not cleared for redistribution. builds-on → Webster and Watson (2002), Page et al. (2021), and Schoormann et al. (2021) Quotation withheld because this record is not cleared for redistribution. extends → Gubareva and Lopes (2020) role taxonomy for pedagogical conversational agents Quotation withheld because this record is not cleared for redistribution. replicates → Strohmann (2021) and Diederich et al. (2022) Quotation withheld because this record is not cleared for redistribution. contradicts → cross-domain conversational-agent literature reviews (Diederich et al., 2022) Quotation withheld because this record is not cleared for redistribution. Only five of the 252 coded articles reported long-term studies of conversational agents in education. Quotation withheld because this record is not cleared for redistribution. Pedagogical CAs act most often as tutors (31%), then as motivators and organizers (21% each), mentors (18%) and moderators (9%). Quotation withheld because this record is not cleared for redistribution. Academic students are the target group in 78% of the studies, high school students in 6%. Quotation withheld because this record is not cleared for redistribution. "Chatbot" appears in 167 of the 252 articles (85%); "virtual companion(ship)" in only 2. Quotation withheld because this record is not cleared for redistribution. 136 publications reference psychological or learning theories, and 80% of the theories mentioned come from educational research. Quotation withheld because this record is not cleared for redistribution. Google Dialogflow (37%) and IBM Watson Assistant (30%) are the two building platforms most often used to implement pedagogical CAs. Quotation withheld because this record is not cleared for redistribution. Most of the reviewed studies evaluate their agents on small samples, so their results may not be representative. Quotation withheld because this record is not cleared for redistribution. Exactly one of the 252 papers offers prescriptive design knowledge in the form of a nascent design theory, and it is confined to argumentation skills. Quotation withheld because this record is not cleared for redistribution. Just 5% of the articles address professional development outside academia. Quotation withheld because this record is not cleared for redistribution. A large share of the reviewed authors design their conversational agent without grounding it in any scientific theory. Quotation withheld because this record is not cleared for redistribution. conversational agent conversational agent Quotation withheld because this record is not cleared for redistribution. pedagogical conversational agent pedagogical conversational agent Quotation withheld because this record is not cleared for redistribution. virtual companion virtual companion Quotation withheld because this record is not cleared for redistribution. social cues social cues Quotation withheld because this record is not cleared for redistribution. computers as social actors computers as social actors Quotation withheld because this record is not cleared for redistribution. 252 — number of articles fully coded in the review after screening Quotation withheld because this record is not cleared for redistribution. 3891 — total records returned by the database search before screening Quotation withheld because this record is not cleared for redistribution. 95 — papers whose research method is prototype development Quotation withheld because this record is not cleared for redistribution. 82 — studies that include an experimental setting Quotation withheld because this record is not cleared for redistribution. 18 — papers that explicitly follow a user-centred design approach Quotation withheld because this record is not cleared for redistribution. five — studies out of 252 that were long-term studies Quotation withheld because this record is not cleared for redistribution. 78% — share of studies whose target group is academic students Quotation withheld because this record is not cleared for redistribution. 80% — share of the cited theories that originate in educational research Quotation withheld because this record is not cleared for redistribution. 37% — share of implementations using Google Dialogflow as building platform Quotation withheld because this record is not cleared for redistribution. 31% — share of pedagogical CAs occupying the tutor role Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at .