phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | pedagogical conversational agents in education— “There is a recent trend of increasing publications and implementations of CAs in education.” | |
| claim-type | critique— “Overall, we detected a deficiency of solidly validated studies of CAs.” | |
| method — evidence | — “To contribute to future research, we conducted an SLR based on Webster and Watson (2002), Page et al. (2021), and Schoormann et al. (2021).” | |
| mechanism | Research projects design pedagogical CAs for short-run demonstration rather than sustained use, so the agents are not maintained past the prototype and fail soon after deployment.— “Since Janssen et al. (2021) proved that most CAs nevertheless fail within a short period, we reckon that this might particularly go back to the rare long-term orientation of the research projects outlined above.” | mechanisms[0] |
| mechanism | Designers draw on only isolated fragments of the psychological/learning knowledge base, so the resulting agents realise only part of their potential.— “articles that are based upon psychological concepts and theories, however, usually only consider isolated aspects (e.g., CAs to promote improved multimedia learning), while a more diverse consideration of the knowledge base was hard to find. For this reason, we assume that existing CAs do not yet realise their full potential.” | mechanisms[1] |
| builds-on | Webster and Watson (2002), Page et al. (2021), and Schoormann et al. (2021)— “To contribute to future research, we conducted an SLR based on Webster and Watson (2002), Page et al. (2021), and Schoormann et al. (2021).” | claim_relations[0] |
| extends | Gubareva and Lopes (2020) role taxonomy for pedagogical conversational agents— “Supplementary, we added the moderator function, because Gubareva and Lopes (2020) focused on academic student support and we covered a broader scope, also including the instructors’ perspective (e.g., Mikic-Fonte et al., 2019; Sakr et al., 2021).” | claim_relations[1] |
| replicates | Strohmann (2021) and Diederich et al. (2022)— “Similar to Strohmann (2021) and Diederich et al. (2022), we found that scenarios of the longterm orientation of CAs have so far been rarely considered, which is reflected in the lack of long-term studies (Diederich et al. 2022).” | claim_relations[2] |
| contradicts | cross-domain conversational-agent literature reviews (Diederich et al., 2022)— “However, contrary to what cross-domain literature reviews reveal, there is a difference in the quality of existing studies: While for CAs in general, a plethora of empirical studies with validated constructs exist (Diederich et al., 2022), those are underrepresented in the educational domain - many publications lack evaluations with a solid… | claim_relations[3] |
| sense | state of the art in pedagogical conversational agents - AS VISIBLE THROUGH THIS REVIEW: what seven databases return, filtered to English and German, and screened by subjective decisions— “The exclusion was guided by pre-defined criteria, to ensure the fit with our RQ and the topicality of the contributions examined.” | senses[0] |
| sense | maturity of a pedagogical conversational agent - inferred from BIBLIOGRAPHIC RECURRENCE: a system named once and not developed further is taken as evidence of a prototype stage— “although a large proportion of these software names are only referenced once, and have mostly not yet been further developed in subsequent publications.” | senses[1] |
| sense | research method - in this field mostly PROTOTYPE DEVELOPMENT (95 of 252 papers), evaluated mainly quantitatively; so 'method' here names what was BUILT more often than what was tested— “Concerning the applied research methods, 95 papers use prototype development ( e.g., Sjöström and Dahlin, 2020; Al Muid et al., 2021). For the evaluation of the developed artefacts, researchers use mainly a quantitative approach (61 articles), and less frequently” | senses[2] |
| control | 186 publications from before 2016 were excluded, leaving 386 for analysis — held — rules out that the review's picture of the field is dominated by work predating the technology it is about— “For that reason, we excluded 186 publications published before 2016 and proceeded with the remaining 386 articles for further analysis.” | controls[0] |
| outcome (stated by the tagger, no quotation) | control — held | controls[0] |
| for which use | limitation — every inclusion and coding decision: INESCAPABLE LIMITATIONS REGARDING SUBJECTIVELY DRIVEN DECISIONS DURING THE SLR, reduced by peer review rather than removed— “In spite of inescapable limitations regarding subjectively driven decisions during the SLR, we purposefully undertook measures to reduce these through a peer-reviewed process in screening, two successive coding waves in an interdisciplinary research team, after querying seven scientific databases.” | limitations[0] |
| for which use | limitation — the completeness of the survey: IT IS STILL POSSIBLE THAT RELEVANT PUBLICATIONS MIGHT HAVE BEEN OVERLOOKED DURING THE SCREENING PROCESS— “Second, despite our extensive review of 252 publications, it is still possible that relevant publications might have been overlooked during the screening process, even though we included a variety of databases and search terms.” | limitations[1] |
| for which use | limitation — the shelf life of the findings: AS THE FIELD IS EVOLVING QUICKLY, WE CAN ONLY PROVIDE THE CURRENT STATE OF THE FIELD AND FUTURE SEARCHES ... MIGHT YIELD DIFFERENT RESULTS— “Ultimately, as the field is evolving quickly, we can only provide the current state of the field and future searches on related topics might yield different results.” | limitations[2] |
| for which use | limitation — this review as a holistic overview of conversational agents in education: publications in any language other than English or German were excluded by the stated criteria— “The following exclusion criteria were applied: missing educational context, no relation to CAs, another language than English or German” | limitations[3] |
| for which use | limitation — the coverage of any systematic literature review including this one: the authors state the selection and analysis process rests on subjective decisions, as is common for SLRs— “First, as is common for SLRs, the selection and analysis process is based on subjective decisions of the authors” | limitations[4] |
| for which use | limitation — the claim that pedagogical conversational agents remain prototypes: it is an assumption drawn from software names being referenced only once and not developed in later publications— “although a large proportion of these software names are only referenced once, and have mostly not yet been further developed in subsequent publications. Thus, we assume that many pedagogical CAs are still in a prototypical stage” | limitations[5] |
| replication | untested | |
| limitation | Selection and analysis rested on the authors' subjective decisions, mitigated but not removed by peer-reviewed screening and two coding waves.— “In spite of inescapable limitations regarding subjectively driven decisions during the SLR, we purposefully undertook measures to reduce these through a peer-reviewed process in screening, two successive coding waves in an interdisciplinary research team, after querying seven scientific databases.” | limitations[0] |
| limitation | Relevant publications may have been missed during screening despite the breadth of databases and search terms.— “Second, despite our extensive review of 252 publications, it is still possible that relevant publications might have been overlooked during the screening process, even though we included a variety of databases and search terms.” | limitations[1] |
| limitation | The review captures only a fast-moving field's current state; later searches may give different results.— “Ultimately, as the field is evolving quickly, we can only provide the current state of the field and future searches on related topics might yield different results.” | limitations[2] |
| limitation | The overview of the field excludes every publication not in English or German, so the reported state of the art is bounded by the reviewers' languages.— “The following exclusion criteria were applied: missing educational context, no relation to CAs, another language than English or German” | limitations[3] |
| limitation | The review declares its own selection subjective, and states that this is normal for the method rather than particular to this team.— “First, as is common for SLRs, the selection and analysis process is based on subjective decisions of the authors” | limitations[4] |
| limitation | The claim about the field's maturity is an inference from how often software names recur, not a measurement of the systems.— “although a large proportion of these software names are only referenced once, and have mostly not yet been further developed in subsequent publications. Thus, we assume that many pedagogical CAs are still in a prototypical stage” | limitations[5] |
| limitation | The review's own object is defined by publication facts as much as by content: papers in any language other than English or German are excluded, as is everything published before 2016. A reader takes the resulting picture as the state of the field, and the record already carries that reading as a sense.— “The following exclusion criteria were applied: missing educational context, no relation to CAs, another language than English or German, duplicates, language learning or health focus, as well as… | limitations[6] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | Only five of the 252 coded articles reported long-term studies of conversational agents in education. | content_tags.findings[0] |
| finding | Pedagogical CAs act most often as tutors (31%), then as motivators and organizers (21% each), mentors (18%) and moderators (9%). | content_tags.findings[1] |
| finding | Academic students are the target group in 78% of the studies, high school students in 6%. | content_tags.findings[2] |
| finding | "Chatbot" appears in 167 of the 252 articles (85%); "virtual companion(ship)" in only 2. | content_tags.findings[3] |
| finding | 136 publications reference psychological or learning theories, and 80% of the theories mentioned come from educational research. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[4] |
| finding | Google Dialogflow (37%) and IBM Watson Assistant (30%) are the two building platforms most often used to implement pedagogical CAs. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[5] |
| finding | Most of the reviewed studies evaluate their agents on small samples, so their results may not be representative. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[6] |
| finding | Exactly one of the 252 papers offers prescriptive design knowledge in the form of a nascent design theory, and it is confined to argumentation skills. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[7] |
| finding | Just 5% of the articles address professional development outside academia. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[8] |
| finding | A large share of the reviewed authors design their conversational agent without grounding it in any scientific theory. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[9] |
| quantity | 252 — number of articles fully coded in the review after screening Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[0] |
| quantity | 3891 — total records returned by the database search before screening Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[1] |
| quantity | 95 — papers whose research method is prototype development Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[2] |
| quantity | 82 — studies that include an experimental setting Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[3] |
| quantity | 18 — papers that explicitly follow a user-centred design approach Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[4] |
| quantity | five — studies out of 252 that were long-term studies Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[5] |
| quantity | 78% — share of studies whose target group is academic students Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[6] |
| quantity | 80% — share of the cited theories that originate in educational research Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[7] |
| quantity | 37% — share of implementations using Google Dialogflow as building platform Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[8] |
| quantity | 31% — share of pedagogical CAs occupying the tutor role Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[9] |
| definition | conversational agent Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[0] |
| definition | pedagogical conversational agent Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[1] |
| definition | virtual companion Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[2] |
| definition | social cues Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[3] |
| definition | computers as social actors Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[4] |
| concept | exclusion criteria Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[0] |
| concept | pedagogical CAs Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[1] |
| concept | systematic literature Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[2] |
| concept | prototype development Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[3] |
When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.
Tag layer © the Meta-Tagging Project, CC BY 4.0. The paper text is © its authors/publisher
and is linked, not reproduced. Excerpts shown are short quotations for identification and verification.
· back to index
· concepts · authors
· עברית
© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824