Building Bridges: AI Custom Chatbots as Mediators between Mathematics and Physics

Building Bridges: AI Custom Chatbots as Mediators between Mathematics and Physics Lademann Henze Becker-Genschow

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Building Bridges: AI Custom Chatbots as Mediators between Mathematics and Physics 2024 10.48550/arxiv.2412.15747 lademann2024-building-bridges-ai-custom-chatbot lademann2024-building-bridges-ai-custom-chatbot.txt, the project's staged copy 752a0241c7df5a6ba379ab5771a02afea5e20e94 open-access physics-education-research

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them existence Quotation withheld because this record is not cleared for redistribution. learning with AI-generated explanations of proportional relationships Quotation withheld because this record is not cleared for redistribution. behavioural-experiment behavioural-experiment Quotation withheld because this record is not cleared for redistribution. mechanisms The chatbot's encouraging, motivating tone raises situational interest relative to neutral textbook prose Quotation withheld because this record is not cleared for redistribution. mechanisms Accessible, conversational, human-like wording in the AI explanation lowers the processing demands of the material and so reduces cognitive load Quotation withheld because this record is not cleared for redistribution. mechanisms Encouraging tone and/or the lowered cognitive load raise students' confidence that they can solve the task themselves Quotation withheld because this record is not cleared for redistribution. mechanisms Because students never used the chatbot themselves, individual learning difficulties were never diagnosed or addressed, blunting any performance effect status: the paper's own words: It is possible that Quotation withheld because this record is not cleared for redistribution. Generalizability is limited by sample size, the snapshot implementation period, and the single topic and target group chosen. what this limits: the generalisation of every result: THE SIZE OF THE SAMPLE AND THE BRIEF IMPLEMENTATION PERIOD (SNAPSHOT) AS WELL AS THE NECESSITY OF SELECTING A PA[rticular topic and group] Quotation withheld because this record is not cleared for redistribution. Materials were delivered centrally, so students never interacted with the chatbot themselves and individual differences in use could not be captured. what this limits: what the intervention actually was: THE LEARNING MATERIALS WERE PROVIDED IN A CENTRALIZED MANNER, WHICH RESULTED IN THE LACK OF CONSIDERATION FOR INDIVIDUAL DIFFERENCES IN UTILIZATION - the students did not use the chatbot themselves Quotation withheld because this record is not cleared for redistribution. The performance data are too thin to support a conclusion about learning outcomes. what this limits: the learning-outcome question specifically: IT IS NOT POSSIBLE TO PROVIDE A DEFINITIVE ANSWER TO RQ5 BASED ON THE DATA COLLECTED IN THIS STUDY Quotation withheld because this record is not cleared for redistribution. The primary outcome did not move, and the conclusion says so plainly while the abstract calls the same result indefinite. what this limits: the claim that AI-generated materials improve learning: the conclusion states no notable differences were identified in the performance test and that the influence remains unclear Quotation withheld because this record is not cleared for redistribution. Generalisability is constrained by three named factors, including the brief period, which the authors themselves call a snapshot. what this limits: extending these results to AI-generated materials in general: the authors name sample size, a brief implementation period they call a snapshot, and the particular topic and target group Quotation withheld because this record is not cleared for redistribution. The mechanism relied on to keep answers accurate is stated as a reduction in risk, not a guarantee. what this limits: retrieval-augmented generation as a fix for factual error in these materials: the paper states it reduces the risk of hallucinations rather than removing it Quotation withheld because this record is not cleared for redistribution. instructing students who did not complete a task to select GUESS, so those responses are dropped from the evaluation eliminates: non-attempts entering the scored set and being read as wrong answers or as performance · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. custom chatbots custom chatbots Quotation withheld because this record is not cleared for redistribution. GPTs GPTs Quotation withheld because this record is not cleared for redistribution. cognitive load cognitive load Quotation withheld because this record is not cleared for redistribution. effect on learning outcomes - AS THE ABSTRACT PUTS IT: an indefinite impact, stated in the same sentence as the significant gains in emotion, interest and self-efficacy Quotation withheld because this record is not cleared for redistribution. effect on learning outcomes - AS THE CONCLUSION PUTS IT: no notable differences in the performance test, and the influence remains unclear Quotation withheld because this record is not cleared for redistribution. custom chatbot - a configured instance of a vendor's LLM, which the paper notes OpenAI designates by a different name Quotation withheld because this record is not cleared for redistribution. replicates → previous research on emotions in chatbot-supported learning (reference [29]) Quotation withheld because this record is not cleared for redistribution. custom chatbots —same-as→ GPTs custom chatbots GPTs strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. Cronbach's alpha Cronbach's alpha Quotation withheld because this record is not cleared for redistribution. Shapiro-Wilk normality test Shapiro-Wilk normality test Quotation withheld because this record is not cleared for redistribution. Mann-Whitney U test Mann-Whitney U test Quotation withheld because this record is not cleared for redistribution. effect size (Cohen's d) effect size (Cohen's d) Quotation withheld because this record is not cleared for redistribution. inter-rater reliability (weighted kappa) inter-rater reliability (weighted kappa) Quotation withheld because this record is not cleared for redistribution. standard deviation standard deviation Quotation withheld because this record is not cleared for redistribution. mean mean Quotation withheld because this record is not cleared for redistribution. Sixth-graders who learned from a custom-chatbot-generated explanation reported more positive-activating emotions, more situational interest and higher self-efficacy, and lower intrinsic and extrinsic cognitive load, than those who learned from textbook material, while learning outcomes showed no clear effect. Quotation withheld because this record is not cleared for redistribution. The two groups did not differ on the overall score of the performance test. Quotation withheld because this record is not cleared for redistribution. On two values of the tabular subtask B the control group scored significantly higher than the AI-material group. Quotation withheld because this record is not cleared for redistribution. On subtask C the AI-material group scored significantly higher. Quotation withheld because this record is not cleared for redistribution. The two groups were equivalent in prior interest in mathematics before the intervention. Quotation withheld because this record is not cleared for redistribution. The reduction in cognitive load in the experimental group did not carry over into any measurable performance gain on the achievement test. Quotation withheld because this record is not cleared for redistribution. Large Language Models (LLMs) Large Language Models (LLMs) Quotation withheld because this record is not cleared for redistribution. chatbots chatbots Quotation withheld because this record is not cleared for redistribution. Retrieval Augmented Generation (RAG) Retrieval Augmented Generation (RAG) Quotation withheld because this record is not cleared for redistribution. multiple representations multiple representations Quotation withheld because this record is not cleared for redistribution. GPTs GPTs Quotation withheld because this record is not cleared for redistribution. type of learning material type of learning material Quotation withheld because this record is not cleared for redistribution. conventional textbook material conventional textbook material Quotation withheld because this record is not cleared for redistribution. emotions emotions Quotation withheld because this record is not cleared for redistribution. situational interest situational interest Quotation withheld because this record is not cleared for redistribution. intrinsic and extrinsic cognitive load intrinsic and extrinsic cognitive load Quotation withheld because this record is not cleared for redistribution. self-efficacy expectation self-efficacy expectation Quotation withheld because this record is not cleared for redistribution. learning time learning time Quotation withheld because this record is not cleared for redistribution. Subject-specific interest Subject-specific interest Quotation withheld because this record is not cleared for redistribution. learning performance learning performance Quotation withheld because this record is not cleared for redistribution. transfer transfer Quotation withheld because this record is not cleared for redistribution. N = 214 — number of sixth-grade German secondary-school students in the randomized controlled study Quotation withheld because this record is not cleared for redistribution. 11.7 years (SD = 0.51) — mean age of the participating students Quotation withheld because this record is not cleared for redistribution. 15 minutes — learning time allotted identically to both groups Quotation withheld because this record is not cleared for redistribution. 30 minutes — time allowed for the performance test measuring learning and transfer Quotation withheld because this record is not cleared for redistribution. = 0.8469 — mean weighted-kappa inter-rater agreement across the rating categories for subtask D Quotation withheld because this record is not cleared for redistribution. M = 2.24; SD = 0.88 vs M = 2.38; SD = 0.95 — last-term mathematics report-card grades of the control group versus the experimental group (German scale, 1 = best) Quotation withheld because this record is not cleared for redistribution. GPT-4.0 — the OpenAI base model the custom chatbot was configured on Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.48550/arxiv.2412.15747.