Adaptive Conversational Agents: Exploring the Effect of Individualized Design on User Experience

Adaptive Conversational Agents: Exploring the Effect of Individualized Design on User Experience Riefle Alamo Alonso Jungnitz Rothmaier Wiegand Zacateco Herrera Benz

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Adaptive Conversational Agents: Exploring the Effect of Individualized Design on User Experience 2023 riefle2023-adaptive-conversational-agents-exp riefle2023-adaptive-conversational-agents-exp.txt, the project's staged copy 90e993636785fede079c4adf981a99d1986d6d0e open-access information-systems

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them existence Quotation withheld because this record is not cleared for redistribution. adaptive conversational agent design Quotation withheld because this record is not cleared for redistribution. behavioural-experiment behavioural-experiment Quotation withheld because this record is not cleared for redistribution. mechanisms Accommodating one's communication style to the interlocutor signals an attitude toward them and thereby steers how the speaker and the interaction are evaluated Quotation withheld because this record is not cleared for redistribution. mechanisms An interaction partner that accommodates the user's needs signals attention and caring, which raises perceived social presence, which in turn produces enjoyment Quotation withheld because this record is not cleared for redistribution. mechanisms Matching the chatbot's communication style to the user's cognitive abilities lowers the cognitive effort the interaction demands and raises its efficiency, which drives re-use intention Quotation withheld because this record is not cleared for redistribution. mechanisms The unequal split between rational and intuitive participants left that group underpowered, producing the absence of significant differences Quotation withheld because this record is not cleared for redistribution. The pilot's sample was small and the authors treat the whole design as provisional pending a larger main study. what this limits: everything reported: DESPITE ITS LIMITED SAMPLE SIZE, OUR PILOT STUDY OFFERS VALUABLE INSIGHTS - a pilot, treated as provisional pending the main study Quotation withheld because this record is not cleared for redistribution. The within-subjects design may bias later user-experience responses; a between-subjects design is being considered instead. what this limits: the user-experience responses specifically: IT IS WORTH CONSIDERING A BETWEEN-SUBJECTS DESIGN ... AS THIS PREVENTS PARTICIPANTS FROM BEING BIASED IN THEIR RESPONSES Quotation withheld because this record is not cleared for redistribution. The cognitive-style group was imbalanced (10 rational vs 4 intuitive), which may explain the absence of significant effects there. what this limits: the null for cognitive style: A POSSIBLE EXPLANATION COULD BE THE IMBALANCE OF THE NUMBER OF PARTICIPANTS THAT WERE CLASSIFIED AS RATIONAL OR INTUITIVE (10 COMPARED TO 4) Quotation withheld because this record is not cleared for redistribution. Only two user characteristics were studied, and separately; other characteristics and their combinations remain unexamined. what this limits: the generality of the personalisation claim: only two user characteristics were studied, separately, so combinations remain untested Quotation withheld because this record is not cleared for redistribution. The summarised benefit does not carry the exceptions the results report: in one group the adapted agent was no better on satisfaction, and the UN-adapted agent was rated higher on enjoyment. what this limits: the conclusion that individualised CA design improves user experience: the results report two exceptions in the rational/intuitive cognitive style group, one of them favouring the un-adapted agent on enjoyment Quotation withheld because this record is not cleared for redistribution. The design used may bias the responses it collects, and the authors say a different design would prevent it. what this limits: these pilot ratings as unbiased measures of user experience: the authors state a within-subjects design was used and that a between-subjects design would prevent bias from previous questions Quotation withheld because this record is not cleared for redistribution. Two user characteristics were examined, separately, and the authors state the interplay between them was not investigated. what this limits: generalising to adaptation on user characteristics in general: two were examined separately, and their combinations and other CA design characteristics were not Quotation withheld because this record is not cleared for redistribution. The constructs are measured entirely by item sets borrowed from other studies, so the measurement inherits whatever those instruments assume. what this limits: these variables as directly observed: every construct is measured with item sets adopted from prior literature rather than developed or validated here Quotation withheld because this record is not cleared for redistribution. the theory of how humans accommodate their speech to an interlocutor, re-applied to design a chatbot that accommodates its style to the user Quotation withheld because this record is not cleared for redistribution. attention check questions embedded in the instrument, with the seven data sets that failed them excluded before analysis eliminates: inattentive responding - participants clicking through a questionnaire about conversational agents without reading it · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. conversational agents conversational agents Quotation withheld because this record is not cleared for redistribution. user experience user experience Quotation withheld because this record is not cleared for redistribution. service satisfaction service satisfaction Quotation withheld because this record is not cleared for redistribution. user experience - operationalised as three separate measured constructs: service satisfaction as the evaluative attribute, enjoyment as the hedonic one, re-use intention as the pragmatic one Quotation withheld because this record is not cleared for redistribution. satisfaction - a construct the paper notes is used for two different things across disciplines: user attitudes ABOUT A SYSTEM, or the perception of SERVICE INTERACTIONS Quotation withheld because this record is not cleared for redistribution. builds-on → communication accommodation theory (Giles et al., 2010) Quotation withheld because this record is not cleared for redistribution. extends → Altmeyer et al. (2020); Danckwerts et al. (2019); Gnewuch et al. (2020) Quotation withheld because this record is not cleared for redistribution. user experience —operationalised-as→ service satisfaction user experience service satisfaction strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. t-test t-test Quotation withheld because this record is not cleared for redistribution. Wilcoxon signed-rank test Wilcoxon signed-rank test Quotation withheld because this record is not cleared for redistribution. p-value p-value Quotation withheld because this record is not cleared for redistribution. median split median split Quotation withheld because this record is not cleared for redistribution. power analysis power analysis Quotation withheld because this record is not cleared for redistribution. effect size (Cohen's d) effect size (Cohen's d) Quotation withheld because this record is not cleared for redistribution. standard deviation standard deviation Quotation withheld because this record is not cleared for redistribution. In the need-for-interaction group, both enjoyment and re-use intention were significantly higher for the chatbot adapted to the user than for the non-adapted chatbot. Quotation withheld because this record is not cleared for redistribution. In the rational/intuitive cognitive-style group, none of the user-experience differences between the adapted and non-adapted chatbot reached statistical significance. Quotation withheld because this record is not cleared for redistribution. The improvement in service satisfaction from individualization was not statistically significant. Quotation withheld because this record is not cleared for redistribution. Individualization improved ratings in both groups except that, for cognitive-style adaptation, service satisfaction was unchanged and enjoyment was actually rated higher for the non-adapted chatbot. Quotation withheld because this record is not cleared for redistribution. Enjoyment of the chatbot adapted to the user's need for interaction averaged 5.70 against 4.90 for the one-size-fits-all chatbot. Quotation withheld because this record is not cleared for redistribution. conversational agents (CAs) conversational agents (CAs) Quotation withheld because this record is not cleared for redistribution. cognitive style cognitive style Quotation withheld because this record is not cleared for redistribution. intuitive cognitive style intuitive cognitive style Quotation withheld because this record is not cleared for redistribution. rational cognitive style rational cognitive style Quotation withheld because this record is not cleared for redistribution. need for interaction need for interaction Quotation withheld because this record is not cleared for redistribution. user characteristics user characteristics Quotation withheld because this record is not cleared for redistribution. enjoyment enjoyment Quotation withheld because this record is not cleared for redistribution. user characteristic user characteristic Quotation withheld because this record is not cleared for redistribution. user experience user experience Quotation withheld because this record is not cleared for redistribution. experimental task experimental task Quotation withheld because this record is not cleared for redistribution. 37 — participants recruited for the pilot study Quotation withheld because this record is not cleared for redistribution. 30 — usable data sets analysed after excluding inattentive participants Quotation withheld because this record is not cleared for redistribution. seven — data sets dropped for failing attention checks Quotation withheld because this record is not cleared for redistribution. 52 — participants in the pre-study used to set the classification thresholds Quotation withheld because this record is not cleared for redistribution. 4.7 — threshold score on the 7-point scale above which a participant counts as having a high need for interaction Quotation withheld because this record is not cleared for redistribution. mean = 5.70 compared to mean = 4.90 — enjoyment rating of the CA adapted to need for interaction versus the one-size-fits-all CA Quotation withheld because this record is not cleared for redistribution. 90 — minimum sample size the power analysis requires for the planned main study Quotation withheld because this record is not cleared for redistribution. n = 8 — participants in the two qualitative pre-tests validating the chatbots Quotation withheld because this record is not cleared for redistribution. 10 compared to 4 — imbalance between participants classified as rational and as intuitive, offered as the reason for the null result in that group Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at .