Perils and opportunities in using large language models in psychological research — TEI-tagged view

Suhaib AbdurahmanpersName, Mohammad AtaripersName, Farzan Karimi-MalekabadipersName, Mona J XuepersName, Jackson TragerpersName, Peter S ParkpersName, Preni GolazizianpersName, Ali OmranipersName, Morteza DehghanipersName (2024date). “Perils and opportunities in using large language models in psychological research.” PNAS Nexustitle·j.

Object of study — the use of large language modelsterm·key as participants and as text-analysis tools in psychological research

Research question — what limitations, ethical implications and potential LLMsterm·ai have specifically for psychological research

Contribution — an empirical case against GPTologyterm·dom: LLMterm·ai outputs lack human variance, fail established nomological networks, and are beaten by smaller fine-tuned models

Full text — tagged

Perils and opportunities in using large language modelsterm·key in psychological research Abstract The emergence of large language modelsterm·key (LLMsterm·ai) has sparked considerable interest in their potential application in psychological research, mainly as a model of the human psyche or as a general text-analysis tool. However, the trend of using LLMsterm·ai without sufficient attention to their limitations and risks, which we rhetorically refer to as “GPTologyterm·dom”, can be detrimental given the easy access to models such as ChatGPTterm·key. Beyond existing general guidelines, we investigate the current limitations, ethical implications, and potential of LLMsterm·ai specifically for psychological research, and show their concrete impact in various empirical studies. Our results highlight the importance of recognizing global psychological diversity, cautioning against treating LLMsterm·ai (especially in zero-shotterm·key settings) as universal solutions for text analysisterm·key, and developing transparent, open methods to address LLMsterm·ai’ opaque nature for reliable, reproducible, and robust inference from AI-generated data. Acknowledging LLMsterm·ai’ utility for task automation, such as text annotation, or to expand our understanding of human psychology, we argue for diversifying human samples and expanding psychology’s methodological toolbox to promote an inclusive, generalizable science, countering homogenization, and over-reliance on LLMsterm·ai.

Keywords: psychology, large language modelsterm·key, natural language processing, psychological diversity, psychological text analysisterm·key Not only are large language modelsterm·key (LLMsterm·ai) such as ChatGPTterm·key becoming increasingly embedded in people’s everyday life in many societies, they are becoming an important tool in psychological research. Here, we highlight the risks associated with the rushed application of these technologies to psychological research, a practice we call “GPTologyterm·dom.” We review and conduct a comprehensive analysis of both the benefits and risks associated with using LLMsterm·ai in psychological research and advocate for the development of reliable applications and the use of open, interpretable models. We also quantify, and warn against, cultural biases of LLMsterm·ai. A more inclusive approach is critical to ensuring reproducible, generalizable, and unbiased scientific insights, when employing LLMsterm·ai to study the human mind.

Introduction Technological innovations have enabled social and behavioral scientists to gather diverse forms of data about human psychology, paving the way for significant breakthroughs in psychological science and neighboring fields. These advancements have played a pivotal role in expanding our understanding of psychological processes. The development of neuroimaging (e.g. fMRI), online survey platforms (e.g. Mechanical Turk), and eye-tracking technology are just a few examples that have revolutionized psychological research in the last few decades. The digital revolution and emergence of “big data” facilitated the establishment of new fields such as computational social science ( 1 ). More recently, there has been a notable paradigm shift in artificial intelligence (AI) with the emergence of large language modelsterm·key (LLMsterm·ai): neural networks characterized by their deep layers and extensive scale, typically consisting of billions to over a hundred trillion parameters and trained on vast text datasets, enabling an unprecedented ability to understand, generate, and translate human language with remarkable subtlety and complexity. These AI models are trained on extensive collections of unlabeled text using self-supervised or semi-supervised learning methods, contributing to their remarkable language understanding and generation capacity. This new technology has been argued to possess the capacity to transform social science research ( 2 ).

Language models have advanced significantly through interdisciplinary collaboration, including contributions from psychology, which laid the foundation for modern language modeling. For example, inspired by the psychology of feedback and learning mechanisms, McClelland and Rumelhart ( 3 ) explored connectionist models demonstrating the potential of neural networks, and Rumelhart et al. ( 4 ) introduced key algorithms like backpropagation. Building upon these developments and motivated by theories in working memory and cognitive processes, Elman ( 5 ) introduced recurrent neural networks, enabling researchers to model sequential data in language models.

The availability of large textual corpora, increased computing power, and advancements in deep-learning techniques have recently contributed to the progress and refinement of language models. Notably, the Transformers architecture ( 6 ), which enables understanding of intricate relationships between different input components in a remarkably efficient and precise manner substantially improved natural language understanding and generation capabilities. Subsequently, LLMsterm·ai, like ChatGPTterm·key ( 7 ), based on different versions of Generative Pre-trained Transformer (GPT; ( 8 , 9 )), have had sizable implications in various domains, powering applications such as human-sounding chatbots and language translation systems. Their success lies in their impressive language generation capabilities and public accessibility, which has permeated research in various fields from medicine ( 10 ) to politics ( 11 ).

Studying how LLMsterm·ai operate offers promising opportunities to gain previously inaccessible insights into cognition as a whole, and perhaps even human cognition ( 12 , 13 ). Additionally, these models’ capabilities in text analysisterm·key and generation could possibly be harnessed by researchers as an easy-to-use method for analyzing textual data, such as coding texts for mental health assessment ( 14 ). There has recently been a rapid string of psychological research output related to and facilitated by these models ( 13 , 15–17 ). In particular, ChatGPTterm·key has been employed in a range of social and behavioral applications, from hate-speech classification ( 18 ) to sentiment analysis ( 19 ), often with promising results.

However, drawing parallels with the turbulent integration of previous technological innovations into psychological research suggests that the hurried or negligent implementation of LLMsterm·ai in psychology could give rise to unintended consequences. For example, when fMRI techniques first appeared, some researchers began haphazardly applying these techniques, resulting in many nonsensical, but statistically significant neural correlates: a phenomenon cleverly illustrated in an fMRI of post-mortem Atlantic Salmon ( 20 ), where the fish displayed apparent significant brain activity during the experimentterm·mth despite being deceased, and a study on the high likelihood of finding spurious correlations in fMRI research due to statistical misapplications ( 21 ). These two episodes now serve as a lesson for psychology students and researchers more broadly: that new technologies should be cautiously integrated into psychological research. Specifically, integrating LLMsterm·ai in the psychological research pipeline, such as in substitution for human participants, necessitates critically examining its limitations. Researchers should not simply ask themselves how they can use a new technology; they need to also ask themselves whether and why they should do so ( 22 ).

By no means do we uniformly cast doubt on the usefulness of LLMsterm·ai in psychological research. Instead, we aim to contextualize current and future opportunities that LLMsterm·ai may offer psychological research and suggest possible ways to navigate their limitations. While acknowledging their potential utility to improve psychological science, we advise caution regarding the unchecked application of LLMsterm·ai, at least in their current state, in psychological studies. To prevent issues like nonsensical, but statistically significant correlations, it is essential to approach LLMsterm·ai with caution, keeping in mind similar challenges the field has faced in recent decades (e.g. the credibility revolution; ( 23 )). The following section provides an overview of the downsides of the hurried use of LLMsterm·ai in psychology, and how it could negatively affect psychological findings if not applied critically and cautiously.

LLMsterm·ai should not replace human participants Many studies of state-of-the-art LLMsterm·ai have concluded that their outputs are highly “human-like” ( 24–26 ). For instance, Webb et al. ( 26 ) examined the analogical reasoning abilities of ChatGPTterm·key and found that it had developed an emergent capacity for zero-shotterm·key reasoning, allowing it to solve a wide range of analogy problems without explicit training. Some have argued that if LLMsterm·ai, such as ChatGPTterm·key, can indeed produce human-like responses to common measures in psychological science (e.g. judgments of actions, endorsements of values, perceptions of social issues), they might as well, at some point in the future, replace the human subject pool. For example, based on a substantive correlation between moral judgments made by humans and a language model (GPT-3.5), Dillion et al. ( 27 ) asked whether LLMsterm·ai can replace human participants in psychological research. While they do acknowledge limitations of the “AI as human participants” position (which is typically supported by showing an LLM-human correlation in some psychological domain; e.g. ( 17 , 27 )), such as their issues with representing different populations, capturing variability in human responses, and oversimplifying complex judgments and behaviors, it is important to stress the ethical and epistemic risks associated with this practice ( 28 ). Similarly, Ref. ( 29 ) warns against the anthropomorphism of AI systems, noting that such tendencies can mislead us into expecting human-like performanceterm·dom from systems that operate on fundamentally different principles. This caution is crucial as we consider the implications of deploying LLMsterm·ai in roles traditionally reserved for human participants.

First, research using LLMsterm·ai to simulate human participants needs to pay more attention to the substantial cross-cultural variation in cognitive processes—including moral judgments—around the globe (see Refs ( 30 , 31 )) and develop robust ways to mimic them using synthetic agents. Models like GPT have been trained chiefly on WEIRDterm·key (Western, Educated, Industrialized, Rich, Democratic; ( 32 )) people’s textual data primarily in English, perpetuating the English-centricity of psychology ( 33 ), hindering efforts to take linguistic diversity seriously. Hence, these models may struggle with accurately representing diverse populations (see a discussion of LLMsterm·ai inherently struggling to represent identity groups due to their training procedures in Wang et al. ( 34 )). For example, ChatGPTterm·key has shown gender biases favoring male perspectives and narratives ( 35 , 36 ), cultural biases toward American perspectives ( 37 ) or majority populations in general ( 38 ), and political biases favoring liberal, environmental, and left-libertarian viewpoints ( 11 , 39 , 40 ). These biases also extend to personality, morality, and stereotypes ( 41–43 ).

Generally, these models’ outputs reflect a WEIRDterm·key psychology such that the AI-human link substantially weakens as we collect “human” data from less WEIRDterm·key populations. In other words, the high AI-human correlation does not replicate when the human sample is less WEIRDterm·key ( 44 ). This is especially concerning because the WEIRD-people problem was originally devised as an awareness-raising rhetorical device to push researchers away from heavily relying on WEIRDterm·key human research participants (e.g. undergraduate students in North America; see Ref. ( 45 )). Substituting human participants with LLMterm·ai outputs would, therefore, be a step backward. Ignoring human diversity in psychology, amplified by the convenience of online samples like MTurk, has already led psychology research to be tunnel-visioned toward an extremely thin slice of human diversity ( 34 ). GPTologyterm·dom will constitute a move toward an even more myopic and less generalizable discipline.

Second, LLMsterm·ai seem to have a “correct answer” bias ( 17 , 40 ). Specifically, LLMsterm·ai fail to produce much variance in their answers to psychology survey questions: even if these questions pertain to topics—like moral judgment—for which there is no actual correct answer, and for which human answers would have diversity-of-thought. Simulating human diversity-of-thought using LLMsterm·ai might generally be nontrivial. For example, simply prompting an LLMterm·ai to respond to a question multiple times and then measuring the response variance, a common strategy in the social sciences ( 17 , 44 , 46 ), does not equate to meaningful variance that can be compared with humans. Generative language models, such as ChatGPTterm·key, compute a probability distribution over possible next words to produce an output sequence in an autoregressive manner ( 6 , 47 ). Specifically, the probability of a word being predicted next is given as P ( z i ) = e z i / T ∑ k = 1 n e z k / T , where z i represents the logit value associated with a potential output word w i given an input sequence { w 1 , … , w i − 1 } , which is, i.e. the model’s internal representation of the word after processing it ( 17 ). The “temperature” parameter ( T ) expresses whether the model chooses the most likely word according to the distribution ( T → 0 ), expressing deterministic behavior, or samples the words according to the distribution ( T → 1 ), supposedly expressing more creative nondeterministic behavior. However, the output variance when repeating the same prompt using a temperature of 1 then simply reflects the output probability over the response options and therefore how sure the model is about its response, which itself is affected by the correct answer bias observed by ParkpersName et al. ( 17 ) and others.

Conceptually, this is akin to repeatedly asking a question to the same participant instead of different participants. However, psychologists are usually interested in studying variance across participants to make inferences about behavioral patterns and the robustness of psychological phenomena (see discussion of LLMterm·ai outputs resembling group averages instead of individual differences in, e.g. ( 27 )). Thus, researchers using LLMsterm·ai to study human behavior need to move beyond methods that simulate single responses—such as merely predicting group averages or simulating an individual’s response across various tasks (e.g. ( 24 , 25 , 27 ))—and instead develop robust methods to emulate the complexity of human samples. Furthermore, LLMsterm·ai are trained on vast amounts of data, which can contain many of the items and tasks used in psychological experimentsterm·mth, thus leading the model to rely on its memory instead of making inferences, exacerbating the problem above. To get unbiased evaluations of human-like LLMterm·ai behavior, researchers need to make sure that their tasks are not part of the model’s training data ( 48 ) or adjust the models not to affect the outcome of their experimentsterm·mth, such as by “unlearning” data from an LLMterm·ai ( 49 ).

To showcase how these issues concretely manifest, we analyze LLMterm·ai responses in the domain of human morality, highlighting how simple prompting strategies fail to capture human response patterns and particularly human variance. We designed a prompt and gave the Moral Foundationsterm·key Questionnaire-2 (MFQ-2; ( 30 )) to GPT-3.5 1,000 times. The MFQ-2 is a new measure published in 2023, and its items are unlikely to exist verbatim in the corpora on which GPT was trained; hence, it does not have the problem of extensively appearing in the training corpora. We then compared GPT’s responses to the distributions of a culturally diverse sample of humans from 19 populations. As can be seen in Fig. 1 (a), GPT produces substantially smaller variance in six moral domains (i.e. care, equality, proportionality, loyalty, authority, and purity) compared with actual human populations (all P s < 0.001 ). GPT-3.5’s variance was 43–121 times smaller than human data in different moral domains, even when using parameter settings for maximum variability in generated responses. Our results, in line with recent evaluations of strategies to induce demographic variance in LLMterm·ai outputs ( 40 , 50 , 51 ) and other emerging works ( 17 , 52 ), indicate that arguments such as that “silicon sampling allows researchers to simulate a diverse population of participants,” ( 27 ) could be premature and that the so-called “silicon sampling,” at least currently, fails in mimicking a diverse population of humans: a finding which is consistent with that of AtaripersName et al. ( 44 ).

ChatGPTterm·key vs. human moral judgments. Note: a) Distributions of moral judgments of humans (light blue) and GPT (light red) in six moral domains. Dashed lines represent averages. b) Inter-correlations between moral values in humans ( N = 3,902 ) and ChatGPTterm·key queries ( N = 1 , 000 ). c) Network of partial correlations between moral values based on a diverse sample of humans from 19 nations ( 30 ) and 1,000 queries of GPT. Blue edges represent positive partial correlations and red edges represent negative partial correlations.

To show that this phenomenon is robust across domains, we extended this analysis of GPT vs. human responses to a broad range of self-report measures, with participants from over 43 countries and spanning various psychological domains, such as personality, cognition, political orientation, and emotions: Big Five Inventory ( 53 ): N = 3 , 924 , Need for Closure ( 54 ): N = 315 , Need for Cognition ( 55 ): N = 900 , Right-wing authoritarianism Scale ( 56 ): N = 1 , 020 , Emphasizing–Systemizing Scale ( 57 ): N = 3 , 141 , Rational–Experiential Inventory Scale ( 58 ): N = 1 , 456 . Across these psychological constructs, we consistently found that ChatGPTterm·key responses generally showed significantly less variance across all measures (see Table S20 for an overview of variance differences across all surveys) and differed significantly between various demographics. For example, when responding to personality surveys, ChatGPTterm·key was significantly more agreeable than politically liberal individuals ( d = − 0.230 , P < 0.001 , 95% CI [ − 0.369, − 0.091]), conservatives ( d = − 0.403 , P < 0.001 , 95% CI [ − 0.550, − 0.257]), and moderates ( d = − 0.285 , P < 0.001 , 95% CI [ − 0.424, − 0.145]) as shown in Fig. 2 . Beyond personality dimensions, ChatGPTterm·key, for example, also endorsed significantly less right-wing authoritarianism than male participants ( d = 0.44 , P < .001 , 95% CI [0.24, 0.65]), White participants ( d = 0.35 , P < 0.001 , 95% CI [0.16, 0.54]), and younger participants (18–24; d = 0.49 , P < 0.001 , 95% CI [0.24, 0.74]), but significantly more than explicitly liberal participants ( d = − 0.23 , P = 0.003 , 95% CI [ − 0.39, − 0.07]) as shown in Fig. 3 . Differences of this kind and strength were observed across all surveys and demographic groups. See the Supplementary Materials for a detailed summary of all demographic differences, across a variety of psychological constructs.

Comparing ChatGPTterm·key against humans grouped by political opinion for responses on the Big Five Inventory. Note: Figure shows the response distribution of humans and ChatGPTterm·key across the five-factor personality constructs and for different human demographics. Figure shows that ChatGPTterm·key gives significantly higher responses on Agreeableness, Conscientiousness and significantly lower responses on Opennessterm·dom and Neuroticism. Importantly, ChatGPTterm·key shows significantly less variance compared with all demographic groups on all personality dimensions.

Comparing ChatGPTterm·key against humans across various demographic variables for the Right-Wing-Authoritarianism scale. Note: Figure shows the response distribution of humans and ChatGPTterm·key on the RWA scale for different human demographics. ChatGPTterm·key shows significantly lower average RWA than male, white, and young participants but not explicitly liberal participants. Importantly, ChatGPTterm·key shows significantly less variance compared with all demographic groups.

Third, we are skeptical that GPT responses can make nomological networks in well-established theoretical frameworks. Dillion et al. ( 27 ) mention that “researchers can give LLMsterm·ai different questions and see if they act as expected within a nomological net (e.g. form a reliable scale).” To demonstrate how GPT constructs a moral psychological nomological network, we looked at inter-correlations between moral domains (Fig. 1 (b)) and the network of moral domains based on partial correlations (Fig. 1 (c)) in a diverse human sample ( N = 3,902; ( 30 )) and 1,000 GPT queries. As shown in Fig. 1 (b) and (c), the networks of moral values are substantially different, indicating that GPT struggles to produce previously established nomological networks.

In sum, probing LLMsterm·ai can be a fruitful direction for future research ( 2 , 59 ). However, we caution against the replacement of human participants with LLMsterm·ai, because (i) synthetic AI-simulated sampling would ignore psychological and linguistic diversity that the field desperately needs in order to investigate beyond WEIRDterm·key psychology (e.g. ( 32–34 , 60 )); (ii) LLMsterm·ai often fail to show meaningful variance (or diversity) in their judgments; (iii) the outputs of current LLMsterm·ai do not seem to replicate previously established nomological networks; and (iv) reliance on LLMsterm·ai may foster epistemic complacency and illusions of understanding, potentially resulting in scientific monocultures that suppress diverse methods, curtail innovation, and increase vulnerability to errors ( 28 ).

Overall, probing the psychology of LLMsterm·ai is scientifically meaningful and practically important. However, it should not replace the scientific study of Homo sapiens but rather supplement it (e.g. ( 27 , 28 )). Of course, these limitations do not imply that all uses of LLMsterm·ai should be avoided in psychological research. For example, some researchers have argued that AI could be regarded as an agent of “assistance,” “improvement,” or “augmentation” (e.g. ( 22 )). Framed in the latter way, LLMsterm·ai can be thought of as tools to help improve the research process in psychology. It is indeed exciting to try different psychological measures (traditionally developed for humans) on AI tools, but such examinations should be cautious and exploratory, and their results should not be hyperbolized. Similarly, LLMsterm·ai have the potential to serve in a role analogous to that of agent-based modelsterm·mth. For example, some recent studies present proofs-of-concept of how AI systems could simulate human behaviors (e.g. ( 61–63 )). Therefore, LLMsterm·ai show promise to become powerful simulators in psychological research, offering novel possibilities for enhancing our understanding through accessible and potentially scalable simulations, particularly when traditional participant-based methods are impractical. However, unlike traditional agent-based modelsterm·mth, which define the environment, agents, interactions, and their consequences to allow for inferences based on these parameters, LLMsterm·ai largely operate as black-boxes. Combined with issues of the outputs being often misaligned from real human behaviors and responses, as discussed above, this remains a serious concern: a collection of nebulous agents that can give misaligned/misleading responses. Thus, more work, particularly around validating LLMterm·ai simulations, is required to establish LLMsterm·ai as a viable alternative to or extension of traditional agent-based modelsterm·mth. Such careful considerations may lead to the development of more robust procedures to implement LLMsterm·ai in psychology, similar to how psychologists have addressed issues with other technologies. When researchers decide to use LLMterm·ai outputs as a proxy for a human sample, they should, for example, take into account task-relevant biases of the models in question (e.g. check for implicit biases against agents involved in a moral scenario when investigating moral judgments). This can be done in conjunction with efforts to reduce LLMterm·ai biases ( 64 ).

LLMsterm·ai are not a panacea for text analysisterm·key In this section, we discuss the perception of LLMsterm·ai as a quasi-universal off-the-shelf text analysisterm·key tool in psychology. We begin by explaining key distinctions in how LLMsterm·ai can be applied for text analysisterm·key. This is followed by an exploration of the currently common practice of overly focusing on zero-shotterm·key capabilities of LLMsterm·ai and an examination of their potential drawbacks. We then contrast LLMsterm·ai with non-LLM-based, top-down, theory-based text analysisterm·key tools, discussing the advantages of these smaller, interpretable methods and how they can benefit researchers. We highlight that researchers must carefully consider the particular demands and subtleties of their study topics when selecting natural language processing (NLP) methods for text analysisterm·key in psychology. This might include leveraging LLM-based approaches in scenarios where large-scale, general-purpose language modeling is necessary. Conversely, in situations requiring theory-driven explorations, top-down methods such as thematic content analysis or dictionary-based techniques (e.g. LIWC) may be more suitable. By thoughtfully combining these methods based on the study’s needs, researchers can achieve a more nuanced and comprehensive understanding of their textual data.

NLP methods using pretrained language models generally fall into two categories: those involving parameter updates (e.g. gradient-based) and those that do not. Fine-tuningterm·mth is a primary method involving parameter updates, where the pretrained LLMterm·ai is further trained on a task-specific dataset. This can range from updating a subset of parameters to adjusting the entire model, depending on task complexity and data availability ( 65–68 ) In contrast, zero-shotterm·key, one-shot, and few-shot learning, can be used without gradient updates by leveraging a pretrained model’s ability to generalize from limited or no task-specific data (i.e. examples), using the model’s pre-existing knowledge and understanding ( 8 , 69 , 70 ).

Recently, there has been a surge of studies using LLMsterm·ai zero-shotterm·key capabilities, that is their capability to perform tasks without any prior training on that specific task ( 8 , 71–73 ), for psychological text analysisterm·key, presumably due to their ease of use and accessibility. For example, Markowitz ( 74 ), Rathje et al. ( 75 ) and Zhu et al. ( 19 ), reported high performanceterm·dom of ChatGPTterm·key as an automated text analysisterm·key tool, such as for sentiment analysis, offensive language, thinking style, or emotion detection. Rathje et al. ( 75 ) further concluded that LLMsterm·ai constitute a viable all-purpose method for psychological text analysisterm·key, arguably more convenient than small(er) language models and traditional techniques in NLP, due to their ability to handle diverse tasks within a single model without needing task-specific adjustments, and their user-friendly design that minimizes the need for complex coding, making them more accessible to psychologists and potentially encouraging broader research engagement. However, this perspective of LLMsterm·ai, as a convenient and comprehensive tool in psychological text analysisterm·key, is challenged by recent critiques emphasizing their limitations, such as inconsistencies in text annotations, difficulties in explaining complex constructs such as implicit hate speech, and a potential lack of depth in specialized or sensitive areas ( 18 , 76 , 77 ).

Furthermore, Reiss ( 77 ) demonstrated that LLMsterm·ai may produce inconsistent outputs when subjected to minor prompt variations and can fall short of scientific thresholds for reliability, sometimes even when pooling outputs from multiple repetitions and prompts. Kocoń et al. ( 76 ) found that LLMsterm·ai may struggle with complex, subjective tasks such as emotion recognition, and Huang et al. ( 18 ) found that LLMsterm·ai can reinforce lay perspectives when detecting implicit hate speech and may thus mislead nonexperts with their explanations in cases where they make incorrect decisions. Furthermore, the difference in convenience between zero-shotterm·key applications of LLMsterm·ai and fine-tuningterm·mth of models may not be as stark as commonly perceived. Fine-tuned small(er) language models for various tasks are now increasingly publicly available. Similarly, there are more and more high-quality and specialized datasets available to researchers for fine-tuningterm·mth language models themselves. Examples of such corpora span across fields, such as morality, personality, or sentiment analysis ( 78–83 ).

Notably, even smaller fine-tuned models can in many cases, where data for fine-tuningterm·mth is available, fare equally well or even better than zero-shotterm·key applications of LLMsterm·ai. For instance, in most reported cases in Rathje et al. ( 75 ), small fine-tuned language models, including older models such as base-BERT (Bidirectional Encoder Representations from Transformers; ( 65 )), outperform even the newest generation of ChatGPTterm·key (GPT-4) on a variety of text annotation tasks. Notably, in Rathje et al. ( 75 ) GPT-4 (zero-shotterm·key) was outperformed on English language sentiment analysis by a fine-tuned model developed before the breakthrough Transformers architecture ( 84 ). Additionally, these smaller language models have since then further undergone significant improvement, such as Sentence-BERT ( 85 ), RoBERTa ( 86 ), and their subsequent iterations, highlighting additional potential performanceterm·dom advantages. While zero-shotterm·key applications of LLMsterm·ai might offer immediate accessibility, the most expedient choice is often not the most effective. Thus, researchers should be cautious of leaning too heavily on the allure of convenience. We explicitly highlight here that appropriate methods do not exclude LLMsterm·ai. There are various alternative ways to utilize LLMsterm·ai, such as by fine-tuningterm·mth the LLMterm·ai on the respective task, or via few-shot prompting which includes few examples of solving a task in a prompt so that the LLMterm·ai can generalize to the given examples ( 8 , 72 , 87 ).

Building on this perspective, we extend these recent studies by investigating ChatGPTterm·key’s ability to annotate moral language across three distinct settings: zero-shotterm·key, few-shot, and fine-tuned. Extracting moral values from (online) texts is a difficult task, where even trained human annotators show significant variance. This is because moral values are often only implicitly signaled in language, and online posts often contain little contextual information due to length constraints. For example, past works (e.g. ( 83 , 88 )) achieved average F1 scores of 0.50 and below despite training and evaluating their classifier for each moral value. Even when adding considerable background knowledge about posts’ content, such as in Lin et al. ( 89 ), reported F1 score remain moderate (between 0.54 and 0.76). Human morality has also been argued to be a particularly difficult concept to grasp for language models, making it an interesting test case for comparing powerful LLMsterm·ai against less complex smaller language models ( 90 ), as well as testing the efficacy of different LLMterm·ai application strategies. Specifically, we gave ChatGPTterm·key 2,983 social media posts that contained moral or nonmoral language and prompted it to determine if any and what specific type of moral language was used. We then compared it to a small BERTterm·key model that was fine-tuned on a separate subset of social media posts (see Supplementary Materials for details). To supplement our comparison of smaller vs. larger and complex vs. less complex models, we repeated this analysis using the Linguistic Inquiry and Word Count (LIWC; ( 91 )), which is a commonly used dictionary-based text-analysis method producing psychologically validated, and importantly interpretable features, from a given text. We compared the annotations of all models against the “ground truth” of human raters.

We find that fine-tuned BERTterm·key vastly outperformed ChatGPTterm·key applied in a zero-shotterm·key setting, achieving an F1 score of 0.48 vs. ChatGPTterm·key’s 0.22. Additionally, ChatGPTterm·key was more extreme in over- or under-predicting a moral sentiment, while BERTterm·key did in all but one case not significantly deviate from trained human annotators. Notably, even the LIWC-based approach outperformed ChatGPTterm·key (zero-shotterm·key) with an F1 score of 0.27 and was significantly less likely and less extreme in deviating from trained human annotators, despite being a magnitudes smaller, less complex, and cheaper model. Notably, ChatGPTterm·key (zero-shotterm·key) showed significantly more false positives, being over 10 times more likely Δ o d d s = + 1 , 240 % , P < 0.001 to predict a text as moral compared to human raters (see Table S2 ). As anticipated, both few-shot learning and fine-tuningterm·mth increased the performanceterm·dom of ChatGPTterm·key in our experimentsterm·mth. When applied in a few-shot setting, ChatGPTterm·key achieved an F1 score of 0.32. Notably, fine-tuningterm·mth ChatGPTterm·key on the same data used for the BERTterm·key model resulted in a superior F1 score of 0.53 outperforming the fine-tuned BERTterm·key model. Particularly, we find that fine-tuningterm·mth ChatGPTterm·key leads to significant improvement in identifying the Loyalty foundation (about 25% increase in F1 score) and Care foundation (about 10% increase in F1 score). See the full overview of foundation-level F1 scores for all classifiers in Table S1 . However, note that in our experimentsterm·mth, ChatGPTterm·key was still more extreme in over- or under-predicting individual moral sentiments compared with BERTterm·key, which replicated the distribution of moral foundationsterm·key most similar to human annotators. For example, ChatGPTterm·key (few-shot) was about twice as likely Δ o d d s = + 90 % , P < 0.001 more likely to predict a text as nonmoral compared with human raters. See Table S2 for an overview of the predicted foundation distributions compared with human ground truth.

Additionally, the performanceterm·dom of both zero-shotterm·key and few-shot ChatGPTterm·key was extremely poor on several foundations, often reaching F1 scores of nearly zero. Rathje et al. ( 75 ) report similarly low performancesterm·dom for the current flagship versions, GPT-4 and GPT-4-Turbo, on the Authority, Purity, and Proportionality foundations. Our fine-tuned models also performed substantially better on average, with fine-tuned BERTterm·key achieving an average F1 score of 0.48 and fine-tuned ChatGPT-3.5 an average score of 0.53, compared with reported F1 scores of 0.34 for GPT-4 and 0.29 for GPT-4-Turbo. In fact, our fine-tuned BERTterm·key model demonstrated superior performanceterm·dom over both GPT-4 and GPT-4-Turbo across all foundations except the Loyalty foundation, and fine-tuned GPT-3.5 outperformed GPT-4 and GPT-4-Turbo (zero-shotterm·key) on all foundations. See Table S3 comparing our fine-tuned model to the GPT-4 results presented by Rathje et al. ( 75 ) on the same moral foundationsterm·key. Interestingly, even our zero-shotterm·key application of GPT-3.5 outperformed the reported GPT-4 results on three foundations (Care, Proportionality, Authority), indicating inconsistent advancements between newer and older models. Further comparisons (Tables S7 and S8 in Rathje et al. ( 75 )) on another moral foundationsterm·key dataset, on which the BERTterm·key model was not fine-tuned (MFTC; ( 81 )), showed that BERTterm·key, despite being applied cross-context, outperformed GPT-4 on the Authority and Purity foundations, and GPT-4-Turbo also on the Loyalty foundation. GPT-4 performed slightly better than BERTterm·key on the Loyalty foundation and significantly better only on the Care foundation. Rathje et al. ( 75 ) argue that the change in performanceterm·dom for the fine-tuned BERTterm·key models between datasets is indicative of fine-tuned models’ inflexibility across different contexts or datasets. We suggest a slightly different narrative, given that the fine-tuned model still outperformed GPT-4-Turbo on three out of four foundations (and two out of four for GPT-4) and performed better on average than both. These findings show that, first, LLMsterm·ai’ supposed cross-context and flexibility advantages may not always hold, and second, that using LLMsterm·ai conveniently “out-of-the-box” can sometimes drastically fail while highlighting that fine-tuningterm·mth can mitigate the very same issues.

It should be noted here that while fine-tuningterm·mth ChatGPTterm·key or applying a few-shot paradigm can yield better results, such processes are significantly more resource-intensive and expensive compared with LIWC or even BERTterm·key. See also the discussion of environmental costs of LLMsterm·ai in the Supplementary Materials , as training and deploying LLMsterm·ai is energy intensive and therefore connected to significant carbon emissions ( 92 , 93 ). Overall, our results show that LLMsterm·ai can be a superior tool but they also show that achieving this level of performanceterm·dom might not always be trivial and convenient. For example, fine-tuningterm·mth can be costly, might not always work, and negates many of the conveniences of using LLMsterm·ai in a zero-shotterm·key setting by requiring a dataset for training and expertise with training models. As such, researchers should weigh the benefits and drawbacks of each method before choosing one, similar to how they would choose a statistical method for their data analysis.

Additionally, ChatGPTterm·key and other closed-source models require (if possible at all) fine-tuningterm·mth on a corporate platform, which introduces constraints compared with models like BERTterm·key which provide full control about the fine-tuningterm·mth process and can be trained offline without additional costs (see the following section for a detailed discussion of issues with closed-source and proprietary vs. open-source models). Few-shot learning, although promising, hinges on the relevance and quality of examples, potentially making it less stable than fine-tuningterm·mth ( 94 ). As such, researchers should choose a model and how to apply it based on the availability of task-specific data, available computing resources, and research-specific considerations (e.g. interpretability, control over the model). Fine-tuningterm·mth is preferable if training data are available. Smaller models (e.g. BERTterm·key) are then preferable to larger models (e.g. ChatGPTterm·key) if computing resources (or financial, for commercial services) are limited. In scenarios where gathering extensive task-specific data is impractical or impossible, LLMterm·ai’s zero-shotterm·key or few-shot capabilities are advantageous.

To further highlight the limitations of zero-shotterm·key LLMterm·ai applications, we examined to what extent ChatGPTterm·key’s zero-shotterm·key annotations were biased toward specific demographics. We did so by using human annotator demographics and psychometrics provided for our test data (see Supplementary Materials for a detailed description of the experimental procedures). We found that ChatGPTterm·key is less likely to agree with conservative ( Δ o d d s = − 4 % , P < .001 ) or collectivist ( Δ o d d s = − 30 % , P < 0.001 ) annotators and more likely to agree with individualistic ( Δ o d d s = + 142 % , P < 0.001 ) annotators. Additionally, it is biased toward younger ( Δ o d d s = − 1 % y e a r , P = 0.033 ), more open-minded ( Δ o d d s = + 1244 % , P < 0.001 ), and agreeable ( Δ o d d s = + 156 % , P < 0.001 ) annotators. Interestingly in terms of the six moral foundationsterm·key ( 30 ), it is biased toward annotators who have a lower preference for equality ( Δ o d d s = − 97 % , P < 0.001 ) and loyalty ( Δ o d d s = − 96 % , P < 0.001 ) and those who endorse more care ( Δ o d d s = + 31 , 159 % , P < 0.001 ) and proportionality ( Δ o d d s = + 21 , 215 % , P < 0.001 ) values. This aligns with our additional findings of ChatGPTterm·key over-weighing the care foundation when responding to the MFQ-2 (see additional analyses in Supplementary Materials ). Potentially, this could be linked to post-hoc measures by OpenAI to avoid an AI that endorses harm or does not care about people’s emotional well-being. Agreement with proportionality and equality endorsing annotators aligns with past findings that ChatGPTterm·key is (left-)libertarian-leaning ( 11 , 39 , 40 ). See Tables S4–S10 for a summary of annotation biases.

Finally, we analyzed how LLMsterm·ai, at least at this stage of their development, fare against theory-based, top-down constrained methods. This is relevant because LLMsterm·ai are primarily considered language analysis tools due to their broad capabilities stemming from their vast parameter sets, training data, and training procedures. However, this flexibility and performanceterm·dom come at the price of reduced interpretability and reproducibilityterm·key (i.e. the trade-off between a higher-performing black box and a lower-performing interpretable method). An often-stated reason for why some researchers of psychological text analysisterm·key prefer neural-network-based models over simple, theory-driven methods is the purportedly superior predictive power of the former. If LLMsterm·ai cannot outperform top-down methods in zero or few-shot settings, this would be another reason for psychologists (and other social scientists) to consider using top-down models rather than LLMsterm·ai (at least the existing ones), because in social sciences, theory and interpretability are of prime concern ( 95 ).

We compared ChatGPTterm·key against Contextualized Construct Representationterm·mth (CCR; ( 52 )), which is a method that combines psychometric scales with small language models (e.g. S-BERT; ( 85 )) to extract psychological information from texts. This includes assessments of individuals’ values, political ideology, cultural norms, religiosity, and need for cognition. By relying on validated psychometric scales, CCR places strong, theory-based, top-down constraints on its underlying small language model, which provides better interpretability and easy application. Comparing ChatGPTterm·key and CCR, we found that ChatGPTterm·key (zero-shotterm·key) fails to outperform CCR in inferring psychological outcomes from human-written essays. CCR substantially outperformed ChatGPTterm·key ratings when prompted to infer psychological constructs directly (Dunnett’s Test; d = − 2.25 , P = 0.005 , 95% CI [ − 3.88, − 0.62]), that is predicting the construct scores directly. See Figs. S30 and S31 for an overview of model performancesterm·dom across all self-report measures (i.e. cultural orientation, personal values, moral judgments, political ideology, need for cognition, and norm violations). CCR and ChatGPTterm·key performed on par when inferring psychological variables from essays about everyday life (Fig. S31 ) when prompted on the item-level, that is, when predicting each scale item separately and then calculating the construct score (Dunnett’s Test; d = 0.21 , P = 0.9404 , 95% CI [ − 1.42, 1.84]).

Taken together, past work and our present findings consistently demonstrate that for many use cases, smaller (fine-tuned) models can be more powerful and less biased than the current large (generative) language models, particularly in zero-shotterm·key and few-shot settings. For example, consider a study examining the language used in online support fora for individuals with anxiety disorders. A researcher using a smaller, specialized language model may be able to uncover subtle nuances and specific patterns of language that are directly relevant to the domain of interest (e.g. worry, intolerance of uncertainty). This targeted approach can yield more profound insights into the experiences of individuals with anxiety, shedding light on their unique challenges and potential interventions. By leveraging specialized language models or top-down methods such as CCR, or LIWC, researchers can strike a balance between comprehensiveness and granularity, enabling a more nuanced exploration of textual data.

Crucially, we do not discourage the use of LLMsterm·ai as a text-analysis tool in all respects. Particularly in cases where data for fine-tuningterm·mth is scarce, such as with new constructs or understudied populations, LLMsterm·ai’ zero-shotterm·key capabilities may still offer acceptable performancesterm·dom and allow researchers to investigate urgent research questions. Methods such as few-shot prompting can be effective and efficient in these cases as they only demand a handful of representative examples. Across our experimentsterm·mth, we found that LLMsterm·ai can achieve high performancesterm·dom but stress that this cannot always be achieved using the model’s zero-shotterm·key capabilities and instead requires exploring techniques, such as few-shot prompting or outright fine-tuningterm·mth of the LLMterm·ai (see also work on optimizing prompt design, such as “Chain-of-Thought” ( 71 , 96 ), a recent technique for eliciting complex multistep reasoning). Similarly, our additional analyses, showing increased performanceterm·dom for ChatGPTterm·key when predicting at the item level instead of the construct level, highlight that LLMsterm·ai can benefit from integrating theory-driven approaches. Extending this line of work, developing methods that can combine the benefits of both approaches is a promising endeavor for future work. With the constant and rapid development of LLMsterm·ai that address performanceterm·dom and bias issues, these concerns could already be mitigated in the near future.

We emphasize the importance of researchers constantly assessing these limitations and opportunities while exercising caution when defaulting to the most convenient choice. As with all empirical methods, LLM-based methods need to be validated and benchmarked. We recommend benchmarking LLM-based findings against more established text-analytic methods to make LLMsterm·ai more useful for psychological inference. For example, the gold-standard measure for text annotation is human annotation, and LLM-based annotations should be comprehensively validated—on a task-by-task basis—against a small labeled corpus before using these models at scale ( 97 ). This approach helps establish the validity of the analysis by reducing the potential biases that may arise from relying solely on automated techniques. Additionally, validation allows researchers to assess the inter-rater reliability of the annotations, providing a measure of the robustness of the analysis. The human element in validation brings valuable insights, subjective judgments, and contextual understanding that may be challenging for LLMsterm·ai to capture accurately.

Reproducibilityterm·key matters Reproducibilityterm·key pertains to replicating and verifying results using the same data and methods ( 98 ). However, particular challenges arise when applying these principles to LLMsterm·ai, particularly proprietary ones, whose black-box nature impedes the reproducibilityterm·key of findings that pertain to them. This limitation poses a significant obstacle to achieving reproducibilityterm·key in studies that rely on LLM-generated data or analyses beyond the often discussed nondeterministic nature of LLMsterm·ai (see discussions of nondeterministic outputs and “temperature” parameter of LLMsterm·ai in Refs. ( 46 , 75 , 99 )) and LLMsterm·ai’ black-box nature impeding interpretability (e.g. ( 99 )). Importantly, these issues can interact to cause further obstacles for the application in psychological research.

For example, LLMterm·ai biases can change over time for LLMsterm·ai that undergo updates over time. Each time modifications are made to ChatGPTterm·key’s algorithms for performanceterm·dom enhancement, the nature and scope of biases embedded in the model may change. This could impact the effectiveness of previously established “best practices” and debiasing strategies ( 15 , 100 ). Currently, ChatGPTterm·key, and to our best knowledge any other closed-source model, does not freely provide past versions that allow researchers to use the model from specific points in time (e.g. “gpt3.5-January-2023”) to reproduce research results. After each major update, it only provides a snapshot, which is then deprecated within 3 months to 1 year. This means that even when updates address and mitigate detected biases, they also introduce the potential for “process reproducibilityterm·key failure” ( 98 ) in the generated data and impede reproducibilityterm·key, critical for scientific rigor ( 15 , 100 , 101 ). This also affects currently emerging practices for reproducibilityterm·key, such as researchers sharing prompts and Application Programming Interface (API) parameters because the effect of these parameters (or the parameters themselves) and the outputs generated from these prompts can change over time. Importantly, new iterations do not guarantee equal or better performanceterm·dom across all tasks. For example, Rathje et al. ( 75 ) report inconsistent results between GPT-3.5 and GPT-4 performanceterm·dom on various text-analysis tasks—for example, GPT-4 sometimes performs more poorly than GPT-3.5—which lends support to our concern that nontransparent changes to the model can cause unforeseen challenges. Additionally, current state-of-the-art LLMsterm·ai come with additional opaque fine-tuningterm·mth, such as reinforcement learning using human feedback (RLHF; ( 102 , 103 )), for which usually neither the training method nor the training data are known. Therefore, researchers should carefully consider the trade-off between forgoing control over these procedures ( 100 ) and fine-tuningterm·mth models themselves.

Researchers should be aware of the current black-box nature of LLMsterm·ai ( 15 , 100 ) not only from an open-science standpoint. More generally, researchers should be interested in access to high-quality, informative semantic representations and the algorithms used to generate outputs, instead of only the outputs. One of the main advantages of computational models is that they allow us to “look under the hood” and thus make inferences about psychological processes that may be difficult to test otherwise. Thus, using proprietary LLMsterm·ai that do not provide this level of access may constitute a missed opportunity for theory-based work in psychology that aims to leverage innovations in computer science. Notably, the lack of transparency in how ChatGPTterm·key generates responses implies that researchers cannot ascertain the underlying mechanisms and origin of biases that may be influencing the outputs ( 101 , 104 , 105 ). Such lack of transparency is antithetical to the scientific principles of opennessterm·dom and replicability that should be central to computational–psychological research ( 106 ).

Computational–psychological research that aims to achieve replicable findings with LLMsterm·ai can use models that have a publicly available open architecture, such as BigScience Large Open-science Open-access Multilingual Language Model (BLOOM; ( 107 )) and Large Language Modelterm·key Meta AI (LLaMA; ( 108 )). These open-source models provide researchers with access to network architecture, including datasets and parameters such as pretrained weights ( 109 ). The less black-box nature of these open-source models can help researchers make the exact version used in their works available to others ( 100 ), thus facilitating transparency and reproducibilityterm·key. In the Supplementary Materials , we detail how a complete research pipeline using a LLaMA-based model can be conducted and show that the results are comparable to a pipeline using a closed-source LLMterm·ai. Despite the model’s significantly smaller size (7 billion vs. 175 billion parameters) and while being self-hosted on consumer-grade hardware, it achieved a comparable performanceterm·dom in a text annotation task discussed above (F1 of 0.23 vs. 0.22) and a similar response pattern when answering self-report psychological measures. Compared with closed-source LLMsterm·ai, open-source LLMsterm·ai may thus come with a replicability benefit and increased control over the model when it comes to computational–psychological research. Additionally, certain approaches aimed to allow for more robust inferences regarding LLMterm·ai outputs, such as “unlearning” ( 49 ) data and tasks used for evaluating LLMterm·ai capabilities from the training data, require an open model architecture (e.g. the ability to access and manipulate the probability distribution over the output tokens).

However, the primary focus should not be a debate of closed-source vs. open-source LLMsterm·ai, but instead addressing the specific problems for scientific research using either approach. Closed-source models, such as ChatGPTterm·key, may introduce ways to save and reload or share model weights and modify their models. For example, OpenAI continuously adds (but also deprecates) available model parameters. As of writing, increased reproducibilityterm·key has been announced as in beta development, and a “seed” parameter increasing deterministic behavior as well as a “system_fingerprint” value to help track model changes was recently added. However, the “seed” does not guarantee reproducibilityterm·key, and while the “system_fingerprint” increases transparency, allowing researchers to identify potential reasons for failure to replicate, it does not increase reproducibilityterm·key itself. Moreover, some problems that might be more pronounced with proprietary models, such as economic motivations, can still affect open-source models (e.g. LLaMA being developed by Meta, a for-profit company). Additionally, opennessterm·dom is a spectrum requiring researchers to closely monitor to what extent LLMsterm·ai fulfill relevant opennessterm·dom criteria for their projects. See Liesenfeld et al. ( 110 ) discussing of LLMterm·ai opennessterm·dom in detail as well as their opennessterm·dom ratings of current LLMsterm·ai https://opening-up-chatgpt.github.io , including their specific opennessterm·dom criteria and evaluation strategies https://github.com/opening-up-chatgpt/opening-up-chatgpt.github.io/tree/main/projects#criteria . As such, researchers should consider to what extent each model and the organization behind it conflicts with scientific and ethical principles of research ( 111–113 ). We caution against mistakenly treating the use of open-source rather than closed-source LLMsterm·ai as sufficient to solve the black-box problem: a problem that remains largely unsolved, even for open-source LLMsterm·ai. The overconfidence with which researchers may trust open-source LLMsterm·ai’ findings as replicable and transparent can systematically cause such researchers to over-rely on these findings, even in out-of-distribution settings where the findings may not generalize.

The reproducibilityterm·key concerns surrounding computational–psychological research done via LLMsterm·ai are further intensified by their ability to adopt different perspectives through prompting : explicit instructions to the model on which the output is conditioned on ( 42 , 114 , 115 ). Prompting has been shown to be a promising technique to increase the range of applications and versatility of LLMsterm·ai ( 116 ) and is also discussed as a strategy to mitigate some of the biases in these models’ outputs ( 117–119 ). One idea is to condition LLMsterm·ai’ outputs on explicit instructions to take the perspective of different groups or demographics. For example, some work has shown that prompting can improve the alignment of group-relevant responses (e.g. if prompted to respond like a Democrat, ChatGPTterm·key will answer opinion poll questions in the direction steered). However, at the moment, prompting nonetheless falls short in closing the gap in demographic group biases ( 40 ) with some studies observing overly simplistic cultural (outgroup) stereotypes instead of perspective-taking ( 34 , 51 ). Wang et al. ( 34 ) argued that due to their training processes, LLMsterm·ai inherently struggle with accurately representing identity groups, often leading to distorted, simplified, and stereotypical portrayals of these groups. Consequently, relying solely on prompting strategies might be insufficient to resolve these issues. Moreover, prompting also introduces significant challenges to reproducibilityterm·key in psychological research, because prompts can be constructed in numerous ways. Past work has shown that slight alterations and modifications in phrasing, context, or order can lead to substantially different responses ( 17 , 74 , 120–123 ).

We extended this literature by directly testing if changes in prompts would affect the results of our previously presented experimentsterm·mth. We repeated the moral sentiment analysis with a modified prompt (see Supplementary Materials ) and tested whether this led to different classification outputs. Furthermore, we repeated the survey responses collection using several modified prompts (see Supplementary Materials ) and tested whether the responses changed. The changes to the prompts were derived from past work that showed how minor changes to the self-report design, such as adding contextual information or changing the response scale, can elicit different response patterns ( 124 ). Our findings demonstrate, in line with recent works on the effect of prompting ( 17 , 74 , 120–122 ), that minor changes in prompts lead to significant differences in outputs. For example, adding a study introductionterm·mth when prompting ChatGPTterm·key to respond to the Big Five Inventory (BFI; ( 53 )), significantly reduced the scores for Opennessterm·dom ( d = − 0.30 ; P < 0.001 ; 95% CI [ − 0.34, − 0.26]), Extraversion ( d = − 0.54 ; P < 0.001 ; 95% CI [ − 0.58, − 0.50]), and Agreeableness ( d = − 0.14 ; P < 0.001 ; 95% CI [ − 0.18, − 0.10]). Similarly, modifying the order in which the types of moral foundationsterm·key are defined in the prompt changed how texts were classified. Using the modified prompt, ChatGPTterm·key was significantly less likely to annotate the care foundation ( − 56%, P < 0.001 ), the equality foundation ( − 62%, P < 0.001 ), the loyalty foundation ( − 33%, P = 0.009 ), and significantly more likely to annotate the authority foundation ( + 71%, P < 0.001 ), and purity foundation ( + 164%, P = 0.020 ). See the Supplementary Materials for extended results and a detailed overview of our procedures.

Our research emphasizes the importance of considering context and specific prompts when using ChatGPTterm·key to emulate diverse human behaviors ( 125 ). Continuous algorithmic changes necessitate ongoing validation of prompting strategies, a task made challenging by the complex and opaque nature of LLMsterm·ai ( 72 , 123 , 126 ). While these challenges do not negate the applicability of LLMsterm·ai in psychological research, they do highlight the need for robust, standardized methods akin to those used in traditional human research ( 124 , 127 ). As this is a rapidly evolving field, new AI models may account for all of the above concerns of diversity, variation, transparency, accuracy, and robustness. The progress that is expected to be made by LLMsterm·ai in the near future requires that empirical work be completed to highlight their strengths, weaknesses, and promises for further enhancement ( 74 ).

Lastly, there is a point to be made about the democratization and accessibility of computational tools. The newest generation of LLMsterm·ai offered as an online service (e.g. ChatGPTterm·key, Gemini, Claude) provide many researchers with an accessible yet powerful tool—representing a pivotal advancement. However, with this greater accessibility comes the responsibility to stay educated on both the capabilities and limitations of these models. Researchers need to maintain a critical perspective, especially given the impressive performanceterm·dom and interactive nature of LLMsterm·ai on some tasks, which might lead to a presumption that they are always the best option as the subject of study or as an assistant for automated text analysisterm·key. This can result in the oversimplification of these powerful tools, such as avoiding necessary fine-tuningterm·mth for convenience or due to a lack of awareness, thereby not leveraging their full capabilities and potentially leading to less desirable results compared with other methods, or ignoring unique challenges related to transparency and replication (as discussed above). It is further crucial to recognize that many advantages attributed to LLMsterm·ai are also present in other models. For instance, models such as BERTterm·key or open-source LLMsterm·ai can be accessed via APIs, providing a convenient and low-cost option for researchers who cannot self-host these technologies. This makes them comparably accessible and usable without extensive coding or technical expertise. Additionally, OpenAI offers embedding models, such as “text-embedding-ada-3,” that can be used in a similar manner to BERTterm·key for downstream tasks. Ultimately, the responsible use of any computational tool hinges on a thorough understanding of its capabilities and the thoughtful consideration of whether it is the most appropriate method for the task at hand. This balanced approach ensures that technological advancements are leveraged effectively and responsibly in research.

Conclusion GPTologyterm·dom —which we define as the hurried and unjustified application of LLMsterm·ai either as “replacements” for human participants, or as an off-the-shelf “one-size-fits-all” method in psychological text analysisterm·key—can lead to a proliferation of low-quality research, especially if the convenience of using LLMsterm·ai such as ChatGPTterm·key leads researchers to rely too heavily on them. While LLMsterm·ai, especially fine-tuned ones, can achieve impressive performancesterm·dom on many tasks, the presence of a WEIRDterm·key bias, along with the opaque and often irreproducible nature of these models, particularly the proprietary ones, makes them a double-edged sword for psychological research. This does not mean that LLMsterm·ai are inadequate to aid psychological research, but researchers must actively exercise caution and critically evaluate the limitations of these models before incorporating them into their research paradigms. In the present work, we empirically quantified some of the biases and limitations of these models across multiple psychological domains including moral judgments, personality traits, cultural orientation, and political ideology, among others. Psychological science, which has witnessed multiple adverse consequences when new technologies were haphazardly and heedlessly used, needs to strive for diversification of research samples, validation of different methods against one another, transparency, and ethical considerations in deploying LLMsterm·ai to ensure that the findings are robust, generalizable, and free from demographic biases. A commitment to rigor and replicability should guide the integration of AI into psychological research, not convenience.

Supplementary Material Contributor Information Suhaib AbdurahmanpersName, Department of Psychology, University of Southern California, Los Angeles, CA 90089, USA; Brain and Creativity Institute, University of Southern California, Los Angeles, CA 90089, USA.

Mohammad AtaripersName, Department of Human Evolutionary Biology, Harvard University, Cambridge, MA 02138, USA; Department of Psychological and Brain Sciences, University of Massachusetts Amherst, Amherst, MA 01003, USA.

Farzan Karimi-MalekabadipersName, Department of Psychology, University of Southern California, Los Angeles, CA 90089, USA; Brain and Creativity Institute, University of Southern California, Los Angeles, CA 90089, USA.

Mona J XuepersName, Department of Human Evolutionary Biology, Harvard University, Cambridge, MA 02138, USA.

Jackson TragerpersName, Department of Psychology, University of Southern California, Los Angeles, CA 90089, USA; Brain and Creativity Institute, University of Southern California, Los Angeles, CA 90089, USA.

Peter S ParkpersName, Department of Physics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.

Preni GolazizianpersName, Brain and Creativity Institute, University of Southern California, Los Angeles, CA 90089, USA; Department of Computer Science, University of Southern California, Los Angeles, CA 90089, USA.

Ali OmranipersName, Brain and Creativity Institute, University of Southern California, Los Angeles, CA 90089, USA; Department of Computer Science, University of Southern California, Los Angeles, CA 90089, USA.

Morteza DehghanipersName, Department of Psychology, University of Southern California, Los Angeles, CA 90089, USA; Brain and Creativity Institute, University of Southern California, Los Angeles, CA 90089, USA; Department of Computer Science, University of Southern California, Los Angeles, CA 90089, USA.

Supplementary Material Supplementary material is available at PNAS Nexus online.

Funding This research was supported in part by DARPA INCAS HR001121C0165 and AFOSR A9550-23-1-0463. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. M.A., M.J.X., and P.S.P. did not receive any funding in this project.

Preprint This manuscript was posted as a preprint: https://osf.io/preprints/psyarxiv/d695y .

Data Availability All data, supplementary study materials, and supporting code used in this work are publicly available through an Open Science Framework (OSF) repository at https://doi.org/10.17605/OSF.IO/NAFZY . The repository includes detailed instructions for replicating our findings.

References Associated Data Supplementary Materials Data Availability Statement All data, supplementary study materials, and supporting code used in this work are publicly available through an Open Science Framework (OSF) repository at https://doi.org/10.17605/OSF.IO/NAFZY . The repository includes detailed instructions for replicating our findings.