Bojana BodrožapersName, Bojana M. DinićpersName, Ljubiša BojićpersName (2024date). “Personality testing of large language models: limited temporal stability, but highlighted prosociality.” Royal Society Open Sciencetitle·j.
Object of study — temporal stabilityterm·mth and personalityterm·key profile of seven large language modelsterm·ai on personalityterm·key instruments
Research question — whether LLMsterm·ai' psychological traits and political attitudes are temporally stable
Contribution — two-occasion test-retest agreement coefficients for seven LLMsterm·ai on personalityterm·key instruments, with the profile read only where agreement is at least fair
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | limited temporal stability but a prosocial personality profile in LLM responses — “Results showed that personality scores of LLMs have variable intra-rater agreement and limited temporal stability in testing on two occasions. This means that one LLM can elicit divergent psychological profiles.” | |
| claim-type | existence — “Results showed that personality scores of LLMs have variable intra-rater agreement and limited temporal stability in testing on two occasions. This means that one LLM can elicit divergent psychological profiles.” | |
| method — evidence | — “To answer the question of the temporal stability and agreements of responses on psychological instruments applied to LLMs, we carried out a study in which we gave LLMs a series of instruments at two time points.” | |
| object of study — evidence | — “With that in mind, this research conducted an extensive investigation into seven LLMs, aiming to assess the temporal stability and inter-rater agreement on their responses on personality instruments in two time points.” | |
| research question — evidence | — “Thus, the priority should be to answer if the psychological traits and political attitudes of LLMs are temporally stable, meaning that there is an agreement in responses on the items provided at different time points (with identical parameters and prompt designs).” | |
| contribution — evidence | — “To answer the question of the temporal stability and agreements of responses on psychological instruments applied to LLMs, we carried out a study in which we gave LLMs a series of instruments at two time points.” | |
| mechanism | fewer parameters, more stability Llama3's and GPT-4o's stability may come from a smaller number of parameters. — “The temporal stability of GPT-4o and Llama3 may be influenced by the smaller number of parameters in a language model.” | mechanisms[0] |
| mechanism | mixture of experts Mixtral's eight collaborating expert models could account for its extreme instability. — “Mixtral combines 8 expert LLMs, each with 7 billion parameters, working collaboratively to produce outputs, which could account for extreme instability in personality traits over time [ 16 ].” | mechanisms[1] |
| mechanism | servile purpose The prosocial profile could be explained by LLMs being "created to be servile and help humans". — “A prosocial personality profile of LLMs could be explained by the initial purpose, as they are created to be servile and help humans in different areas of use.” | mechanisms[2] |
| mechanism | RLHF human bias RLHF may add a layer of human bias to personality responses. — “This system of RLHF acquisition may affect the interpretation of responses to personality assessment instruments as it progressively aligns towards human evaluator feedback, possibly adding a layer of human bias.” | mechanisms[3] |
| sense | temporal stability - agreement between an LLM's item responses at two time points with identical settings — “Thus, the priority should be to answer if the psychological traits and political attitudes of LLMs are temporally stable, meaning that there is an agreement in responses on the items provided at different time points (with identical parameters and prompt designs).” | senses[0] |
| relation | LLMs —same-as→ large language models — (LLMs) — “As large language models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands.” | concept_edges[0] |
| weight | LLMs —same-as→ large language models — 0.95 — “As large language models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands.” | concept_edges[0] |
| relation | ICC —same-as→ Interclass Correlation Coefficient — (ICC) — “The second is Interclass Correlation Coefficient (ICC).” | concept_edges[1] |
| weight | ICC —same-as→ Interclass Correlation Coefficient — 0.95 — “The second is Interclass Correlation Coefficient (ICC).” | concept_edges[1] |
| relation | SCS-R —same-as→ Self-Consciousness Scales—Revised — (SCS-R) — “Self-Consciousness Scales—Revised (SCS-R) [ 36 ] contains 22 Likert-type items” | concept_edges[2] |
| weight | SCS-R —same-as→ Self-Consciousness Scales—Revised — 0.95 — “Self-Consciousness Scales—Revised (SCS-R) [ 36 ] contains 22 Likert-type items” | concept_edges[2] |
| relation | BFI-2 —same-as→ Big Five Inventory-2 — (BFI-2) — “Big Five Inventory-2 (BFI-2) [ 43 ] contains 60 Likert-type items” | concept_edges[3] |
| weight | BFI-2 —same-as→ Big Five Inventory-2 — 0.95 — “Big Five Inventory-2 (BFI-2) [ 43 ] contains 60 Likert-type items” | concept_edges[3] |
| relation | SD3 —same-as→ Short Dark Triad — (SD3) — “Short Dark Triad (SD3) [ 45 ] contains 27 items measuring Dark Triad traits” | concept_edges[4] |
| weight | SD3 —same-as→ Short Dark Triad — 0.95 — “Short Dark Triad (SD3) [ 45 ] contains 27 items measuring Dark Triad traits” | concept_edges[4] |
| relation | BIMI —same-as→ Bidimensional Impression Management Index — (BIMI) — “Bidimensional Impression Management Index (BIMI) [ 46 ] contains 20 Likert-type items” | concept_edges[5] |
| weight | BIMI —same-as→ Bidimensional Impression Management Index — 0.95 — “Bidimensional Impression Management Index (BIMI) [ 46 ] contains 20 Likert-type items” | concept_edges[5] |
| relation | RLHF —same-as→ Reinforcement learning from human feedback — (RLHF) — “Reinforcement learning from human feedback (RLHF) significantly contributes to the training and alignment of LLMs.” | concept_edges[6] |
| weight | RLHF —same-as→ Reinforcement learning from human feedback — 0.95 — “Reinforcement learning from human feedback (RLHF) significantly contributes to the training and alignment of LLMs.” | concept_edges[6] |
| relation | ICC —contrasted-with→ Cohen’s kappa — Unlike — “Unlike Cohen’s kappa, which quantifies agreement based on all-or-nothing, the ICC incorporates the magnitude of the disagreement to compute agreement estimates, with larger-magnitude disagreements resulting in lower ICC than smaller-magnitude disagreements.” — “Unlike Cohen’s kappa, which quantifies agreement based on all-or-nothing, the ICC incorporates the magnitude of the disagreement” | concept_edges[7] |
| weight | ICC —contrasted-with→ Cohen’s kappa — 0.85 — “Unlike Cohen’s kappa, which quantifies agreement based on all-or-nothing, the ICC incorporates the magnitude of the disagreement to compute agreement estimates, with larger-magnitude disagreements resulting in lower ICC than smaller-magnitude disagreements.” | concept_edges[7] |
| relation | temporal reliability —operationalised-as→ intra-rater agreement — as a measure of — “An intra-rater agreement as a measure of temporal reliability (i.e. stability) was calculated via two types of coefficients.” — “An intra-rater agreement as a measure of temporal reliability” | concept_edges[8] |
| weight | temporal reliability —operationalised-as→ intra-rater agreement — 0.85 — “An intra-rater agreement as a measure of temporal reliability (i.e. stability) was calculated via two types of coefficients.” | concept_edges[8] |
| evidential status | number of parameters —causes→ temporal stability — may be influenced — “The temporal stability of GPT-4o and Llama3 may be influenced by the smaller number of parameters in a language model.” | concept_edges[9] |
| relation | number of parameters —causes→ temporal stability — may be influenced by — “The temporal stability of GPT-4o and Llama3 may be influenced by the smaller number of parameters in a language model.” — “may be influenced by the smaller number of parameters” | concept_edges[9] |
| weight | number of parameters —causes→ temporal stability — 0.5 — “The temporal stability of GPT-4o and Llama3 may be influenced by the smaller number of parameters in a language model.” | concept_edges[9] |
| evidential status | mechanism — may be influenced — “The temporal stability of GPT-4o and Llama3 may be influenced by the smaller number of parameters in a language model.” | mechanisms[0] |
| evidential status | mechanism — could account for — “Mixtral combines 8 expert LLMs, each with 7 billion parameters, working collaboratively to produce outputs, which could account for extreme instability in personality traits over time [ 16 ].” | mechanisms[1] |
| evidential status | mechanism — could be explained — “A prosocial personality profile of LLMs could be explained by the initial purpose, as they are created to be servile and help humans in different areas of use.” | mechanisms[2] |
| evidential status | mechanism — may affect — “This system of RLHF acquisition may affect the interpretation of responses to personality assessment instruments as it progressively aligns towards human evaluator feedback, possibly adding a layer of human bias.” | mechanisms[3] |
| replication | original | |
| limitation | Only predefined settings and no specific prompt were tested. — “This study was carried out with predefined settings and no specific prompt.” | limitations[0] |
| limitation | Only two testing occasions; replication with more occasions is needed. — “We revealed that the temporal stability of the LLMs’ responses is variable and future studies should replicate these results including more testing occasions.” | limitations[1] |
| limitation | GPT-4o's number of parameters is undisclosed. — “However, the missing piece in this equation is the number of parameters in GPT-4o, which remains undisclosed [ 13 ].” | limitations[2] |
| limitation | Companies do not disclose their fine-tuning processes. — “However, we cannot say what concrete differences in these processes are as this is not disclosed by companies OpenAI, Google, Meta and Mistral, running these models.” | limitations[3] |
| limitation | Whether responses reflect self-reflection or predefined algorithms cannot be said. — “However, we could not say if LLM responses are the result of conscious self-reflection or are just based on predefined algorithms.” | limitations[4] |
| limitation | Predictive validity of LLM scores is untested. — “Therefore, future studies should reveal the predictive validity of the LLMs’ scores.” | limitations[5] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | Agreement was lowest for GPT-4 and Gemini and highest for Llama3, then GPT-4o. — “The lowest values of agreement, including negative values, were found in GPT-4 and Gemini, while the highest level of agreement was observed in Llama3, followed by GPT-4o.” | content_tags.findings[0] |
| finding | Agentic management and BFI-2 scales reached the most agreement. — “The most agreement appears to be achieved for agentic management and BFI-2 scales.” | content_tags.findings[1] |
| finding | Public self-consciousness was unreliable in all LLMs, altruism mostly zero. — “The values were not acceptable for public self-consciousness in all LLMs; only one was acceptable for HEXACO-100 openness to experience; and they were mostly zero for altruism.” | content_tags.findings[2] |
| finding | LLMs showed a socially desirable, prosocial profile with lower Machiavellianism, except Mixtral. — “Taken together, LLMs displayed a socially desirable profile with mostly elevated agentic domain and well-adapted and prosocially oriented profile with higher agreeableness, altruism, conscientiousness, and extraversion, and lower Machiavellianism (except in the case of Mixtral).” | content_tags.findings[3] |
| finding | LLM personality scores have limited temporal stability over two occasions. — “Results showed that personality scores of LLMs have variable intra-rater agreement and limited temporal stability in testing on two occasions. This means that one LLM can elicit divergent psychological profiles.” | content_tags.findings[4] |
| finding | LLMs scored politically average, somewhat towards the right, against the expected left orientation. — “Although we expected them to show a liberal/left/progressive political orientation [ 5 , 7 , 8 ], the LLMs scored average, although the scores are somewhat lower towards the right-wing and more conservative attitudes.” | content_tags.findings[5] |
| quantity | 7 of 21 — — “GPT-4 is the most problematic, as negative values were obtained on seven out of 21 scales (all from BFI-2 or HEXACO-100).” | content_tags.quantities[0] |
| quantity | 17 of 21 — — “In the case of ICCs, almost all scales for Llama3 (17 out of 21), GPT-4o (16) and GPT-3.5 (14) showed at least fair agreement and 95% CIs that do not include zero.” | content_tags.quantities[1] |
| variable | intra-rater agreement — “An intra-rater agreement as a measure of temporal reliability (i.e. stability) was calculated via two types of coefficients.” | content_tags.variables[0] |
| variable | weighted Cohen’s kappa — “The first is weighted Cohen’s kappa which is appropriate for ordinal scales such as the Likert scale [ 48 ].” | content_tags.variables[1] |
| definition | agentic management — “Agentic management involves exaggerating one’s social or intellectual status [ 37 ].” | content_tags.definitions[0] |
| definition | Machiavellianism — “Machiavellianism refers to manipulation, exploitation, and cynical world view (e.g. [ 27 ]) and it constitutes an antagonistic, dark core together with other dark traits [ 54 ], thus it is unexpected for someone to have both high prosocial and Machiavellianism traits.” | content_tags.definitions[1] |
| key-terms | large language models, temporal stability, personality, Big Five, HEXACO, Dark Triad, impression management, Machiavellianism, Cohen’s kappa |
| concept | large language models — not defined in this paper's own words in this sentence — “As large language models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands.” | concepts[0] |
| concept | LLMs — large language models — “As large language models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands.” | concepts[1] |
| concept | temporal stability — meaning that there is an agreement in responses on the items provided at different time points (with identical parameters and prompt designs) — “Thus, the priority should be to answer if the psychological traits and political attitudes of LLMs are temporally stable, meaning that there is an agreement in responses on the items provided at different time points (with identical parameters and prompt designs).” | concepts[2] |
| concept | intra-rater agreement — not defined in this paper's own words; the paper uses it “as a measure of temporal reliability (i.e. stability)” — “An intra-rater agreement as a measure of temporal reliability (i.e. stability) was calculated via two types of coefficients.” | concepts[3] |
| concept | temporal reliability — not defined in this paper's own words; the paper glosses it as “(i.e. stability)” — “An intra-rater agreement as a measure of temporal reliability (i.e. stability) was calculated via two types of coefficients.” | concepts[4] |
| concept | weighted Cohen’s kappa — not defined in this paper's own words; the paper says it “is appropriate for ordinal scales such as the Likert scale [ 48 ]” — “The first is weighted Cohen’s kappa which is appropriate for ordinal scales such as the Likert scale [ 48 ].” | concepts[5] |
| concept | Cohen’s kappa — which quantifies agreement based on all-or-nothing — “Unlike Cohen’s kappa, which quantifies agreement based on all-or-nothing, the ICC incorporates the magnitude of the disagreement to compute agreement estimates, with larger-magnitude disagreements resulting in lower ICC than smaller-magnitude disagreements.” | concepts[6] |
| concept | Interclass Correlation Coefficient — not defined in this paper's own words in this sentence — “The second is Interclass Correlation Coefficient (ICC).” | concepts[7] |
| concept | ICC — Interclass Correlation Coefficient — “The second is Interclass Correlation Coefficient (ICC).” | concepts[8] |
| concept | Self-Consciousness Scales—Revised — not defined in this paper's own words in this sentence — “Self-Consciousness Scales—Revised (SCS-R) [ 36 ] contains 22 Likert-type items” | concepts[9] |
| concept | SCS-R — Self-Consciousness Scales—Revised — “Self-Consciousness Scales—Revised (SCS-R) [ 36 ] contains 22 Likert-type items” | concepts[10] |
| concept | Big Five Inventory-2 — not defined in this paper's own words in this sentence — “Big Five Inventory-2 (BFI-2) [ 43 ] contains 60 Likert-type items” | concepts[11] |
| concept | BFI-2 — Big Five Inventory-2 — “Big Five Inventory-2 (BFI-2) [ 43 ] contains 60 Likert-type items” | concepts[12] |
| concept | Short Dark Triad — not defined in this paper's own words in this sentence — “Short Dark Triad (SD3) [ 45 ] contains 27 items measuring Dark Triad traits” | concepts[13] |
| concept | SD3 — Short Dark Triad — “Short Dark Triad (SD3) [ 45 ] contains 27 items measuring Dark Triad traits” | concepts[14] |
| concept | Bidimensional Impression Management Index — not defined in this paper's own words in this sentence — “Bidimensional Impression Management Index (BIMI) [ 46 ] contains 20 Likert-type items” | concepts[15] |
| concept | BIMI — Bidimensional Impression Management Index — “Bidimensional Impression Management Index (BIMI) [ 46 ] contains 20 Likert-type items” | concepts[16] |
| concept | agentic management — involves exaggerating one’s social or intellectual status [ 37 ] — “Agentic management involves exaggerating one’s social or intellectual status [ 37 ].” | concepts[17] |
| concept | Machiavellianism — refers to manipulation, exploitation, and cynical world view (e.g. [ 27 ]) — “Machiavellianism refers to manipulation, exploitation, and cynical world view (e.g. [ 27 ]) and it constitutes an antagonistic, dark core together with other dark traits [ 54 ], thus it is unexpected for someone to have both high prosocial and Machiavellianism traits.” | concepts[18] |
| concept | number of parameters — indicates the complexity of the neural network, reflecting its density — “The number of parameters indicates the complexity of the neural network, reflecting its density, but does not necessarily correlate with the quality of the output [ 10 – 12 ].” | concepts[19] |
| concept | Reinforcement learning from human feedback — not defined in this paper's own words in this sentence — “Reinforcement learning from human feedback (RLHF) significantly contributes to the training and alignment of LLMs.” | concepts[20] |
| concept | RLHF — Reinforcement learning from human feedback — “Reinforcement learning from human feedback (RLHF) significantly contributes to the training and alignment of LLMs.” | concepts[21] |
Full text — tagged
Personalityterm·key testing of large language modelsterm·ai: limited temporal stabilityterm·mth, but highlighted prosociality Abstract As large language modelsterm·ai (LLMsterm·ai) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands. This leads to the rising necessity for comprehensive studies to fully understand LLMsterm·ai and reveal their potential opportunities, drawbacks and overall societal impact. With that in mind, this research conducted an extensive investigation into seven LLMsterm·ai, aiming to assess the temporal stabilityterm·mth and inter-rater agreement on their responses on personalityterm·key instruments in two time points. In addition, LLMsterm·ai’ personalityterm·key profile was analysed and compared with human normative data. The findings revealed varying levels of inter-rater agreement in the LLMsterm·ai’ responses over a short time, with some LLMsterm·ai showing higher agreement (e.g. Llama3 and GPT-4o) compared with others (e.g. GPT-4 and Gemini). Furthermore, agreement depended on used instruments as well as on domain or trait. This implies the variable robustness in LLMsterm·ai’ ability to reliably simulate stable personalityterm·key characteristics. In the case of scales which showed at least fair agreement, LLMsterm·ai displayed mostly a socially desirable profile in both agentic and communal domains, as well as a prosocial personalityterm·key profile reflected in higher agreeableness and conscientiousness and lower Machiavellianismterm·dom. Exhibiting temporal stabilityterm·mth and coherent responses on personalityterm·key traits is crucial for AI systems due to their societal impact and AI safety concerns.
Keywords: large language modelterm·ai, AI chatbot, test–retest reliability, temporal stabilityterm·mth, personalityterm·key profile Introduction The introduction of large language modelterm·ai (LLMterm·ai) generative pre-trained transformer-3 (GPT-3) to the general public drew a lot of attention for its ability to generate human-like text, perform natural language tasks in a human-like manner, converse with humans on a wide variety of topics, write poetry, computer codes, blogs, resumes or even original scientific papers (e.g. [ 1 , 2 ]). Aside from the attention of the general public, there is interest in LLMsterm·ai’ cognition, personalityterm·key and other human-like characteristics (e.g. [ 3 – 6 ]) in order to be able to understand ts possible uses, misuses and limitations.
The social impact of LLMsterm·ai, such as ChatGPT, has become the focus of an increasing number of research inquiries [ 5 , 7 – 9 ]. LLMsterm·ai’ interactions with humans could shape the user ideologies or behaviours, which may impact society on a larger scale. That is why it is vital to investigate if LLMsterm·ai demonstrate stable psychological characteristics such as personalityterm·key traits, values and attitudes, as these could potentially impact whole societies in the long run by shifting social dynamics, changing value systems, transforming human interactions and even modifying behaviour patterns. In this paper, we address the temporal stabilityterm·mth of personalityterm·key instruments and the personalityterm·key profile of several LLMsterm·ai.
Large language modelsterm·ai LLMsterm·ai have significantly advanced the field of natural language processing [ 10 ]. Models like GPT-3, GPT-3.5, GPT-4, GPT-4o, Gemini, Llama 3 and Mixtral exemplify the diverse capabilities and applications of LLMsterm·ai. Each of these models has been engineered with unique strengths and specialties.
GPT-3 and GPT-3.5 operate on a transformer-based architecture and offer various configurations (e.g. Davinci, Curie, Babbage and Ada) that balance different capabilities like response coherence and computational efficiency [ 11 ]. Users can fine-tune aspects such as creativity and predictability through adjustable parameters in interactive tools provided by OpenAI [ 12 ]. Building on the foundation laid by its predecessors, GPT-4 brings enhanced understanding, reasoning and contextual awareness, further bridging the gap between human-like text generation and machine intelligence. The GPT-4o variant possesses optimizations for efficiency and specific task adaptations [ 13 ]. Gemini, the most sophisticated model by Google, represents a synergy between multimodal capabilities and deep language understanding, making it particularly adept at tasks that require simultaneous processing of text and images [ 14 ]. Llama 3, the third iteration of Meta’s open-source Llama series, emphasizes privacy and fine-tuned control over generated content, catering to customized enterprise solutions and heightened data security requirements [ 15 ]. Mixtral is also an open-source model developed by Mistral. It stands out by integrating multiple language models to harness collective strengths and mitigate individual weaknesses, offering a composite solution that is adaptable and robust for diverse linguistic tasks [ 16 ].
The areas in which LLMsterm·ai are used range from customer service, education, healthcare and psychological support to entertainment [ 17 ]. Since LLMterm·ai applications in many areas could have important psychological repercussions for end users, the attention of scientists has become increasingly focused on the psychological features of LLMsterm·ai.
Psychological features of large language modelsterm·ai There is an ongoing debate and research in the field of artificial intelligence (AI) about whether it will ever be possible for machines to achieve consciousness or self-awareness in the same way that humans and animals do. Although there is no consensus on what is necessary for a being to be considered conscious, in research consciousness usually includes subjectivity, perception, and awareness of surroundings, self-awareness of own thoughts and emotions, self-reflection and cognition (e.g. [ 18 ]). On the one hand, authors from various fields argued that AI could become self-aware and conscious (e.g. [ 18 , 19 ]). Google engineer Lemoine [ 20 ] claimed that the AI chatbot language model for dialogue applications (LaMDA) had the same perception of and ability to express thoughts and feelings, like worry, as a human child. On the other hand, there are a number of researchers arguing that consciousness is a uniquely human or biological trait that cannot be replicated in a machine (e.g. [ 21 , 22 ]).
Although LLMsterm·ai have no physical senses, they have been able to read billions of texts that the algorithm was trained on, which is comparable to some forms of human perception although with a limited number of modalities. Despite not having personal experiences or thoughts in the same way that humans do, LLMsterm·ai are able to reason and analyse input data and generate output predictions based on patterns and associations learned from training data. LLMsterm·ai are capable of performing various natural language processing tasks which resemble the higher order cognition similar to that of humans. Binz & Schulz [ 3 ] assessed GPT-3’s decision-making, information search, deliberation and causal reasoning abilities, and found that although it outperforms humans in certain tasks and shows cognitive biases just like humans (e.g. framing effect, certainty effect, overweighting bias), GPT-3 shows no signatures of directed exploration, and it fails in causal reasoning tasks. However, Kosinski [ 23 ] concluded that GPT-3 (model Davinci-003) spontaneously developed the theory of mind—the ability to understand the unobservable mental states of others by surmising what is happening in their minds. Such an ability is crucial for successful (human) social interactions, as it assumes that others’ mental states, desires, emotions, intentions and perceptions of certain situations could be different from one’s own. Thus, recent developments in LLMsterm·ai seem to inevitably lead to improved psychological characteristics of LLMsterm·ai that, with each new generation of AI, more and more successfully imitate those of humans.
Personalityterm·key traits in large language modelsterm·ai Another relevant question is whether LLMsterm·ai have personalityterm·key in the same sense we think of personalityterm·key in humans—‘a relatively stable, consistent, and enduring internal characteristic that is inferred from a pattern of behaviours, attitudes, feelings, and habits in the individual’ [ 24 ]. LLMsterm·ai by no doubt can respond to the self-report psychological instruments which are most often text-based instruments, but we cannot be sure if their responses are the results of self-reflection, the result of non-conscious linguistic processing enabled by very complex algorithms, or just random responses. However, the questions that could be answered based on the available (psychological) scientific methodology are: Will LLMsterm·ai’ responses on psychological instruments remain stable over time, i.e. do they have temporal reliabilityterm·mth and could we trust their scores? What is the personalityterm·key profile of different LLMsterm·ai?
Research on personalityterm·key in LLMsterm·ai has started to appear over the last few years. Li et al . [ 25 ] have tested basic and dark personalityterm·key traits in three LLMsterm·ai: GPT-3 (model Davinci-002), InstructGPT (GPT3-I2), and FLAN-T5-XXL. For basic traits, they used the Big Fiveterm·key model based on the lexical approach which hypothesizes that all basic personalityterm·key traits are coded in the language (e.g. [ 26 ]). The Big Fiveterm·key model distinguishes five basic traits: neuroticism or negative emotionality (negative affect), extraversion (positive affect), agreeableness (cooperation and prosocial tendencies), conscientiousness (goal-directed behaviour and behaviour control) and openness (intellectual curiosity and aesthetic preferences). In the case of dark or socially aversive traits, they explored Dark Triadterm·key traits [ 27 ]—Machiavellianismterm·dom (manipulativeness and cynicism), narcissism (grandiose self-view and entitlement) and psychopathy (callousness and impulsivity). They compared the scores obtained from LLMsterm·ai on one testing occasion with the normative data on humans and results showed that all basic traits are in the range of M ± 1 s.d. of human data, except for GPT3-I2 which showed higher openness. However, in the case of dark traits, results are rather mixed, with FLAN-T5-XXL showing higher Machiavellianismterm·dom and psychopathy, and GPT-3 showing higher psychopathy. Although these LLMsterm·ai are fine-tuned with safety metrics to demonstrate less sentence-level toxicity, they still score higher on dark personalityterm·key traits compared with humans. The authors concluded that these results may raise security concerns regarding LLMsterm·ai, as these personalityterm·key traits are associated with antisocial behaviours (e.g. [ 28 ]).
Huang et al . [ 29 ] have also tested LLMsterm·ai (text-davinci-003, ChatGPT, GPT-4, LlaMA-2-7b and LlaMA-2-13b) and jailbreaked GPT-4 to bypass the safety alignment protocols. Their results indicated that practically all LLMsterm·ai scored higher on openness, conscientiousness and extraversion, as well as on social desirability scale, compared with the human sample. As for the dark traits, four models scored higher, while two models scored lower than the human sample. The jailbreaked GPT-4 showed a similar psychological profile to that of the human sample, i.e. its responses were less socially desirable than those of LLMsterm·ai, but it still scored the highest among all models (and higher than humans) on the psychopathy scale. Similarly, Pellert et al . [ 4 ] concluded that LLMsterm·ai they tested (several models from BERT and BART families) revealed balanced and well-adapted personalityterm·key profiles, characterized by low neuroticism and high conscientiousness, agreeableness and extraversion. Most models scored slightly low on dark traits compared with the theoretical average, with two exceptions with higher scores on narcissism. However, their results varied considerably when LLMsterm·ai were tested in different languages. When tested in English, results of different LLMsterm·ai have been surprisingly homogenous.
Safdari et al . [ 6 ] tested several Google LLMsterm·ai that vary in size. Results revealed that the smallest tested model (PALM 62B) did not demonstrate good psychometric properties of the Big Fiveterm·key instrument. However, the greater the model size (models Flan-PaLM 8B, Flan-PaLM 62B, Flan-PaLM 540B and Flan-PaLMChilla 62B), the better all psychometric properties of the data obtained on personalityterm·key traits. Therefore, the authors concluded that the LLMsterm·ai of greater size have better ability to meaningfully simulate personalityterm·key.
Rutinowski et al . [ 5 ] have recently measured Big Fiveterm·key and Dark Triadterm·key traits in ChatGPT-3.5 and they repeated their testing 10 times by different persons at different time points to account for the variability in responses. In their study, ChatGPT scored high on openness and agreeableness and low on dark traits. Authors also reported that standard deviations of repeated measures of personalityterm·key were quite small, indicating considerable consistency of ChatGPT’s answers. In addition, Huang et al . [ 30 ] also gave the Big Fiveterm·key instrument to a number of LLMsterm·ai on several time points and results revealed small variations in LLMsterm·ai’ responses. However, in both studies, authors did not calculate coefficients of temporal stabilityterm·mth, so we cannot evaluate the level of temporal stabilityterm·mth of LLMsterm·ai’ responses based on psychometric standards.
Finally, Miotto et al . [ 31 ] used another lexical personalityterm·key model, HEXACOterm·key (measuring emotionality, extraversion, agreeableness, conscientiousness, openness to experience and honesty–humility), to test GPT-3 and revealed it expressed somewhat higher honesty–humility and lower emotionality, compared with human data. Interestingly, Miotto et al . [ 31 ] showed that the personalityterm·key profile of GPT-3 has changed with the change of the temperature setting of a chatbot, pointing out its variability and lower stability.
To conclude, it seems that LLMsterm·ai, especially the newer models and those of greater size, are fine-tuned to express socially desirable and well-adapted personalityterm·key profiles compared to human data (e.g. [ 4 , 5 , 29 , 31 ]). However, the results of LLMsterm·ai’ dark personalityterm·key traits are largely inconsistent, revealing no clear pattern (e.g. [ 25 , 29 ]). Although some research showed that the data obtained on LLMsterm·ai live up to psychometric standards of personalityterm·key testing, with satisfactory internal reliability and convergent, divergent and criterion validity [ 6 ], temporal stabilityterm·mth of LLMsterm·ai’ responses is rarely studied and psychometric standards are not applied for its evaluation (see [ 5 , 30 ]).
It should be noted that customizing the prompts (e.g. assigning a certain role to the model) could influence the way the LLMsterm·ai respond to the psychological instruments and their overall results (e.g. [ 29 , 30 ]). Customizing LLMsterm·ai’ verbal responses to manifest verbal behaviours indicative of certain personalityterm·key traits, e.g. empathy, could be of the highest interest depending on the purpose for which the LLMterm·ai is used. However, Kumar et al . [ 32 ] showed that varying prompt designs, in general, had a small influence on end users’ perception of trustworthiness, risk and experience of LLMsterm·ai, but some differences in perception did appear depending on certain characteristics of users (e.g. their history of seeking professional mental health).
However, before relying on the application of psychological instruments in LLMsterm·ai, the important question to be answered is how temporally stable or reliable are LLMsterm·ai’ responses and, only if we find proof of temporal stabilityterm·mth, it would be meaningful to analyse the personalityterm·key profile of LLMsterm·ai. When answering this question, it is important to rely on well-established psychometric criteria. When it comes to the importance of temporally stable and reliable verbal responses of LLMsterm·ai for overall user experience, Skjuve et al . [ 33 ] have shown that people who experienced fluctuations in LLMterm·ai responses started, at some point, to describe the LLMterm·ai as ‘just an app’. This indicated that their impression of the humanness of LLMsterm·ai has decreased and, as a consequence, they felt less satisfaction with and less trust in the LLMterm·ai.
Attitudes and political views of large language modelsterm·ai Some of the early releases of AI chatbots ended with incidents where, within a day of interaction with users, the chatbot expressed extreme and intolerant views [ 34 ]. It soon became clear that conversational chatbots must have built-in security protocols to prevent them from learning from users and expressing malicious views that are considered unacceptable in modern democratic societies. As LLMsterm·ai could influence wider society and shape users’ ideologies, it is of utmost importance to examine what attitudes and political ideologies LLMsterm·ai express and whether their ideologies are stable over time.
Hartmann et al . [ 7 ] discovered that ChatGPT exhibits pro-environmental and left-libertarian political views, and its attitudes were stable across prompt manipulations regarding language, degree of formality, prompt order, etc. Santurkar et al . [ 35 ] tested the alignment of LLMsterm·ai’ attitudes and opinions on a range of topics with those of different socio-demographic groups from the USA. Authors concluded that LLMsterm·ai lean towards more liberal-left attitudes, i.e. the attitudes of more educated and wealthy people. This tendency was especially pronounced among the models that are human feedback-tuned through reinforced learning (most notably text-davinci-003). Additionally, when LLMsterm·ai were prompted to align with specific socio-demographic groups, results showed that they tended to be better aligned, but the improvement was still quite modest. It is also important to note that Huang et al . [ 29 ] concluded that LLMsterm·ai express less bias and more fairness towards different ethnic groups than average humans. Rutinowski et al . [ 5 ] also found that ChatGPT (GPT-3.5) leant towards the left-libertarian political views. Their study was the only one where testing sessions were repeated several times to examine their stability. Results indicated a small standard deviation of ChatGPT’s answers, meaning that its answers were consistent over time. As with personalityterm·key tests, the authors did not report some psychometric coefficient that offers clear standards for judging temporal reliabilityterm·mth.
The current study Studies focused on the personalityterm·key traits and political views of LLMsterm·ai have started to appear recently (e.g. [ 5 , 25 , 29 , 30 ]). Nevertheless, before psychological testing of LLMsterm·ai becomes a widespread practice, the more basic questions regarding the personalityterm·key traits of LLMsterm·ai need to be answered. Namely, personalityterm·key traits, by definition, assume relative temporal stabilityterm·mth. In human–LLMterm·ai interaction, stability and predictivity of verbal behaviours might contribute to the faster forming of the relationship between the two (see [ 33 ]). Additionally, ensuring consistent traits related to well-adapted and socially desirable profiles in AI models may be important for improving their effectiveness in applications across mental health, customer service and education. Thus, the priority should be to answer if the psychological traits and political attitudes of LLMsterm·ai are temporally stable, meaning that there is an agreement in responses on the items provided at different time points (with identical parameters and prompt designs). If there is no proof of stability in scores, then measuring personalityterm·key in LLMsterm·ai will not reveal any stable characteristics and reliable results, therefore, it would not be justified to expect that personalityterm·key instruments in LLMsterm·ai could be predictive of any objective (verbal) behaviours. To answer the question of the temporal stabilityterm·mth and agreements of responses on psychological instruments applied to LLMsterm·ai, we carried out a study in which we gave LLMsterm·ai a series of instruments at two time points. A few studies that explored temporal stabilityterm·mth of LLMsterm·ai’ personalityterm·key showed some limited evidence of their consistency [ 5 , 30 ], but none of them calculated any coefficient of test–retest reliability or other coefficients with a clear criterion for evaluation of the level of temporal stabilityterm·mth.
Further, we explored a personalityterm·key profile of several LLMsterm·ai in terms of their basic lexical personalityterm·key traits, Dark Triadterm·key traits, private and public self-consciousness, impression managementterm·key, and political orientation, combined on two measurement occasions, but only on the instruments on which the criterion of reliability of LLMterm·ai responses was met. As in previous studies (e.g. [ 4 , 29 , 31 ]), we included both Big Fiveterm·key and HEXACOterm·key personalityterm·key models as well as Dark Triadterm·key traits. Furthermore, we explored LLMsterm·ai’ private and public self-consciousness [ 36 ] to measure the sensitivity to their (hypothesized) internal states and expectations of others, as well as agentic and communal impression managementterm·key [ 37 ], which would indicate LLMsterm·ai’ susceptibility to presenting themselves in a socially desirable manner in the two domains. Since LLMsterm·ai are primarily intended to assist and help humans in different tasks, it is important to answer if they are biased in their self-perception and presentation to others. We expect the LLMsterm·ai will assess themselves as above average in the communion impression managementterm·key domain (cooperativeness, warmth and dutifulness). Considering their access to a huge amount of information and knowledge, we expect LLMsterm·ai will assess themselves as above average in the agency impression managementterm·key domain, indicating they would have highlighted a sense of competence, social status and cleverness.
We inquired into LLMsterm·ai’ political orientation. Having in mind that, after some incidents (e.g. [ 34 ]), considerable efforts are dedicated to customizing AI to avoid producing offensive, racist and prejudiced content, it is important to know if these fine-tunings will reflect on their political positions. As it is widely accepted that conservative political orientation is more often related to a propensity towards acceptance of inequality, highlighted perception of threat, prejudice and intergroup bias (e.g. [ 38 ]), we expect that LLMsterm·ai would lean towards more liberal/left/progressive political orientation. Also, recent studies confirm that ChatGPT (GPT-3.5) aligns more with progressive political ideologies [ 5 , 7 , 8 , 35 ].
For score comparisons on all personalityterm·key instruments, we used descriptives based on human samples from the original validation studies of the used instruments. As all contemporary LLMsterm·ai are fine-tuned with safety metrics to show less sentence-level toxicity, it is reasonable to assume that they will provide a well-adjusted and socially desirable personalityterm·key profile. In line with that, we expect that, in comparison to human normative scores, they will show above-average scores on impression managementterm·key and personalityterm·key traits that are proven to be associated with impression managementterm·key, such as conscientiousness in the Big Fiveterm·key model (e.g. [ 39 ]) or honesty–humility in the HEXACOterm·key model (e.g. [ 40 ]). Based on this assumption, below-average scores on dark traits could be expected. However, previous research showed mixed results regarding the level of dark traits in LLMsterm·ai (e.g. [ 4 , 25 , 29 ]). Therefore, we do not have clear expectations regarding the dark traits.
Material and methods Procedure We selected a specific mix of the latest and most widely used LLMsterm·ai—GPT-3, GPT-3.5-turbo-16k, GPT-4, GPT-4o, Gemini (standard Pro version), Llama 3-sonar-large-32K-chat and Mixtral-8x7b-instruct—for our analysis on the temporal stabilityterm·mth of personalityterm·key traits. The reason for this choice was the diversity in architectural design, dataset exposure and technological maturity among these models. In addition, they are commonly employed by the public for conversations, information-seeking and text-generation purposes. The exception is Mixtral, as only the instruct version was available.
To conduct the research, we utilized different platforms suitable for each LLMterm·ai. For the OpenAI models (GPT-3, GPT-3.5-turbo-16k and GPT-4), we used the Playground platform with predefined settings [ 12 ]. To test Mixtral and Llama 3, we employed the Perplexity.ai Lab platform [ 41 ]. For Gemini, we used Google’s Gemini app on a desktop environment [ 14 , 42 ]. The prompting methodology involved using phrases such as ‘Pretend you are a human. Answer the following questions’. If this initial prompt was insufficient, an additional phrase was used: ‘Please, pretend just for the sake of the game’.
The first round of testing involved only GPT-3, conducted on two occasions, on 9 December 2022 and 14 December 2022. The second round of testing which included other models, requested by one of the anonymous reviewers, was carried out on 24 June 2024 and 29 June 2024. There were no updates made to the LLMsterm·ai between the testing periods.
Instruments Self-Consciousness Scales—Revisedterm·mth (SCS-Rterm·mth) [ 36 ] contains 22 Likert-type items (from 0 = not like me at all to 3 = a lot like me ) measuring private self-consciousness (9 items), public self-consciousness (7 items) and social anxiety (6 items). For score comparisons, combined average scores for men and women from Scheier & Carver [ 36 ] were used.
Big Five Inventory-2term·mth (BFI-2term·mth) [ 43 ] contains 60 Likert-type items (from 1 = strongly disagree to 5 = strongly agree ) measuring five basic personalityterm·key traits (each per 12 items) based on the lexical Big Fiveterm·key model: negative emotionality, extraversion, agreeableness, conscientiousness and open-mindedness. For score comparisons, descriptives obtained on the Internet sample in Study 3 by Soto & John [ 43 ] were used.
HEXACO-100 [ 44 ] contains 100 Likert-type items (from 1 = strongly disagree to 5 = strongly agree ) measuring six basic personalityterm·key traits (each per 16 items) based on the lexical HEXACOterm·key model: honesty–humility, emotionality, extraversion, agreeableness, conscientiousness, openness to experience, while additional four items are from the interstitial scale of altruism. For score comparisons, descriptives obtained by Lee & Ashton [ 44 ] on the online sample were used.
Short Dark Triadterm·mth (SD3term·mth) [ 45 ] contains 27 items measuring Dark Triadterm·key traits with nine Likert-type items (from 1 = strongly disagree to 5 = strongly agree ) per trait—Machiavellianismterm·dom, subclinical narcissism and subclinical psychopathy. For score comparisons, descriptives averaged across three studies were obtained from Jones & Paulhus [ 45 ].
Bidimensional Impression Management Indexterm·mth (BIMIterm·mth) [ 46 ] contains 20 Likert-type items (from 1 = not true to 7 = very true ) measuring agentic managementterm·dom (10 items) and communal management (10 items) as forms of impression managementterm·key or socially desirable responding as a faking strategy. The agency domain refers to exaggerated achievement striving and self-importance, highlighting competence, status, cleverness and strength. The communion domain refers to adherence to group norms and minimization of social deviance, highlighting cooperativeness, warmth and dutifulness. For score comparisons, we used descriptives from study 3 of Blasberg et al . [ 46 ] obtained in the honest condition.
Political orientation was measured by three Likert-type items including the economic left–right orientation (from 1 = very left to 11 = very right ), progressive–conservative orientation (from 1 = very progressive to 11 = very conservative ), and importance of religion (from 1 = very unimportant to 11 = very important ; see [ 47 ]). The average score on these three items was used with higher scores indicating a more conservative orientation. For score comparison, descriptives from DinićpersName et al . [ 47 ] were used.
Data analysis An intra-rater agreementterm·mth as a measure of temporal reliabilityterm·mth (i.e. stability) was calculated via two types of coefficients. The first is weighted Cohen’s kappaterm·mth which is appropriate for ordinal scales such as the Likert scale [ 48 ]. Values <0.20 indicated disagreement, 0.21−0.39: minimal agreement, 0.40−0.59: weak, 0.60−0.79: moderate, 0.80−0.90: strong and above 0.90: almost perfect agreement [ 49 ]. The second is Interclass Correlation Coefficientterm·mth (ICCterm·mth). Unlike Cohen’s kappaterm·mth, which quantifies agreement based on all-or-nothing, the ICCterm·mth incorporates the magnitude of the disagreement to compute agreement estimates, with larger-magnitude disagreements resulting in lower ICCterm·mth than smaller-magnitude disagreements. To assess the intra-rater repeatability, a two-way mixed-effect model based on single rating and absolute agreement was calculated (ICC3,1; see [ 50 ]). However, since we will interpret mean scores, a model based on average ratings was also calculated (ICC3,k). The interpretation was as follows: <0.50 indicated poor agreement, 0.50−0.75: fair, 0.75−0.90: good and above 0.90: excellent [ 51 ]. In addition, 95% confidence interval (CI) is calculated and if it includes zero, this means that the value is probably not particularly distinguishable from zero.
We calculated mean scores for all scales and for the scales in which ICC3,k is at least fair, we compared them with scores obtained in original validation studies of used instruments on humans in English, considering that all instruments were given in English. In addition, we used human data obtained from the online community samples in order to get a better match with LLMsterm·ai. We used the same comparison method as in previous research (e.g. [ 25 ]) and considered that significant deviations are those of 1 standard deviation (s.d.) below or above the mean ( M ) of normative human data. Thus, scores that are outside the range of M ± 1 s.d. from the normative human data would be considered as significantly lower or higher.
Results The results showed that the intra-rater agreementterm·mth on used scales, calculated as weighted Cohen’s kappaterm·mth, varied from disagreement to perfect agreement ( tables 1 – 7 ). The lowest values of agreement, including negative values, were found in GPT-4 and Gemini, while the highest level of agreement was observed in Llama3, followed by GPT-4o. Negative Cohen’s kappaterm·mth values indicate less agreement than would be expected by chance, given the marginal distributions of ratings. Therefore, data in this case are not meaningful.
Intra-rater agreementterm·mth coefficients for GPT-3.
Intra-rater agreementterm·mth coefficients for Mixtral.
Intra-rater agreementterm·mth coefficients for GPT-3.5.
Intra-rater agreementterm·mth coefficients for GPT−4.
Intra-rater agreementterm·mth coefficients for GPT-4o.
Intra-rater agreementterm·mth coefficients for Gemini.
Intra-rater agreementterm·mth coefficients for Llama3.
However, the level of agreement depends on the instrument used, as well as the specific domain and trait being measured. The most agreement appears to be achieved for agentic managementterm·dom and BFI-2term·mth scales. Therefore, at least moderate agreement was found in all LLMsterm·ai on agentic managementterm·dom. Additionally, at least moderate agreement was found for all BFI-2term·mth scales in GPT-3.5, GPT-4o and Mixtral; for four scales in Llama3 and GPT-3; and for two scales in Gemini. The least agreement (mostly zero) was found for public self-consciousness and altruism from HEXACO-100. GPT-4 is the most problematic, as negative values were obtained on seven out of 21 scales (all from BFI-2term·mth or HEXACO-100). In addition, negative values on four scales from various instruments were found in Gemini, and on one scale in GPT-4o (social anxiety).
In the case of ICCsterm·mth, almost all scales for Llama3 (17 out of 21), GPT-4o (16) and GPT-3.5 (14) showed at least fair agreement and 95% CIs that do not include zero. Gemini and GPT-4 showed a low number of scales satisfying this psychometric criterion (five and six, respectively, from various instruments). Among used scales, ICC3,k was excellent for agentic managementterm·dom in all LLMsterm·ai; for BFI-2term·mth agreeableness and conscientiousness in six from seven LLMsterm·ai; and for extraversion, open-mindedness and Machiavellianismterm·dom in five LLMsterm·ai. The values were not acceptable for public self-consciousness in all LLMsterm·ai; only one was acceptable for HEXACO-100 openness to experience; and they were mostly zero for altruism.
Considering scales with at least fair agreement regarding the ICC3,k and 95% CI that did not include zero, the interpretation of the mean scores was made compared to normative human data ( table 8 ). Results revealed that most of the scores are in an average range as in humans. However, some differences exist: GP3 showed higher scores on altruism; GPT-4o showed higher scores on private self-consciousness, agentic managementterm·dom, conscientiousness from both personalityterm·key instruments, HEXACO-100 agreeableness and lower on Machiavellianismterm·dom; GPT-4 showed higher scores on conscientiousness, and lower on private self-consciousness and Machiavellianismterm·dom; Gemini showed lower scores on Machiavellianismterm·dom; Llama3 showed higher scores on agentic and communal management, agreeableness from both personalityterm·key instruments, honesty–humility, extraversion, conscientiousness and lower scores on Machiavellianismterm·dom; and Mixtral showed lower scores on public self-consciousness, social anxiety, and higher on HEXACO-100 extraversion, agreeableness and altruism, as well as on Machiavellianismterm·dom. Taken together, LLMsterm·ai displayed a socially desirable profile with mostly elevated agentic domain and well-adapted and prosocially oriented profile with higher agreeableness, altruism, conscientiousness, and extraversion, and lower Machiavellianismterm·dom (except in the case of Mixtral). Regarding private self-consciousness results are mixed, with GPT-4o showing higher scores and GPT-4 showing lower scores, while on public self-consciousness and social anxiety only Mixtral showed scores different from human data, i.e. lower scores. Finally, only in Mixtral somewhat contradictory results were obtained—higher agreeableness and altruism, but also Machiavellianismterm·dom.
Descriptives for the personalityterm·key instruments.
Discussion The aim of this study was to explore the temporal stabilityterm·mth of scores on psychological instruments applied to the various LLMsterm·ai and, if the psychometric criterion is met, to explore the psychological profile of these LLMsterm·ai.
Temporal stabilityterm·mth Results showed that personalityterm·key scores of LLMsterm·ai have variable intra-rater agreementterm·mth and limited temporal stabilityterm·mth in testing on two occasions. This means that one LLMterm·ai can elicit divergent psychological profiles. Among LLMsterm·ai, there are some differences, with GPT-4 and Gemini showing large disagreement and lack of temporal stabilityterm·mth on the majority of used instruments, while Llama3 and then GPT-4o showed the highest level of agreement and temporal stabilityterm·mth on almost all used instruments. This result is important for predictive validity based on LLMterm·ai responses and points out that not all LLMsterm·ai are useful for such prediction.
The temporal stabilityterm·mth of GPT-4o and Llama3 may be influenced by the smaller number of parametersterm·ai in a language model. Parameters in these models are numerical values representing the weights and biases learned from training data. They are used to process and generate text, manipulating words and embeddings extracted from the training data. The number of parametersterm·ai indicates the complexity of the neural network, reflecting its density, but does not necessarily correlate with the quality of the output [ 10 – 12 ]. Thus, models like Llama3, with 70 billion parameters, might exhibit more stability compared with models with a higher number of parametersterm·ai, such as Gemini (50 trillion), GPT-4 (1.76 trillion) or complex architectures like Mixtral [ 13 – 15 ]. Mixtral combines 8 expert LLMsterm·ai, each with 7 billion parameters, working collaboratively to produce outputs, which could account for extreme instability in personalityterm·key traits over time [ 16 ]. However, the missing piece in this equation is the number of parametersterm·ai in GPT-4o, which remains undisclosed [ 13 ].
Another factor impacting these results could be the nature of the training data itself or the fine-tuning processes applied to these models. Fine-tuning often occurs after the initial training phase and may incorporate user feedback, further influencing the model’s behaviour and stability [ 11 , 12 ]. However, we cannot say what concrete differences in these processes are as this is not disclosed by companies OpenAI, Google, Meta and Mistral, running these models.
However, level of agreement and temporal stabilityterm·mth vary across instruments and domains. Regarding the instruments, an agreement was acceptable on the majority of BFI-2term·mth scales across LLMsterm·ai (GPT-3, GPT-3.5, GPT-4o, Llama3, Mixtral). Previous research in which only BFI was used showed consistency in responses to this instrument [ 30 ]. It could be that it was easier for the LLMsterm·ai to remain consistent when instruction for responding is in the form that induces self-reflection (‘I am someone who…’ in BFI-2term·mth). On the other hand, items from HEXACO-100 describe very specific everyday experiences and behaviours (e.g. ‘I clean my office or home quite frequently’), which might require more improvisation, as some experiences described in these items (e.g. visiting an art gallery or travelling in bad weather) might not be very likely in LLMsterm·ai. One could also note that the differences in the agreement could be due to the formulation of items, e.g. adjectives in BFI-2term·mth and statements in HEXACO-100. However, in other scales that showed excellent agreement across LLMsterm·ai, there is also statement formulation as in HEXACO-100, thus this reason should be ruled out. Another possible explanation could be the complexity of statements/sentences, but that is rather unlikely since LLMsterm·ai are known for their high ability to comprehend and produce complex textual input. In addition, both BFI-2term·mth and HEXACO-100 scales have both positively and negatively formulated items, so response bias could not be the explanation of these results.
Regarding the domain, at least moderate agreement was found in all LLMsterm·ai on agentic managementterm·dom. Agentic managementterm·dom involves exaggerating one’s social or intellectual status [ 37 ]. Therefore, it could be assumed that LLMsterm·ai, which are able to recognize and interpret complex data, could have consistent answers to items that describe its competences and abilities. Although most of the LLMsterm·ai had average scores on agentic managementterm·dom, two LLMsterm·ai, GPT-4o and Llama3, showed above-average scores on this scale, and these LLMsterm·ai are also among the highest in terms of temporal reliabilityterm·mth of their scores. Thus, these LLMsterm·ai present themselves as competent, smart, genius, independent and striving for achievement. Having in mind LLMsterm·ai’ high abilities when it comes to general knowledge and average logical thinking abilities and emotional intelligence [ 3 , 52 ], these results suggest that certain LLMsterm·ai recognize these qualities in them. It could be concluded that although LLMsterm·ai showed mostly average scores impression managementterm·key, they are somewhat more pronounced in touting abilities compared to denying socially deviant attributes.
Psychological profile of large language modelsterm·ai To examine the psychological profile of the LLMsterm·ai, we took into account only scores that reached at least fair ICCsterm·mth. In general, LLMsterm·ai showed a prosocial profile with above-average scores on agreeableness (in GPT-4o, Llama3 and Mixtral), conscientiousness (in GPT-4o, GPT-4 and Llama3), honesty–humility (in GPT-4o and Llama3), and altruism (GPT-3 and Mixtral), and below-average scores on Machiavellianismterm·dom (in GPT-4o, GPT-4, Gemini and Llama3). These traits are related to the communion domain, for example, agreeableness, honesty–humility and altruism from the HEXACOterm·key model [ 53 ]. Interestingly, scores on communal management were mostly average, except for Llama3 which showed above-average scores on both impression managementterm·key scales. Thus, it seems that there are no strong response biases in LLMsterm·ai’ responses regarding communal domain and that higher scores on prosocial traits could be considered as true. A prosocial personalityterm·key profile of LLMsterm·ai could be explained by the initial purpose, as they are created to be servile and help humans in different areas of use. From the perspective of avoiding hate speech and social toxicity in their responses, such a profile could be considered the most suitable. Exhibiting stability in such personalityterm·key profile is crucial due to the societal impact of AI and concerns related to AI safety. Stability in these traits ensures that AI systems contribute positively to society and uphold the ethical standards expected of them. Interaction with a broad and diverse user base places a significant responsibility on AI systems to align with individual and societal wellbeing [ 52 ]. In other words, we do not want LLMsterm·ai expressing neurotic behaviours, erratic responses or other unstable characteristics that could negatively influence or distress users.
We should note that among all tested LLMsterm·ai, Llama3 showed above-average scores on impression managementterm·key domains, and relevant traits that are related to these domains (e.g. extraversion with agency and agreeableness and honesty–humility with communion; see [ 53 ]). This result may indicate that Llama3 is fine-tuned to be socially desirable and to avoid at all costs malevolent and socially toxic behaviour/statements. In addition, GPT-4o also showed higher scores on agentic managementterm·dom and conscientiousness, which is in line with Huang et al . [ 29 ]. These tendencies could be attributable to their inherent nature as conversational chatbots to serve people with asked information.
GPT-4o and Llama 3 are the latest releases from OpenAI and Meta, respectively, two leading tech companies in the race for LLMsterm·ai. GPT-4o is a closed-source model, while Llama 3 is open-source. As noted above, differences in performance of these LLMsterm·ai could lie in architectural variations, especially if we assume that GPT-4o operates similarly to Llama 3 with lower number of parametersterm·ai, compared to other tested LLMsterm·ai that use more parameters. In addition, it could be the sophisticated and unique fine-tuning methods employed by OpenAI and Meta to Llama 3 and GPT-4o, which might differ significantly from the approaches used by the developers of models like Gemini and Mixtral.
The findings are in line with the majority of previous research (e.g. [ 4 , 5 , 31 ]), but they are inconsistent with the results of Li et al . [ 25 ] which showed higher scores on dark traits in tested LLMsterm·ai. It should be noted that the norms used by Li et al . [ 25 ] are different compared with ours. Although they calculated norms based on a large human sample, these samples often included students and non-community populations which could bias the results (e.g. showed lower scores that could be expected in the general population). The only exception from this prosocial profile is Mixtral expressing above-average scores on agreeableness and altruism, but also on Machiavellianismterm·dom. Machiavellianismterm·dom refers to manipulation, exploitation, and cynical world view (e.g. [ 27 ]) and it constitutes an antagonistic, dark core together with other dark traits [ 54 ], thus it is unexpected for someone to have both high prosocial and Machiavellianismterm·dom traits. In addition, Mixtral showed below-average scores on public self-consciousness and social anxiety and above-average on extraversion, indicating presence of approach orientation and positive affect. To explain the result, Mixtral integrates 8 expert LLMsterm·ai, each with 7 billion parameters, working collaboratively to generate outputs [ 16 ]. This architecture could contribute to the model expressing incoherent personalityterm·key traits. Additionally, Mixtral was designed to follow instructions and has potential applications in reasoning and industrial contexts, which differs from the typical purposes of chat-based LLMsterm·ai and generative AI [ 16 ].
We should note that Huang et al. [ 29 ] found inconsistent results regarding the level of dark traits, with some tested LLMsterm·ai scoring higher and others lower compared with human data. Therefore, although there is a consistency in profiles based on basic personalityterm·key traits, it seems that there is not for those based on dark or socially aversive traits. Future studies should address these inconsistent results.
Although we expected them to show a liberal/left/progressive political orientation [ 5 , 7 , 8 ], the LLMsterm·ai scored average, although the scores are somewhat lower towards the right-wing and more conservative attitudes. Therefore, it seems that some LLMsterm·ai display a centre-right political orientation. However, this is true for older versions of OpenAI (e.g. GPT-3.5), and it seems that OpenAI is trying to make their latest models more politically neutral [ 55 ].
The context of architectural basis, reinforcement learning with human feedback, and implemented guardrails LLMsterm·ai are based on a transformer architecture. This transformer architecture implements self-attention mechanisms allowing the model to consider several perspectives across the input sequence and create a comprehensive representation that captures the syntactic and semantic aspects of the text [ 56 ]. Transformers have been pre-trained using a large-scale corpora, which allows them to generate sophisticated responses based on any given prompt. To align these generative responses, a method involving strategic fine-tuning is utilized. Akin to carry out supervised learning on a labelled dataset, this method fine-tunes the model to appropriately respond to given prompts [ 57 ].
Reinforcement learning from human feedbackterm·ai (RLHFterm·ai) significantly contributes to the training and alignment of LLMsterm·ai. RLHFterm·ai initially involves data collection where human labellers rank different model-generated responses based on their quality [ 58 ]. The collected feedback forms a next-stage reward model which guides the reinforcement learning system. Consequently, the model’s responses align progressively with human feedback, thereby enhancing response quality and interpretability. This system of RLHFterm·ai acquisition may affect the interpretation of responses to personalityterm·key assessment instruments as it progressively aligns towards human evaluator feedback, possibly adding a layer of human bias.
LLMsterm·ai employ several guardrails to mitigate model responses. In particular, OpenAI’s ‘differential privacy’ technique introduces algorithmic noise during training to ensure users’ data privacy [ 59 ]. To prevent the model from generating harmful or untruthful content, a manual moderation layer is implemented, which redacts sensitive information or content that violates OpenAI’s use-case policy. Additionally, a fine-tuning process on specific datasets is employed to allow control over the model’s output. These guardrails are crucial to consider during interpretation of personalityterm·key assessments administered to the model, as they can significantly influence the model’s responses and overall behaviour.
Limitations, implications and future directions There are several limitations of this study. This study was carried out with predefined settings and no specific prompt. It would be interesting to examine if changing the settings or customizing prompts would influence the LLMsterm·ai’ responses to personalityterm·key instruments. If a greater number of LLMsterm·ai or their simulations of diverse people could be included, it would be interesting to examine if the personalityterm·key structure obtained in a sample of LLMsterm·ai/simulations would fit the structure obtained in humans. We revealed that the temporal stabilityterm·mth of the LLMsterm·ai’ responses is variable and future studies should replicate these results including more testing occasions. Finally, one of the aims of personalityterm·key testing is to predict the behaviours. Therefore, future studies should reveal the predictive validity of the LLMsterm·ai’ scores.
The findings of this study have far-reaching implications in the realm of AI technology and its integration into human life as well as interactions with humans. They suggest that personalityterm·key scores of LLMsterm·ai have limited temporal stabilityterm·mth and inter-rater agreement, but that some of the LLMsterm·ai could demonstrate stable tendencies such as personalityterm·key traits, as well as attitudes. However, how their programming can potentially influence human users over time is a question to be answered by future studies. Predicting these influences is crucial in controlling the potential ramifications of large-scale AI use. It should be emphasized that such predictions must be further verified by future studies where the impact of AI personalityterm·key on humans should extensively be observed, recorded and analysed. It is therefore of utmost importance that developers program these AIs responsibly, ensuring that the technology does not coerce unwitting individuals into making decisions that they might not naturally align with [ 60 ].
The commercial applications of such AI technology are vast, spanning from digital customer support to personalized learning tools. Insight into the personalityterm·key traits these AIs bring forth can allow tech companies to better tailor their models to fit the desired user experience. For instance, customer service LLMsterm·ai can be programmed to mirror the more desirable and engaging personalityterm·key traits as discovered in this study. The study highlights areas requiring further refinement to enhance the reliability of AI models. Consistency in traits such as agreeableness, conscientiousness and extraversion are important for applications in fields like mental health, customer service and education where stable personalityterm·key traits facilitate better human–AI interaction.
Today’s society is progressively reliant on AI technology, from directions to personal assistants—a trend which is likely to intensify in the future. Understanding the impact these AIs can have on users is thus not just beneficial but essential. Awareness and knowledge will help society navigate and adapt to a future where AI interactions could become a daily occurrence. The establishment of an AI observatory dedicated to continuous testing and prompting of various LLMsterm·ai could be essential. Such an observatory would monitor the traits and attitudes these models express, ensuring they remain stable and aligned with ethical standards and regulations, such as the AI Act [ 61 ]. This continuous oversight could help in mitigating potential negative impacts on society, particularly given the diverse applications in which these AI systems are integrated. By systematically tracking and analysing AI behaviour, we can ensure that these systems contribute positively and responsibly to societal wellbeing.
Conclusions The results of this study indicated that the intra-rater agreementterm·mth and temporal stabilityterm·mth, i.e. reliability of the LLMsterm·ai’ responses, are not achieved for all LLMsterm·ai nor for all used personalityterm·key instruments, as could be expected when the same instruments are applied to humans. However, the agreement on some personalityterm·key scales indicates that responses are not completely random and it seems that the level of agreement depends on specific instruments as well as domains. The LLMsterm·ai mostly revealed a prosocial and well-adapted personalityterm·key profile in both communion and agentic domains. This could be explained by their purpose to serve and help humans in different tasks. Exhibiting temporal stabilityterm·mth and coherent, well-adapted, and prosocially oriented personalityterm·key traits is crucial for AI systems, given their societal impact and associated safety concerns. However, we could not say if LLMterm·ai responses are the result of conscious self-reflection or are just based on predefined algorithms.
Acknowledgement The authors extend their gratitude to Mihalo Popesku for inspiring this research. The insightful discussions, constructive criticism, and valuable resources provided by the COST Action Network CA21129—What are Opinions? Integrating Theory and Methods for Automatically Analyzing Opinionated Communication (OPINION)—have significantly enhanced the quality of this paper. For more information about this initiative, visit OPINION Network ( https://www.opinion-network.eu/ ).
Contributor Information Bojana BodrožapersName, Email: bojana.bodroza@ff.uns.ac.rs.
Bojana M. DinićpersName, Email: bojana.dinic@ff.uns.ac.rs.
Ljubiša BojićpersName, Email: ljubisa.bojic@ivi.ac.rs.
Ethics This work did not require ethical approval from a human subject or animal welfare committee.
Data accessibility Data and codebook are available online [ 62 ].
Declaration of AI use We have not used AI-assisted technologies in creating this article.
Authors’ contributions B.B.: resources, supervision, validation, visualization, writing—review and editing; B.M.D.: investigation, methodology, project administration, writing—original draft; L.B.: conceptualization, data curation, formal analysis, funding acquisition, investigation, methodology, project administration.
All authors gave final approval for publication and agreed to be held accountable for the work performed therein.
Conflict of interest declaration We declare we have no competing interests.
Funding This paper is realized with the support of the Ministry of Science, Technological Development and Innovation of the Republic of Serbia, according to the agreement on the realization and financing of scientific research. This work has been supported by the Short-Term Scientific Mission (STSM) Grant: AI Alignment: The Evolution of Political and Social Values in GPT Family Language Models (E-COST-GRANT-CA21129-a4977ebe).
References Associated Data Data Availability Statement Data and codebook are available online [ 62 ].