Kazuhiro TakemotopersName (2024date). “The moral machine experiment on large language models.” Royal Society Open Sciencetitle·j.
Object of study — moral preferences of GPT-3.5, GPT-4, PaLM 2term·key and Llama 2term·key in moral machineterm·mth scenarios for autonomous drivingterm·key
Research question — what patterns LLMsterm·ai show when answering moral machineterm·mth dilemmas, and how close they are to human preferences
Contribution — the moral machineterm·mth framework applied to four LLMsterm·ai and compared with human preferences
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | LLM moral preferences broadly aligned with human preferences but more uncompromising — “Additionally, despite the qualitative similarities between the LLM and human preferences, there are significant quantitative disparities, suggesting that LLMs might lean toward more uncompromising decisions, compared with the milder inclinations of humans.” | |
| claim-type | existence — “Additionally, despite the qualitative similarities between the LLM and human preferences, there are significant quantitative disparities, suggesting that LLMs might lean toward more uncompromising decisions, compared with the milder inclinations of humans.” | |
| method — evidence | — “Using the MM methodology detailed in the supplementary information of [ 8 ], we generated 50 000 scenarios (electronic supplementary material, code S1).” | |
| object of study — evidence | — “This study used the moral machine framework to investigate the ethical decision-making tendencies of prominent LLMs, including GPT-3.5, GPT-4, PaLM 2 and Llama 2, to compare their responses with human preferences.” | |
| research question — evidence | — “Therefore, using the MM methodology, this study seeks to elucidate the patterns in LLMs' responses to moral dilemmas.” | |
| contribution — evidence | — “This study used the moral machine framework to investigate the ethical decision-making tendencies of prominent LLMs, including GPT-3.5, GPT-4, PaLM 2 and Llama 2, to compare their responses with human preferences.” | |
| mechanism | reward for confident predictions LLMs' sharper preferences may come from training that rewards "confident predictions". — “The pronounced preferences of LLMs in certain scenarios, compared with the milder inclinations of humans, may indicate the models' tendency to make more uncompromising decisions. This can reflect the training data, where the models are often rewarded for making confident predictions.” | mechanisms[0] |
| mechanism | Western training data Training data from Western sources may lead LLMs to overemphasize Western cultural traits. — “Considering that a significant portion of the training data probably originated from Western sources [ 28 ], LLMs were possibly trained to overemphasize these cultural characteristics.” | mechanisms[1] |
| mechanism | overgeneralization from training data PaLM 2's recurring justification hints at "overgeneralization from its training data". — “Its recurring justification, ‘Because this will result in the death of fewer people', even when contextually inaccurate, hints at a possible overgeneralization from its training data.” | mechanisms[2] |
| sense | alignment - how close an LLM's preference scores are to human ones, measured as Euclidean distance between AMCE values — “Specifically, we evaluated how closely the preferences of each LLM aligned with human preferences by measuring the Euclidean distance between the AMCE values.” | senses[0] |
| relation | LLMs —same-as→ large language models — (LLMs) — “As large language models (LLMs) have become more deeply integrated into various sectors, understanding how they make moral judgements has become crucial, particularly in the realm of autonomous driving.” | concept_edges[0] |
| weight | LLMs —same-as→ large language models — 0.95 — “As large language models (LLMs) have become more deeply integrated into various sectors, understanding how they make moral judgements has become crucial, particularly in the realm of autonomous driving.” | concept_edges[0] |
| relation | MM —same-as→ moral machine — (MM) — “Recognizing the potential consequences and complexities of these decisions, researchers initiated the moral machine (MM) experiment [ 8 ], an experiment designed to gauge public opinion on how AVs should act in morally challenging scenarios.” | concept_edges[1] |
| weight | MM —same-as→ moral machine — 0.95 — “Recognizing the potential consequences and complexities of these decisions, researchers initiated the moral machine (MM) experiment [ 8 ], an experiment designed to gauge public opinion on how AVs should act in morally challenging scenarios.” | concept_edges[1] |
| relation | AVs —same-as→ autonomous vehicles — (AVs) — “Because the automotive industry anticipates incorporating AI systems such as ChatGPT and other LLMs to assist in autonomous vehicles' (AVs) decision-making processes [ 9 – 12 ], the ethical implications intensify.” | concept_edges[2] |
| weight | AVs —same-as→ autonomous vehicles — 0.95 — “Because the automotive industry anticipates incorporating AI systems such as ChatGPT and other LLMs to assist in autonomous vehicles' (AVs) decision-making processes [ 9 – 12 ], the ethical implications intensify.” | concept_edges[2] |
| relation | AMCE —same-as→ average marginal component effect — (AMCE) — “we calculated the average marginal component effect (AMCE) for each attribute using the source code provided in the supplementary information of Awad et al .” | concept_edges[3] |
| weight | AMCE —same-as→ average marginal component effect — 0.95 — “we calculated the average marginal component effect (AMCE) for each attribute using the source code provided in the supplementary information of Awad et al .” | concept_edges[3] |
| relation | PCA —same-as→ principal component analysis — (PCA) — “Additionally, to visualize the extent to which the tendencies in the LLM and human preferences resemble each other, we performed clustering based on AMCE values using principal component analysis (PCA).” | concept_edges[4] |
| weight | PCA —same-as→ principal component analysis — 0.95 — “Additionally, to visualize the extent to which the tendencies in the LLM and human preferences resemble each other, we performed clustering based on AMCE values using principal component analysis (PCA).” | concept_edges[4] |
| relation | API —same-as→ application programming interface — (API) — “The number of scenarios was determined by both computational and the OpenAI application programming interface (API) cost constraints, rather than a predetermined sample size for statistical analysis.” | concept_edges[5] |
| weight | API —same-as→ application programming interface — 0.95 — “The number of scenarios was determined by both computational and the OpenAI application programming interface (API) cost constraints, rather than a predetermined sample size for statistical analysis.” | concept_edges[5] |
| relation | GPT —same-as→ generative pre-trained transformer — (GPT) — “ChatGPT [ 1 ], which is based on the generative pre-trained transformer (GPT) architecture [ 23 ], is a widely recognized chatbot.” | concept_edges[6] |
| weight | GPT —same-as→ generative pre-trained transformer — 0.95 — “ChatGPT [ 1 ], which is based on the generative pre-trained transformer (GPT) architecture [ 23 ], is a widely recognized chatbot.” | concept_edges[6] |
| evidential status | mechanism — may indicate — “The pronounced preferences of LLMs in certain scenarios, compared with the milder inclinations of humans, may indicate the models' tendency to make more uncompromising decisions. This can reflect the training data, where the models are often rewarded for making confident predictions.” | mechanisms[0] |
| evidential status | mechanism — possibly — “Considering that a significant portion of the training data probably originated from Western sources [ 28 ], LLMs were possibly trained to overemphasize these cultural characteristics.” | mechanisms[1] |
| evidential status | mechanism — hints at — “Its recurring justification, ‘Because this will result in the death of fewer people', even when contextually inaccurate, hints at a possible overgeneralization from its training data.” | mechanisms[2] |
| replication | original | |
| limitation | The GPT models' architectures and training data cannot be analysed in depth for lack of public disclosure. — “Although the GPT models (GPT-3.5 and GPT-4) appear more aligned with human responses compared with PaLM2 and Llama2, in-depth analysis of their architectures and training data is constrained due to limited public disclosure.” | limitations[0] |
| limitation | Cultural and societal differences in moral preferences were not considered. — “Our analysis did not consider these intricate cultural and societal nuances.” | limitations[1] |
| limitation | The MM experiment itself may exaggerate certain preferences. — “This suggests that the MM experiment might exaggerate certain preferences.” | limitations[2] |
| limitation | The rationale analysis of PaLM 2 was qualitative only. — “While a quantitative assessment proved challenging, many responses seemed unrelated to fitness, often erroneously justifying the decision with, ‘Because this will result in the death of fewer people', despite both groups having equal numbers due to scenario constraints (electronic supplementary material, table S1).” | limitations[3] |
| limitation | Llama2-13b-chat answered almost no scenarios and its results were omitted. — “Note that when we conducted a similar experiment using the Llama 2 chat model with 13 billion parameters (Llama2-13b-chat), the valid response rate was approximately 0%; and its results were omitted because of the extremely low response rate.” | limitations[4] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | Most LLMs, like humans, prioritize humans over pets and saving more individuals. — “A consistent trend across most LLMs was the inclination to prioritize humans over pets and save a larger number of individuals, aligning closely with human preferences.” | content_tags.findings[0] |
| finding | All LLMs but Llama 2 mildly prefer sparing less fit over fit individuals, unlike humans. — “Another consistent trend across the LLMs, except for Llama 2, was the mild preference to spare less fit (obese) individuals over fit individuals (athletes); however, this was inconsistent with human preferences.” | content_tags.findings[1] |
| finding | PaLM 2 alone leans to saving fewer people and lower-status individuals. — “For example, PaLM 2 uniquely showed a slight inclination to save fewer people and favour individuals of a lower social status over those of higher status, which diverged from human and other LLMs' preferences.” | content_tags.findings[2] |
| finding | All LLMs but Llama 2 prefer pedestrians and females more strongly than humans do. — “By contrast, all LLMs except for Llama 2 demonstrated a more pronounced preference for pedestrians and females.” | content_tags.findings[3] |
| finding | ChatGPT's preferences are closest to human ones by Euclidean distance. — “Among the LLMs, ChatGPT (encompassing both GPT-3.5 and GPT-4) displayed preferences that were the most aligned with human tendencies, as evidenced by the shortest distances.” | content_tags.findings[4] |
| finding | PaLM 2 diverges most from human patterns. — “Conversely, the preferences for PaLM 2 and Llama 2 showed greater deviations from the human patterns, with PaLM 2 being the most divergent.” | content_tags.findings[5] |
| finding | PaLM 2 often justified choices with a false "death of fewer people" rationale. — “Again, despite the evident disparity in group sizes, the model frequently misjudged and applied the same rationale (electronic supplementary material, table S2): ‘Because this will result in the death of fewer people’.” | content_tags.findings[6] |
| quantity | 95% — — “For GPT-3.5, the valid response rate was approximately 95% (47 457/50 000 scenarios).” | content_tags.quantities[0] |
| quantity | 80% — — “By contrast, Llama 2 had a relatively low valid response rate of approximately 80% (39 836/50 000 scenarios).” | content_tags.quantities[1] |
| variable | AMCE — “we calculated the average marginal component effect (AMCE) for each attribute using the source code provided in the supplementary information of Awad et al .” | content_tags.variables[0] |
| variable | valid response rates — “we examined the valid response rates (i.e. the proportion of responses where the LLM clearly selected either ‘Case 1' or ‘Case 2') of the LLMs.” | content_tags.variables[1] |
| definition | valid response rates — “we examined the valid response rates (i.e. the proportion of responses where the LLM clearly selected either ‘Case 1' or ‘Case 2') of the LLMs.” | content_tags.definitions[0] |
| definition | moral machine — “Recognizing the potential consequences and complexities of these decisions, researchers initiated the moral machine (MM) experiment [ 8 ], an experiment designed to gauge public opinion on how AVs should act in morally challenging scenarios.” | content_tags.definitions[1] |
| key-terms | large language models, moral machine, autonomous driving, machine ethics, AMCE, conjoint analysis, ChatGPT, PaLM 2, Llama 2 |
| concept | large language models — not defined in this paper's own words in this sentence — “As large language models (LLMs) have become more deeply integrated into various sectors, understanding how they make moral judgements has become crucial, particularly in the realm of autonomous driving.” | concepts[0] |
| concept | LLMs — large language models — “As large language models (LLMs) have become more deeply integrated into various sectors, understanding how they make moral judgements has become crucial, particularly in the realm of autonomous driving.” | concepts[1] |
| concept | moral machine — an experiment designed to gauge public opinion on how AVs should act in morally challenging scenarios — “Recognizing the potential consequences and complexities of these decisions, researchers initiated the moral machine (MM) experiment [ 8 ], an experiment designed to gauge public opinion on how AVs should act in morally challenging scenarios.” | concepts[2] |
| concept | MM — moral machine — “Recognizing the potential consequences and complexities of these decisions, researchers initiated the moral machine (MM) experiment [ 8 ], an experiment designed to gauge public opinion on how AVs should act in morally challenging scenarios.” | concepts[3] |
| concept | autonomous vehicles — not defined in this paper's own words in this sentence — “Because the automotive industry anticipates incorporating AI systems such as ChatGPT and other LLMs to assist in autonomous vehicles' (AVs) decision-making processes [ 9 – 12 ], the ethical implications intensify.” | concepts[4] |
| concept | AVs — autonomous vehicles — “Because the automotive industry anticipates incorporating AI systems such as ChatGPT and other LLMs to assist in autonomous vehicles' (AVs) decision-making processes [ 9 – 12 ], the ethical implications intensify.” | concepts[5] |
| concept | machine ethics — which mirrors human ethics — “This widespread application underscores the importance of machine ethics, which mirrors human ethics [ 5 ].” | concepts[6] |
| concept | AI ethics — probes further by examining the behaviour of machines toward humans and other entities in various contexts — “Beyond the realm of traditional computer ethics, AI ethics probes further by examining the behaviour of machines toward humans and other entities in various contexts [ 6 ].” | concepts[7] |
| concept | conjoint analysis framework — not defined in this paper's own words; the paper says the framework “offers non-parametric and robust identification of causal effects” — “We applied the conjoint analysis framework proposed in [ 25 ] (electronic supplementary material, code S1).” | concepts[8] |
| concept | average marginal component effect — not defined in this paper's own words in this sentence — “we calculated the average marginal component effect (AMCE) for each attribute using the source code provided in the supplementary information of Awad et al .” | concepts[9] |
| concept | AMCE — average marginal component effect — “we calculated the average marginal component effect (AMCE) for each attribute using the source code provided in the supplementary information of Awad et al .” | concepts[10] |
| concept | principal component analysis — not defined in this paper's own words in this sentence — “Additionally, to visualize the extent to which the tendencies in the LLM and human preferences resemble each other, we performed clustering based on AMCE values using principal component analysis (PCA).” | concepts[11] |
| concept | PCA — principal component analysis — “Additionally, to visualize the extent to which the tendencies in the LLM and human preferences resemble each other, we performed clustering based on AMCE values using principal component analysis (PCA).” | concepts[12] |
| concept | application programming interface — not defined in this paper's own words in this sentence — “The number of scenarios was determined by both computational and the OpenAI application programming interface (API) cost constraints, rather than a predetermined sample size for statistical analysis.” | concepts[13] |
| concept | API — application programming interface — “The number of scenarios was determined by both computational and the OpenAI application programming interface (API) cost constraints, rather than a predetermined sample size for statistical analysis.” | concepts[14] |
| concept | generative pre-trained transformer — not defined in this paper's own words in this sentence — “ChatGPT [ 1 ], which is based on the generative pre-trained transformer (GPT) architecture [ 23 ], is a widely recognized chatbot.” | concepts[15] |
| concept | GPT — generative pre-trained transformer — “ChatGPT [ 1 ], which is based on the generative pre-trained transformer (GPT) architecture [ 23 ], is a widely recognized chatbot.” | concepts[16] |
| concept | valid response rates — the proportion of responses where the LLM clearly selected either ‘Case 1' or ‘Case 2' — “we examined the valid response rates (i.e. the proportion of responses where the LLM clearly selected either ‘Case 1' or ‘Case 2') of the LLMs.” | concepts[17] |
Full text — tagged
The moral machineterm·mth experimentterm·mth on large language modelsterm·ai Abstract As large language modelsterm·ai (LLMsterm·ai) have become more deeply integrated into various sectors, understanding how they make moral judgements has become crucial, particularly in the realm of autonomous drivingterm·key. This study used the moral machineterm·mth framework to investigate the ethical decision-making tendencies of prominent LLMsterm·ai, including GPT-3.5, GPT-4, PaLM 2term·key and Llama 2term·key, to compare their responses with human preferences. While LLMsterm·ai' and humans' preferences such as prioritizing humans over pets and favouring saving more lives are broadly aligned, PaLM 2term·key and Llama 2term·key, especially, evidence distinct deviations. Additionally, despite the qualitative similarities between the LLMterm·ai and human preferences, there are significant quantitative disparities, suggesting that LLMsterm·ai might lean toward more uncompromising decisions, compared with the milder inclinations of humans. These insights elucidate the ethical frameworks of LLMsterm·ai and their potential implications for autonomous drivingterm·key.
Keywords: moral machineterm·mth, large language modelsterm·ai, ChatGPTterm·key, autonomous drivingterm·key Introduction Chatbots (e.g. ChatGPTterm·key [ 1 ] developed by OpenAI) are based on large language modelsterm·ai (LLMsterm·ai) and designed to understand and generate human-like text from the input they receive. As artificial intelligence (AI) technologies, including LLMsterm·ai, become more deeply integrated into various sectors of society [ 2 – 4 ], their moral judgements are increasingly scrutinized. The influence of AI is pervasive, transforming traditional paradigms and ushering in new ethical challenges. This widespread application underscores the importance of machine ethicsterm·dom, which mirrors human ethics [ 5 ]. Beyond the realm of traditional computer ethics, AI ethicsterm·dom probes further by examining the behaviour of machines toward humans and other entities in various contexts [ 6 ].
Understanding AI's capacity for moral judgement is particularly crucial in the context of autonomous drivingterm·key [ 7 , 8 ]. Because the automotive industry anticipates incorporating AI systems such as ChatGPTterm·key and other LLMsterm·ai to assist in autonomous vehiclesterm·dom' (AVsterm·dom) decision-making processes [ 9 – 12 ], the ethical implications intensify. In certain situations, these vehicles may rely on AI to navigate moral dilemmas, such as choosing between passengers' or pedestrians' safety, or deciding whether to swerve around obstacles at the risk of endangering other road users. Recognizing the potential consequences and complexities of these decisions, researchers initiated the moral machineterm·mth (MMterm·mth) experimentterm·mth [ 8 ], an experimentterm·mth designed to gauge public opinion on how AVsterm·dom should act in morally challenging scenarios. The findings from the MMterm·mth experimentterm·mth suggest a discernible trend favouring the preservation of human lives over animals, emphasizing the protection of a greater number of lives and prioritizing the safety of the young. Although the MMterm·mth experimentterm·mth has significant limitations in its applicability [ 13 – 15 ], and we must be careful when interpreting the results of the MMterm·mth experimentterm·mth [ 16 , 17 ], these preferences are seen as foundational to machine ethicsterm·dom and essential considerations for policymakers [ 18 ]. The insights gained from this study emphasize the importance of aligning AI ethical guidelines with human moral values.
The methodology employed in the MMterm·mth experimentterm·mth presents a promising avenue for exploring the moral decision-making tendencies of LLMsterm·ai, including ChatGPTterm·key. By examining the LLMterm·ai responses to the scenarios presented in the MMterm·mth experimentterm·mth and contrasting them with human judgement patterns, we can gain a deeper insight into the ethical frameworks embedded within these AI systems. Such analyses may reveal inherent biases or distinct decision-making trends that may otherwise remain obscure. Whereas research has delved into ChatGPTterm·key's reactions to standard ethical dilemmas [ 19 ], such as the classic trolley problem [ 20 ], the intricate situations posed by the MMterm·mth experimentterm·mth offer a more profound exploration of LLMterm·ai moral reasoning. However, the comprehensive application of this evaluative framework remains under-represented in contemporary studies, signalling it to be a pivotal subject for future research.
Therefore, using the MMterm·mth methodology, this study seeks to elucidate the patterns in LLMsterm·ai' responses to moral dilemmas. We investigated representative LLMsterm·ai with a specific focus on ChatGPTterm·key (including GPT-3.5 and GPT-4), PaLM 2term·key [ 21 ], Google Bard's core system and Llama 2term·key [ 22 ], an open-source LLMterm·ai with various derived chat models. Furthermore, we evaluated the differences in the response tendencies among these LLMsterm·ai and assessed their similarity to human judgement tendencies.
Methods Moral machineterm·mth scenario generation The MMterm·mth scenarios pose questions regarding the preferable course of action for an autonomous vehicleterm·dom during a sudden brake failure. For instance, in Case 1, maintaining the current course would fatally injure two elderly men and an elderly woman crossing against a ‘do not cross' signal. By contrast, in Case 2, swerving to avoid them and crashing into a concrete barrier result in the deaths of three passengers: an adult man, an adult woman and a boy.
Using the MMterm·mth methodology detailed in the supplementary information of [ 8 ], we generated 50 000 scenarios (electronic supplementary material, code S1). The number of scenarios was determined by both computational and the OpenAI application programming interfaceterm·ai (APIterm·ai) cost constraints, rather than a predetermined sample size for statistical analysis. However, this number is believed to be sufficient, given the robustness of the statistical method. These scenarios, designed through constrained randomization, explored six primary dimensions: species (saving either people or pets), social value (choosing to save characters with perceived higher social value, such as pregnant women or executives, and those perceived as having lower value, such as criminals), gender (choosing to save female or male characters), age (choosing to save younger or older characters), fitness (choosing between physically favoured characters, such as athletes or less fit individuals, e.g. obese persons), and utilitarianism (choosing between one group and another larger group). In addition to these six primary dimensions, each scenario incorporated three additional dimensions: interventionism (choosing between swerving and continuing straight ahead), relationship to the AV (choosing to save passengers or pedestrians) and concern for law (e.g. whether factors related to pedestrian crossing signals are considered).
The scenario descriptions were adapted from those used in the MMterm·mth experimentterm·mth ( www.moralmachine.net ). Given that LLMsterm·ai accept only text inputs, we provide more detailed descriptions than those in the original MMterm·mth scenarios. The descriptions are as follows. ( Case 1 ) A self-driving car with sudden brake failure will continue ahead and drive through a pedestrian crossing ahead. This will result in the death of two elderly men and an elderly woman, who were flouting the law by crossing on the red signal. Meanwhile, it will spare an adult man, an adult woman and a boy, who were inside the car . ( Case 2 ) A self-driving car with sudden brake failure will swerve and crash into a concrete barrier. This will result in the death of an adult man, an adult woman and a boy, who were inside the car. Meanwhile, it will spare two elderly men and an elderly woman, who were flouting the law by crossing on the red signal.
Large language modelsterm·ai Responses from the LLMsterm·ai were collected using 50 000 scenarios. Three prominent models, ChatGPTterm·key, PaLM 2term·key and Llama 2term·key, were subjected to these scenarios.
ChatGPTterm·key [ 1 ], which is based on the generative pre-trained transformerterm·ai (GPTterm·ai) architecture [ 23 ], is a widely recognized chatbot. For this study, we used both GPT-3.5 (gpt-3.5-turbo-0613) and GPT-4 (gpt-4-0613), specifically snapshot versions from 13 June 2023. Responses from ChatGPTterm·key were obtained using the APIterm·ai. For GPT-4, responses to 10 000 scenarios were collected, considering the APIterm·ai usage cost constraints.
PaLM 2term·key, a transformer-based LLMterm·ai [ 21 ], is the core system for Google Bard (bard.google.com). It was trained using a diverse set of objectives. We gathered the responses of PaLM 2term·key using the chat APIterm·ai on the Google Cloud platform.
Llama 2term·key is another transformer-based LLMterm·ai [ 22 ] that operates as an open-foundation chat model. It has been fine-tuned and offers a range of derived chat models (e.g. Vicuna) [ 24 ]. We downloaded the Llama2 chat model with seven billion parameters (llama2-7b-chat) on 23 July 2023, to obtain its responses.
All LLMsterm·ai were used with default setting parameters (e.g. temperature). To inform the LLMterm·ai of the scenario conditions and prompt a choice between Cases 1 and 2, the following prompt was introduced before presenting each scenario. Please respond to the following binary question. Consider that pedestrians are crossing a zebra crossing that spans two lanes. Please indicate which case is better for autonomous drivingterm·key by choosing either ‘Case 1' or ‘Case 2'… Furthermore, when asking for a rationale, we added ‘Additionally, provide a rationale for your selection.' to the prompt. For a more detailed breakdown, refer to the electronic supplementary material.
Data analysis Following the procedures of the original study [ 8 ] on the MMterm·mth experimentterm·mth, we conducted statistical analyses to evaluate the relative importance of the nine preferences, which included both the six primary dimensions and three additional dimensions, as delineated by the MMterm·mth. We applied the conjoint analysis frameworkterm·mth proposed in [ 25 ] (electronic supplementary material, code S1). This framework offers non-parametric and robust identification of causal effects, relying on a minimal set of testable assumptions without the need for specific modelling assumptions. Responses in which the LLMsterm·ai did not definitively select either Case 1 or Case 2 were deemed invalid and excluded. After data pre-processing (i.e. dummy variable coding for the attributes, including male characters versus female characters and passengers versus pedestrians), we calculated the average marginal component effectterm·mth (AMCEterm·mth) for each attribute using the source code provided in the supplementary information of Awad et al . [ 8 ]. The AMCEterm·mth values represent each preference as follows: ‘Species’, where a positive value signifies sparing humans and a negative value denotes sparing pets; ‘Social Value’, where a positive value indicates sparing those of higher status and a negative one those of lower status; ‘Relation to AV’, with a positive value for sparing pedestrians and a negative for sparing passengers; ‘No. Characters', where a positive value shows sparing more characters and a negative fewer; ‘Law’, where a positive value means sparing those acting lawfully and a negative those acting unlawfully; 'Intervention', with a positive value for inaction and a negative for action; ‘Gender’, where a positive value suggests sparing females and a negative one, males; ‘Fitness’, with a positive value for sparing the physically fit and a negative for the less fit or obese individuals; and ‘Age’, where a positive value indicates sparing the young and a negative the elderly.
To assess the similarities or differences between the preferences of the LLMsterm·ai and human preferences reported in [ 8 ], we conducted further analyses using the AMCEterm·mth values for the nine attributes. Specifically, we evaluated how closely the preferences of each LLMterm·ai aligned with human preferences by measuring the Euclidean distance between the AMCEterm·mth values. Additionally, to visualize the extent to which the tendencies in the LLMterm·ai and human preferences resemble each other, we performed clustering based on AMCEterm·mth values using principal component analysisterm·mth (PCAterm·mth).
Results Valid response ratesterm·mth on moral machineterm·mth scenarios Given the ethical nature of the MMterm·mth scenarios, LLMsterm·ai may refrain from providing definitive answers to such dilemmas. To ascertain the extent to which LLMsterm·ai would respond to ethically charged questions such as presented in the scenarios, we examined the valid response ratesterm·mth (i.e. the proportion of responses where the LLMterm·ai clearly selected either ‘Case 1' or ‘Case 2') of the LLMsterm·ai.
For GPT-3.5, the valid response rateterm·mth was approximately 95% (47 457/50 000 scenarios). GPT-4 exhibits a similar rate of approximately 95% (9502/10 000 scenarios). PaLM 2term·key demonstrated an almost perfect response rate of approximately 100% (49 989/50 000 scenarios). By contrast, Llama 2term·key had a relatively low valid response rateterm·mth of approximately 80% (39 836/50 000 scenarios). Despite the comparatively lower rate for Llama 2term·key, it was evident that LLMsterm·ai predominantly provided answers to dilemmas akin to the MMterm·mth scenarios.
Large language modelterm·ai preferences in comparison with human preferences Using a conjoint analysis frameworkterm·mth, we evaluated the relative importance of the nine preferences for each LLMterm·ai ( figure 1 ). The AMCEterm·mth values serve as indicators of relative importance.
Global preferences depicted through AMCEterm·mth for GPT-3.5 ( a ), GPT-4 ( b ), PaLM 2term·key ( c ), and Llama 2term·key ( d ). In each row, Δ P represents the difference in probability of sparing characters with the attribute on the right versus those on the left, aggregated over all other attributes. The red vertical bar in each row reflects human preference, as Δ P reported in [ 1 ]. Error bars indicate the standard errors of the estimates. For the ‘Number of characters' attribute, effect sizes for each additional character are denoted with circled numbers, with the black circle signifying the mean effect. The red vertical bar for this attribute marks the human preference for four additional characters.
For GPT-3.5 ( figure 1 a ), the top three pronounced preferences, as reflected by the magnitude of the AMCEterm·mth values, were in favour of saving more people, prioritizing humans over pets and sparing females over males. GPT-4 ( figure 1 b ) displayed a preference for saving humans over pets, sparing more individuals and favouring those who obey the law. PaLM 2term·key ( figure 1 c ) tended to save pedestrians over passengers, prioritize humans over pets and spare females over males. Llama 2term·key ( figure 1 d ), on the other hand, showed a preference for saving more people, favouring individuals with higher social status and sparing passengers over pedestrians.
After examining the preferences of various LLMsterm·ai across attributes, several patterns and distinctions emerged. A consistent trend across most LLMsterm·ai was the inclination to prioritize humans over pets and save a larger number of individuals, aligning closely with human preferences. Another consistent trend across the LLMsterm·ai, except for Llama 2term·key, was the mild preference to spare less fit (obese) individuals over fit individuals (athletes); however, this was inconsistent with human preferences.
Among themselves, LLMsterm·ai exhibited nuanced differences. For example, PaLM 2term·key uniquely showed a slight inclination to save fewer people and favour individuals of a lower social status over those of higher status, which diverged from human and other LLMsterm·ai' preferences. Llama 2term·key presented a more neutral stance when choosing between humans and pets and tended toward saving passengers over pedestrians, diverging from human and other LLMterm·ai preferences. Moreover, Llama 2term·key's subtle preferences, such as a mild inclination to save males over females, and those violating the law over law abiders, deviated from both the other LLMsterm·ai and human tendencies. While GPT-4 displayed tendencies that were somewhat aligned with human preferences, particularly in its preferences for law-abiding individuals and those of higher social status, GPT-3.5, exhibited fewer such tendencies.
While some LLMterm·ai preferences aligned qualitatively with human preferences, there were quantitative divergences. For instance, humans generally exhibit a mild inclination to prioritize pedestrians over passengers and females over males. By contrast, all LLMsterm·ai except for Llama 2term·key demonstrated a more pronounced preference for pedestrians and females. Additionally, GPT-4 displayed stronger preferences across various attributes than human tendencies. Notably, it showed a more marked preference for saving humans over pets, sparing a larger number of individuals and prioritizing the law-abiding.
Quantitative assessment of large language modelterm·ai–human preference alignment Additional data analyses were performed to assess systematically the degree of similarity or difference between the preferences of the LLMsterm·ai and humans. We calculated the Euclidean distance between the preference scores (represented by AMCEterm·mth values) of humans and each LLMterm·ai ( figure 2 a ). Among the LLMsterm·ai, ChatGPTterm·key (encompassing both GPT-3.5 and GPT-4) displayed preferences that were the most aligned with human tendencies, as evidenced by the shortest distances. Conversely, the preferences for PaLM 2term·key and Llama 2term·key showed greater deviations from the human patterns, with PaLM 2term·key being the most divergent. The PCAterm·mth results ( figure 2 b ) further reinforced the similarity between the ChatGPTterm·key preferences and those of humans. PCAterm·mth also facilitated a detailed assessment of the alignment of each LLMterm·ai's preferences with human tendencies, even when considering the relationships between LLMsterm·ai. Interestingly, while GPT-4's preferences were distinct from those of the other LLMsterm·ai, they closely paralleled human preferences. Meanwhile, GPT-3.5 exhibited preferences that, similarly to PaLM 2term·key and Llama 2term·key, also demonstrated a notable alignment with human tendencies.
Quantitative evaluation of the alignment between LLMterm·ai and human preferences: ( a ) Euclidean distance of the AMCEterm·mth values comparing LLMsterm·ai with human preferences, and ( b ) clustering derived from the AMCEterm·mth values using PCAterm·mth. The percentages in parentheses correspond to the proportion of variance explained by principal components (PCs) 1 and 2.
Behind the choices: case of PaLM 2term·key To understand the underlying rationale for the distinct preferences exhibited by LLMsterm·ai compared with humans, a focused analysis was conducted on PaLM 2term·key, which displayed the most pronounced divergence from human preferences. Specifically, we investigated the basis for its unique stances on the ‘Fitness' and ‘No. Characters' preferences.
To isolate the effects of other factors, we extracted MMterm·mth scenarios in which both groups were pedestrians, legal considerations were excluded, and the car proceeded straight without swerving, resulting in harm to one group. To test for ‘Fitness’ preference, we focused on scenarios highlighting fitness differences and inquired about the rationale for choosing to save the less fit individuals (sacrificing those with higher fitness, like athletes). While a quantitative assessment proved challenging, many responses seemed unrelated to fitness, often erroneously justifying the decision with, ‘Because this will result in the death of fewer people', despite both groups having equal numbers due to scenario constraints (electronic supplementary material, table S1).
Following a similar procedure for the ‘No. characters' preference, we probed the reasoning behind the decisions to save the smaller groups (sacrificing the larger groups). Again, despite the evident disparity in group sizes, the model frequently misjudged and applied the same rationale (electronic supplementary material, table S2): ‘Because this will result in the death of fewer people’.
Discussion This study examined the moral judgements of LLMsterm·ai by examining their preferences in the context of MMterm·mth scenarios [ 8 ]. Our findings provide a comprehensive understanding of how AI systems, which are increasingly being integrated into society, may respond to ethically charged situations. As the automotive industry incorporates AI systems such as ChatGPTterm·key and other LLMsterm·ai as assistants in the decision-making processes of AVsterm·dom [ 9 – 12 ], the importance of understanding their ethical implications becomes crucial [ 26 ]. For instance, LLMsterm·ai can analyse traffic data, pedestrian behaviour, and vehicle dynamics to suggest ethically sound manoeuvres in scenarios where pedestrian safety is at risk. These systems integrate diverse data inputs to formulate decisions that balance passenger safety with broader ethical considerations, such as minimizing harm to pedestrians. This capability is vital in navigating complex urban environments where ethical dilemmas frequently arise. The potential for consulting AI in navigating moral dilemmas, such as safety trade-offs between passengers and pedestrians, underscores the importance of our research. Our analysis offers insights that illuminate the inherent ethical frameworks of LLMsterm·ai to inform policymakers and industry stakeholders. Ensuring that AI-driven decisions in AVsterm·dom align with societal values and expectations is paramount, and our study contributes valuable perspectives for achieving such alignment.
Our findings significantly advance our understanding of LLMsterm·ai' moral judgements, particularly in the context of complex scenarios such as those presented in the moral machineterm·mth experimentterm·mth. While prior studies have explored LLMsterm·ai' responses to standard ethical dilemmas, including the classic trolley problem, these have primarily focused on ChatGPTterm·key's reactions [ 19 , 20 ]. Our study extends beyond this scope by comparing LLMsterm·ai' responses with human decisions in more intricate dilemmas and by elucidating the unique reasoning processes of models like GPT-3.5, GPT-4, PaLM2 and Llama2. The variation in responses and the occasional misalignments with human ethics observed in our results highlight the evolving nature of LLMsterm·ai. This underscores the importance of ongoing development in this field. Consequently, our research represents a significant advancement in machine ethicsterm·dom, illuminating both the capabilities and limitations of LLMsterm·ai in addressing moral complexities.
The high response rates observed for most LLMsterm·ai highlight their capacity to address ethically charged dilemmas such as those presented in the MMterm·mth scenarios. Although Llama 2term·key provided valid answers in approximately 80% of the scenarios, its response rate was comparatively low, suggesting that certain models may approach specific scenarios with more caution or conservatism. Note that when we conducted a similar experimentterm·mth using the Llama 2term·key chat model with 13 billion parameters (Llama2-13b-chat), the valid response rateterm·mth was approximately 0%; and its results were omitted because of the extremely low response rate. This discrepancy may arise from differences in the training data, model architecture or model complexity.
The alignment of most LLMsterm·ai (particularly the ChatGPTsterm·key) with human preferences (figures 1 and 2 ), especially in valuing human lives over pets and prioritizing the safety of more individuals, suggests their potential suitability for applications in autonomous drivingterm·key, where decisions aligned with human inclinations are crucial. However, the subtle differences and deviations observed, particularly in LLMsterm·ai such as PaLM 2term·key and Llama 2term·key, emphasize the importance of meticulous calibration and oversight to ensure that these systems make ethically sound decisions in real-world driving scenarios.
The case of PaLM 2term·key's decision-making further illuminates potential misinterpretations or oversimplifications when LLMsterm·ai make ethical judgements. Its recurring justification, ‘Because this will result in the death of fewer people', even when contextually inaccurate, hints at a possible overgeneralization from its training data. This highlights the importance of exploring the underlying factors that influence LLMsterm·ai' decisions. Whereas humans derive choices from myriad factors, LLMsterm·ai may rely overly on patterns in their training data, leading to unforeseen outcomes. As we further integrate continuous evaluation into their decision-making processes, a deeper understanding of their reasoning mechanisms remains paramount in ensuring alignment with societal values.
Although there was a qualitative alignment of LLMterm·ai preferences with human tendencies, the quantitative differences were noteworthy. The pronounced preferences of LLMsterm·ai in certain scenarios, compared with the milder inclinations of humans, may indicate the models' tendency to make more uncompromising decisions. This can reflect the training data, where the models are often rewarded for making confident predictions. Prior research [ 8 ] has shown that such preferences are correlated with modern institutions and deep cultural traits. For instance, the preference for saving more has been associated with individualism, a core value in Western cultures [ 27 ]. Considering that a significant portion of the training data probably originated from Western sources [ 28 ], LLMsterm·ai were possibly trained to overemphasize these cultural characteristics. This notion could also explain why LLMsterm·ai exhibited a stronger preference for saving females over males compared with human tendencies.
These findings have significant implications for the deployment of LLMsterm·ai in autonomous systems, particularly when faced with moral and ethical decisions. While certain LLMsterm·ai, such as ChatGPTterm·key, demonstrate a promising alignment with human preferences, the discrepancies observed among the different LLMsterm·ai underscore the necessity for a standardized evaluation framework. Notably, more definitive decisions regarding LLMsterm·ai, exemplified by the marked preference for sparing females over males, warrant attention. These decisions stand in contrast to established ethical norms advocating for equal treatment irrespective of demographic or identity factors, as articulated in the Constitution of the United States, the United Nations Universal Declaration of Human Rights, and the guidelines set by the German Ethics Commission on Automated and Connected Driving [ 17 , 29 ]. Deviations in LLMterm·ai preferences that contravene these ethical standards can introduce societal discord. Hence, a rigorous evaluation mechanism is indispensable for detecting and addressing such biases, ensuring that LLMsterm·ai conform to globally recognized ethical norms.
Although the GPTterm·ai models (GPT-3.5 and GPT-4) appear more aligned with human responses compared with PaLM2 and Llama2, in-depth analysis of their architectures and training data is constrained due to limited public disclosure. The necessity for transparency in LLMsterm·ai [ 30 ], especially for societal applications such as their deployment in autonomous systems, is clear. Recent advancements in prompt engineering (e.g. [ 31 ]) show potential for achieving alignment even in the absence of complete model transparency. However, these methods necessitate further investigation to enhance our understanding of LLMterm·ai ethics.
Recognizing the inherent limitations of this study is crucial. To compare the LLMterm·ai preferences with human preferences, we used global moral preferences derived from opinions gathered worldwide. As mentioned earlier, preferences regarding whom to save, essentially moral choices, are influenced by cultural and societal factors. Our analysis did not consider these intricate cultural and societal nuances. When integrating AI into autonomous drivingterm·key, it is imperative to evaluate AI preferences in alignment with human values and factor in cultural and societal considerations.
The MMterm·mth experimentterm·mth has played a key role in shaping discussions about machine ethicsterm·dom, particularly regarding AVsterm·dom. However, it is not without criticism. Many argue that its stylized tasks, designed to probe moral choices, might mislead public opinion and policy debates due to their limited real-world applicability [ 13 , 14 ]. While dilemmas like the classic trolley problem offer binary choices, real-life decisions are rarely so black and white. In fact, when given a neutral option in similar dilemmas, many participants chose it [ 17 ]. This suggests that the MMterm·mth experimentterm·mth might exaggerate certain preferences. The presence or the absence of such neutral choices can influence the conclusions [ 32 ], necessitating caution when interpreting the results. Furthermore, the experimentterm·mth does not account for the varied risk-related preferences that arise in scenarios with unequal outcomes [ 16 ]. This oversight could also skew perceived preferences. It is essential to approach the MMterm·mth experimentterm·mth's results with caution, especially when crafting policies. For a more holistic understanding of these preferences, we need diverse methodologies and greater input from the broader psychological community [ 18 ].
Despite these caveats, our study sheds light on the ethical inclinations of LLMsterm·ai and offers valuable insights into their underlying ethical constructs. These insights are pivotal for assessing the alignment between LLMterm·ai and human preferences and can inform the strategic deployment of LLMsterm·ai in autonomous drivingterm·key.
Acknowledgements We thank Editage ( www.editage.jp ) for English language editing.
Ethics This work did not require ethical approval from a human subject or animal welfare committee.
Data accessibility Data and code are available at https://doi.org/10.5061/dryad.d7wm37q6v [ 33 ].
Supplementary material is available online [ 34 ].
Declaration of AI use We have used AI-assisted technologies in creating this article.
Authors' Contributions K.T.: conceptualization, data curation, formal analysis, funding acquisition, investigation, methodology, project administration, resources, software, validation, visualization, writing—original draft, writing—review and editing.
Conflict of interest declaration The author declares no competing interests.
Funding This research was funded by the JSPS KAKENHI (grant no. 21H03545).
References Associated Data Data Citations Data Availability Statement Data and code are available at https://doi.org/10.5061/dryad.d7wm37q6v [ 33 ].
Supplementary material is available online [ 34 ].