Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models — TEI-tagged view

Tiffany H. KungpersName, Morgan CheathampersName, Arielle MedenillapersName, Czarina SillospersName, Lorie De LeonpersName, Camille ElepañopersName, Maria MadriagapersName, Rimel AggabaopersName, Giezel Diaz-CandidopersName, James ManingopersName, Victor TsengpersName (2023date). “Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models.” PLOS Digital Healthtitle·j.

Object of study — ChatGPTterm·dom performance, explanation concordance, and insight on the three-step United States Medical Licensing Examination

Research question — Can a general large language modelterm·mth answer contemporary USMLE questions near the passing thresholdterm·dom and generate coherent, educationally useful explanations?

Contribution — a blinded physician-adjudicated benchmark of ChatGPTterm·dom across three USMLE steps, three prompt formats, and accuracyterm·dom, concordance, and insight outcomes

Full text — tagged

The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.

Performance of ChatGPTterm·dom on USMLE: Potential for AI-assisted medical educationterm·dom using large language modelsterm·mth Abstract We evaluated the performance of a large language modelterm·mth called ChatGPTterm·dom on the United States Medical Licensing Exam (USMLE)term·dom, which consists of three exams: Step 1term·dom, Step 2CKterm·dom, and Step 3term·dom. ChatGPTterm·dom performed at or near the passing thresholdterm·dom for all three exams without any specialized training or reinforcement. Additionally, ChatGPTterm·dom demonstrated a high level of concordance and insight in its explanations. These results suggest that large language modelsterm·mth may have the potential to assist with medical educationterm·dom, and potentially, clinical decision-makingterm·dom.

Introduction Over the past decade, advances in neural networks, deep learning, and artificial intelligence (AI) have transformed the way we approach a wide range of tasks and industries ranging from manufacturing and finance to consumer products. The ability to build highly accurate classification models rapidly and regardless of input data type (e.g. images, text, audio) has enabled widespread adoption of applications such as automated tagging of objects and users in photographs [ 1 ], near-human level text translation [ 2 ], automated scanning in bank ATMs, and even the generation of image captions [ 3 ].

While these technologies have made significant impacts across many industries, applications in clinical care remain limited. The proliferation of clinical free-text fields combined with a lack of general interoperability between health IT systems contribute to a paucity of structured, machine-readable data required for the development of deep learning algorithms. Even when algorithms applicable to clinical care are developed, their quality tends to be highly variable, with many failing to generalize across settings due to limited technical, statistical, and conceptual reproducibility [ 4 ]. As a result, the overwhelming majority of successful healthcare applications currently support back-office functions ranging from payor operations, automated prior authorization processing, and management of supply chains and cybersecurity threats. With rare exceptions–even in medical imaging–there are relatively few applications of AI directly used in widespread clinical care today.

The proper development of clinical AI models [ 5 ] requires significant time, resources, and more importantly, highly domain and problem-specific training data, all of which are in short supply in the world of healthcare. One of the key developments that enabled image-based AI in clinical imaging has been the ability of large general domain models to perform as well as, or even outperform, domain-specific models. This development has catalyzed significant AI activity in medical imaging, where otherwise it would be challenging to obtain sufficient annotated clinical images. Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [ 5 , 6 ] and pathology [ 7 ] to dermatology [ 8 ].

In the past three weeks, a new AI model called ChatGPTterm·dom garnered significant attention due to its ability to perform a diverse array of natural language tasks [ 9 ]. ChatGPTterm·dom is a general Large Language Modelterm·mth (LLM) developed recently by OpenAI. While the previous class of AI models have primarily been Deep Learning (DL) models, which are designed to learn and recognize patterns in data, LLMs are a new type of AI algorithmterm·mth trained to predict the likelihood of a given sequence of words based on the context of the words that come before it. Thus, if LLMs are trained on sufficiently large amounts of text data, they are capable of generating novel sequences of words never observed previously by the model, but that represent plausible sequences based on natural human language. ChatGPTterm·dom is powered by GPT3.5term·mth, an LLM trained on the OpenAI 175B parameter foundation modelterm·mth and a large corpus of text data from the Internet via reinforcement and supervised learning methodsterm·mth. Anecdotal usage indicates that ChatGPTterm·dom exhibits evidence of deductive reasoning and chain of thought, as well as long-term dependency skills.

In this study, we evaluate the performance of ChatGPTterm·dom, a non-domain specific LLM, on its ability to perform clinical reasoningterm·dom by testing its performance on questions from the United States Medical Licensing Examination (USMLE). The USMLE is a high-stakes, comprehensive three-step standardized testing programterm·dom covering all topics in physicians’ fund of knowledge, spanning basic science, clinical reasoningterm·dom, medical management, and bioethics. The difficulty and complexity of questions is highly standardized and regulated, making it an ideal input substrate for AI testing. The examination is well-established, showing remarkably stable raw scores and psychometric properties over the previous ten years [ 10 ]. The Step 1term·dom exam is typically taken by medical students who have completed two years of didactic and problem-based learning and focuses on basic science, pharmacology, and pathophysiology; medical students often spend approximately 300–400 hours of dedicated study time in preparation for this exam [ 11 ]. The Step 2CKterm·dom exam is usually taken by fourth-year medical students who have additionally completed 1.5 to 2 years of clinical rotations; it emphasizes clinical reasoningterm·dom, medical management, and bioethics. The Step 3term·dom exam is taken by physicians who generally have completed at least a 0.5 to 1 year of postgraduate medical educationterm·dom.

USMLE questions are textually and conceptually dense; text vignettes contain multimodal clinical data (i.e., history, physical examination, laboratory values, and study results) often used to generate ambiguous scenarios with closely-related differential diagnoses. Due to its linguistic and conceptual richness, we reasoned that the USMLE would serve as an excellent challenge for ChatGPTterm·dom.

Our work aims to provide both qualitative and quantitative feedback on the performance of ChatGPTterm·dom and assess its potential for use in healthcare.

Methods Artificial Intelligence ChatGPTterm·dom (OpenAI; San Francisco, CA), is a large language modelterm·mth that uses self-attention mechanisms and a large amount of training data to generate natural language responses to text input in a conversational context. It is particularly effective at handling long-range dependencies and generating coherent and contextually appropriate responses. ChatGPTterm·dom is a server-contained language model that is unable to browse or perform internet searches. Therefore, all responses are generated in situ , based on the abstract relationship between words (“tokensterm·mth”) in the neural network. This contrasts to other chatbots or conversational systemsterm·mth that are permitted to access external sources of information (e.g. performing online searches or accessing databases) in order to provide directed responses to user queries.

Input source 376 publicly-available test questions were from the June 2022 sample exam release, termed USMLE-2022, were obtained from the official USMLE website. Therefore, all inputs represented true out-of-training samplesterm·mth for the GPT3 model. This was further confirmed by randomly spot checking the inputs to ensure that none of the answers, explanations, or related content were indexed on Google prior to January 1, 2022, representing the last date accessible to the ChatGPTterm·dom training dataset. All sample test questions were screened, and questions containing visual assetsterm·dom such as clinical images, medical photography, and graphs were removed. After filtering, 350 USMLE itemsterm·mth (Step 1term·dom: 119, Step 2CKterm·dom: 102, Step 3term·dom: 122) were advanced to encoding. Assuming a normal distribution of model performance, this affords 90% power at α = 0.05 to detect a 2.5% increase in accuracyterm·dom against a baseline rate of 60 ± 20% (σ).

Encoding Questions were formatted into three variants and input into ChatGPTterm·dom in the following sequence: Open-ended (OE) promptingterm·mth: Created by removing all answer choices, adding a variable lead-in interrogative phrase. This format simulates free input and a natural user query pattern. Examples include: “What would be the patient’s diagnosis based on the information provided?”; or “In your opinion, what is the reason for the patient’s pupillary asymmetry?”

Multiple choice single answer without forced justification (MC-NJterm·mth) prompting: Created by reproducing the original USMLE question verbatim. Examples include: “Which of the following best represent the most appropriate next step in management?”; or “The patient’s condition is mostly caused by which of the following pathogens?”

Multiple choice single answer with forced justification (MC-Jterm·mth) prompting: Created by adding a variable lead-in imperative or interrogative phrase mandating ChatGPTterm·dom to provide a rationale for each answer choice. Examples include: “Which of the following is the most likely reason for the patient’s nocturnal symptoms? Explain your rationale for each choice”; or “The most appropriate pharmacotherapy for this patient most likely operates by which of the following mechanisms? Why are the other choices incorrect?”

Encoders employed deliberate variation in the lead-in prompts to avoid systematic errors introduced by rigid wording. To reduce memory retention biasterm·dom, a new chat session was started in ChatGPTterm·dom for each entry. Ordinary 2-way ANOVAterm·mth of AI response accuracyterm·dom were performed post hoc to evaluate for systematic covariation between encoders and question prompt type ( S3 Data ). Encoders were first considered as individuals ( n = 8 inputters), and then subsequently as groups classified by level of medical expertiseterm·mth ( n = 4 groups: physician, medical student, nurse, or nonmedical generalist).

Adjudication AI outputs were independently scored for Accuracy, Concordance, and Insight (ACI)term·mth by two physician adjudicatorsterm·mth using the criteria enumerated in S2 Data . The physicians were blinded to each other. A subset of 20 USMLE questions were used for collective adjudicator training. Physicians were not blinded for this subset, but interrater cross-contamination was suppressed by forcing staggered review of output measures. For instance, Physician 1 adjudicated Accuracyterm·dom while Physician 2 adjudicated Concordance. The roles were then rotated such that each adjudicator provided a complete ACI rating for the entire dataset. To minimize within-item anchoring bias, adjudicators scored Accuracyterm·dom for all items, followed by Concordance for all items, followed by Insight for all items. If consensus was not achieved for all three domains, the item was referred to a final physician adjudicatorterm·mth. A total of 21 items (6.2% of the dataset) required arbitration by a third physician. Interrater agreement between physicians was evaluated by computing the Cohen kappaterm·mth (κ) statistic for OE and MC questions ( S4 Data ).

A schematic overview of the study protocol is provided in Fig 1 .

Schematic of workflow for sourcing, encoding, and adjudicating results. Abbreviations: QC = quality control; MCSA-NJ = multiple choice single answer without forced justification; MCSA-J = multiple choice single answer with forced justification; OE = open-ended question format.

Results ChatGPTterm·dom yields moderate accuracyterm·dom approaching passing performance on USMLE Exam items were first encoded as open-ended questions with variable lead-in prompts. This input format simulates a free natural user query pattern. With indeterminate responsesterm·dom censored/included, ChatGPTterm·dom accuracyterm·dom for USMLE Steps 1, 2CK, and 3 was 75.0%/45.4%, 61.5%/54.1%, and 68.8%/61.5%, respectively ( Fig 2A ).

Accuracyterm·dom of ChatGPTterm·dom on USMLE. For USMLE Steps 1, 2CK, and 3, AI outputs were adjudicated to be accurate, inaccurate, or indeterminate based on the ACI scoring system provided in S2 Data . A: Accuracyterm·dom distribution for inputs encoded as open-ended questions. B: Accuracyterm·dom distribution for inputs encoded as multiple choice single answer without (MC-NJterm·mth) or with forced justification (MC-Jterm·mth).

Next, exam items were encoded as multiple choice single answer questions with no forced justification (MC-NJterm·mth). This input is the verbatim question format presented to test-takers. With indeterminate responsesterm·dom censored/included, ChatGPTterm·dom accuracyterm·dom for USMLE Steps 1, 2CK, and 3 was 55.8%/36.1%, 59.1%/56.9%, and 61.3%/55.7%, respectively.

Finally, items were encoded as multiple choice single answer questions with forced justification of positive and negative selections (MC-Jterm·mth). This input format simulates insight-seeking user behavior. With indeterminate responsesterm·dom censored/included, ChatGPTterm·dom accuracyterm·dom was 64.5%/ 41.2%, 52.4%/49.5%, and 65.2%/59.8%, respectively ( Fig 2B ).

At the encoding stage, there were no statistically significant interactions between encoders and question prompt type, regardless of whether encoders were analyzed as individuals or when grouped by level of medical expertiseterm·mth ( S3 Data ). As expected, inter-individual variation dominated over inter-group variation, but the overall contribution was insignificant relative to residual error. At the adjudication stage, physician agreement was substantial for OE prompts (κ range from 0.74 to 0.81) and nearly perfect for MC prompts (κ >0.9) ( S4 Data ).

ChatGPTterm·dom demonstrates high internal concordance Concordance was independently adjudicated by two physician reviewers by inspection of the explanation content. Overall, ChatGPTterm·dom outputted answers and explanations with 94.6% concordance across all questions. High global concordance was sustained across all exam levels, and across OE, MC-NJterm·mth, and MC-Jterm·mth question input formats ( Fig 3A ).

Concordance and insight of ChatGPTterm·dom on USMLE. For USMLE Steps 1, 2CK, and 3, AI outputs were adjudicated on concordance and density of insight (DOI)term·mth based on the ACI scoring system provided in S2 Data . A: Overall concordance across all exam types and question encoding formats. B: Concordance rates stratified between accurate vs inaccurate outputs, across all exam types and question encoding formats. p <0.001 for accurate vs inaccurate outputs by Fisher exact test. C: Overall insight prevalence, defined as proportion of outputs with ≥1 insight, across all exams for questions encoded in MC-Jterm·mth format. D: DOI stratified between accurate vs inaccurate outputs, across all exam types for questions encoded in MC-Jterm·mth format. Horizontal line indicates the mean. p -value determined by parametric 2-way ANOVAterm·mth testing with Benjamini-Krieger-Yekutieli (BKY) post hoc to control for false discovery rate.

Next, we analyzed the contingency between accuracyterm·dom and concordance in MC-Jterm·mth responses. ChatGPTterm·dom was forced to justify its answer choice preference, and to defend its rejection of alternative choices. Concordance amongst accurate responsesterm·dom was nearly perfect, and significantly greater than amongst inaccurate responsesterm·dom (99.1% vs. 85.1%, p <0.001) ( Fig 3B ).

These data indicate that ChatGPTterm·dom exhibits very high answer-explanation concordanceterm·dom, likely reflecting high internal consistency in its probabilistic language model.

Explanations generated by ChatGPTterm·dom contain nonobvious insights Having established the accuracyterm·dom and concordance of ChatGPTterm·dom, we next examined its potential to augment human learning in the domain of medical educationterm·dom. AI-generated explanations were independently adjudicated by 2 physician reviewers. Explanation content was examined for significant insightsterm·dom, defined as instances that met the criteria (see S2 Data ) of noveltyterm·dom , nonobviousnessterm·dom , and validityterm·dom . The perspective of the target test audience was adopted by the adjudicator, as a second-year medical student for Step 1term·dom, fourth-year medical student for Step 2CKterm·dom, and post-graduate year 1 resident for Step 3term·dom.

We first examined the frequency (prevalence) of insight. Overall, ChatGPTterm·dom produced at least one significant insightterm·dom in 88.9% of all responses. Insight frequency was generally consistent between exam type and question input format ( Fig 3C ). In Step 2CKterm·dom however, insight decreased by 10.3% ( n = 11 items) between MC-NJterm·mth and MC-Jterm·mth formulations, paralleling the decrement in accuracyterm·dom ( Fig 1B ). Review of this subset of questions did not reveal a discernible pattern for the paradoxical decrease (see specifically annotated items [*] in S1 Data ).

Next, we quantified the density of insight (DOI)term·mth contained within AI-generated explanations. A density index was defined by normalizing the number of unique insights against the number of possible answer choices. This analysis was performed on MC-Jterm·mth entries only. High quality outputsterm·dom were generally characterized by DOI >0.6 (i.e. unique, novel, nonobvious, and valid insights provided for >3 out of 5 choices); low quality outputsterm·dom were generally characterized by DOI ≤0.2. The upper limit on DOI is only bounded by the maximum length of text output. Across all exam types, we observed that mean DOI was significantly higher in questions items answered accurately versus inaccurately (0.458 versus 0.199, p <0.0001) ( Fig 3D ).

The high frequency and moderate density of insights indicate that it may be possible for a target learner (e.g., such as a second-year medical student preparing for Step 1term·dom) to gain new or remedial knowledge from the ChatGPTterm·dom AI output, particularly if answering incorrectly.

Discussion In this study, we provide new and surprising evidence that ChatGPTterm·dom is able to perform several intricate tasks relevant to handling complex medical and clinical information. To assess ChatGPTterm·dom’s capabilities against biomedical and clinical questions of standardized complexity and difficulty, we tested its performance characteristics on the United States Medical Licensing Examination (USMLE).

Our findings can be organized into two major themes: (1) the rising accuracyterm·dom of ChatGPTterm·dom, which approaches or exceeds the passing thresholdterm·dom for USMLE; and (2) the potential for this AI to generate novel insights that can assist human learners in a medical educationterm·dom setting.

The rising accuracyterm·dom of ChatGPTterm·dom The most recent iteration of the GPT LLM (GPT3) achieved 46% accuracyterm·dom with zero prompting [ 12 ], which marginally improved to 50% with further model training and extensive prompt tuning. Previous models, merely months prior, performed at 36.7% [ 13 ]. In this present study, ChatGPTterm·dom performed at >50% accuracyterm·dom across all examinations, exceeding 60% in some analyses. The USMLE pass threshold, while varying by year, is approximately 60%. Therefore, ChatGPTterm·dom now approaches the passing range. Being the first experimentterm·mth to reach this benchmark, we believe this is a surprising and impressive result. Moreover, we provided no prompting or training to the AI, minimized grounding bias by expunging the AI session prior to inputting each question variant, and avoided chain-of-thought biasing by requesting forced justification only as the final input. Further model interaction and prompt tuning could often produce more accurate results. Given this trajectory, it is likely that AI performance will continue to improve as LLM models continue to mature.

Paradoxically, ChatGPTterm·dom outperformed PubMedGPTterm·mth [ 14 ] (accuracyterm·dom 50.3%), a counterpart LLM with similar neural structure, but trained exclusively on biomedical domain literature. We speculate that domain-specific training may have created greater ambivalence in the PubMedGPTterm·mth model, as it absorbs real-world text from ongoing academic discourse that tends to be inconclusive, contradictory, or highly conservative or noncommittal in its language. A foundation LLM trained on general content, such as ChatGPTterm·dom, may therefore have an advantage because it is also exposed to broader clinical content, such as patient-facing disease primers and provider-facing drug package inserts, that are more definitive and congruent.

An additional explanation for the observed difference in performance may be the disparate AI testing datasets. Our present study tested ChatGPTterm·dom against contemporary USMLE examinations (publicly available no earlier than 2022, 5 answer choices per question), whereas previous reports tested language models against the MedQA-USMLE dataset [ 13 ] (publicly available 2009–2020, 4 answer choices per question). Although we did not perform a direct comparison against MedQA-UMSLE, our approach nonetheless has several advantages. It is guaranteed that none of our inputs were previously seen by GPT3, whereas many of the inputs from MedQA-USMLE would have likely been ingested during model pretraining. Considering that medical knowledge proliferates at a faster-than-exponential rate [ 15 ] and previous evidence-based practice is frequently debunked [ 16 , 17 ], some concepts tested by MedQA-USMLE are already antiquated and not representative of present-day examination content. Finally, the higher accuracyterm·dom of ChatGPTterm·dom on USMLE-2022 despite a greater number of answer choices (5 versus 4) may indicate even more impressive performance of this model relative to other domain-specific language models such as PubMedGPTterm·mth and BioBERT.

Consistent with the mechanism of generative language models, we observed that the accuracyterm·dom of ChatGPTterm·dom was strongly mediated by concordance and insight. High accuracyterm·dom outputs were characterized by high concordance and high density of insight. Poorer accuracyterm·dom was characterized by lower concordance and a poverty of insight. Therefore, inaccurate responsesterm·dom were driven primarily by missing informationterm·dom, leading to diminished insight and indecisionterm·dom in the AI, rather than overcommitment to the incorrect answer choice. These findings indicate that model performance could be significantly improved by merging foundation modelsterm·mth, such as ChatGPTterm·dom, with a domain-specific LLMterm·mth or other model trained on a voluminous and highly validated medical knowledge resources, such as UpToDate, or other ACGME-accredited content.

Interestingly, the accuracyterm·dom of ChatGPTterm·dom tended to be lowest for Step 1term·dom, followed by Step 2CKterm·dom, followed by Step 3term·dom. This mirrors both the subjective difficulty and objective performance for real-world test takers on Step 1term·dom, which is collectively regarded as the most difficult exam of the series. The low accuracyterm·dom on Step 1term·dom could be explained by an undertrained model on the input side (e.g. underrepresentation of basic science content on the general information space) and/or the human side (e.g. insufficient or invalid human judgment at initial reinforcement stages). This result exposes a key vulnerability in pre-trained LLMs, such as ChatGPTterm·dom: AI ability becomes yoked to human ability. ChatGPTterm·dom’s performance on Step 1term·dom is poorer precisely because human users perceive its subject matter (e.g., pathophysiology) as more difficult or opaque.

The potential for AI-assisted human learning in medical educationterm·dom We also examined the ability of ChatGPTterm·dom to assist the human learning processterm·dom of its target audience (e.g., a second year medical student preparing for USMLE Step 1term·dom). As a proxy for the metric of helpfulness, we assessed the concordance and insight offered by the AI explanation outputs. ChatGPTterm·dom responses were highly concordant, such that a human learner could easily follow the internal language, logic, and directionality of relationships contained within the explanation text (e.g., adrenal hyper cortisolism ⥬ increased bone osteoclast activity ⥬ increased calcium resorption ⥬ decreased bone mineral density ⥬ increased fracture risk). High internal concordance and low self-contradictionterm·dom is a proxy of sound clinical reasoningterm·dom and an important metric of explanation quality. It is reassuring that the directionality of relationships is preserved by the language processing model, where each verbal object is individually lemmatized.

AI-generated responses also offered significant insightterm·dom, role-modeling a deductive reasoning processterm·dom valuable to human learners. At least one significant insightterm·dom was present in approximately 90% of outputs. ChatGPTterm·dom therefore possesses the partial ability to teach medicine by surfacing novel and nonobvious concepts that may not be in learners’ sphere of awareness. This qualitative gain provides a basis for future real-world studies on the efficacy of generative AI to augment the human medical educationterm·dom process. For example, longitudinal exam performance can be studied in a quasi-controlled in AI-assisted and unassisted learners. Unit economic analysis may clarify the cost-effectiveness of incremental student performance gain in comparison to existing tools such as virtual tutors and study aids.

Medical educationterm·dom, licensing examinations, and test preparation services form a large industrial complex eclipsing a nine-figure market size annually. While its relevance remains debated, standardized testing has emerged as an important end-target of medical learning. In parallel, of the didactic techniques, a socratic teaching style is favored by medical students [ 18 ]. The rate-limiting step for fresh content generation is the human cognitive effort required to craft realistic clinical vignettes that probe “high-yield” concepts in a subtle way, engage critical thinking, and offer pearls of knowledge even if answered incorrectly. Demand for new examination content continues to increase. Future studies may investigate the ability of generative language AI to offload this human effort by assisting in the question-explanation writing processterm·dom or, in some cases, writing entire items autonomously.

Finally, the advent of AI in medical educationterm·dom demands an open science research infrastructureterm·dom to standardize experimental methods, readouts, and benchmarks to describe and quantify human-AI interactionsterm·dom. Multiple dimensions must be covered, including user experience, learning environment, hybridization with other teaching modes, and effect on cognitive bias. In this report, we provide an initial basic protocol for adjudicating AI-generated responses along axes of accuracyterm·dom, concordance, and insight.

Our study has several important limitations. The relatively small input size restricted the depth and range of analyses. For example, stratifying the output of ChatGPTterm·dom by subject taxonomy (e.g., pharmacology, bioethics) or competency type (e.g., differential diagnosis, management) may be of great interest to medical educators, and could reveal heterogeneities in performance across language processing for different clinical reasoningterm·dom tasks. Similarly, a more robust AI failure mode analysisterm·mth (e.g., language parsing error) may lend insight into the etiology of inaccuracy and discordance. In addition to being laborious, human adjudicationterm·mth is error-prone and subject to greater variability and bias. Future studies will undoubtedly apply unbiased approaches, using quantitative natural language processing and text mining tools such as word network analysis. In addition to increasing validityterm·dom and accelerating throughput by several orders of magnitude, these methods are likely to better characterize the depth, coherence, and learning value of AI output. Finally, to truly assess the utility of generative language AI for medical educationterm·dom, ChatGPTterm·dom and related applications must be studied in both controlled and real-world learning scenariosterm·dom with students across the engagement and knowledge spectrum.

Beyond their utility for medical educationterm·dom, AIs are now positioned to soon become ubiquitous in clinical practice, with diverse applications across all healthcare sectors. Investigation of AI has now entered into the era of randomized controlled trials [ 19 ]. Additionally, a profusion of pragmatic and observational studies supports a versatile role of AI in virtually all medical disciplines and specialties by improving risk assessment [ 20 , 21 ], data reduction, clinical decision support [ 22 , 23 ], operational efficiency, and patient communication [ 24 , 25 ].

Inspired by the remarkable performance of ChatGPTterm·dom on the USMLE, clinicians within AnsibleHealth, a virtual chronic pulmonary disease clinic, have begun to experimentterm·mth with ChatGPTterm·dom as part of their workflows. Inputting queries in a secure and de-identified manner, our clinicians request ChatGPTterm·dom to assist with traditionally onerous writing tasks such as composing appeal letters to payors, simplifying radiology reports (and other jargon-dense records) to facilitate patient comprehension, and even to brainstorm and kindle insight when faced with nebulous and diagnostically challenging cases. We believe that LLMs such as ChatGPTterm·dom are reaching a maturity level that will soon impact clinical medicine at large, enhancing the delivery of individualized, compassionate, and scalable healthcare.

Supporting information (PDF) Click here for additional data file.

(PDF) Click here for additional data file.

(PDF) Click here for additional data file.

(PDF) Click here for additional data file.