Automatic diagnosis of the 12-lead ECG using a deep neural network. — TEI-tagged view

Antônio H. RibeiropersName, Manoel Horta RibeiropersName, Gabriela M. M. PaixãopersName, Derick M. OliveirapersName, Paulo R. GomespersName, Jéssica A. CanazartpersName, Milton P. S. FerreirapersName, Carl R. AnderssonpersName, Peter W. MacfarlanepersName, Wagner MeirapersName, Thomas B. SchönpersName, Antonio Luiz P. RibeiropersName (2020date). “Automatic diagnosis of the 12-lead ECG using a deep neural network..” Nature Communicationstitle·j.

Object of study — deep-learning classification of six abnormalities in 12-lead ECGterm·dom exams

Research question — whether a DNNterm·mth trained on millions of labeled 12-lead ECGsterm·dom can match or beat physicians

Contribution — a DNNterm·mth trained on over 2 million labeled ECGsterm·key that recognizes 6 abnormalities with F1 above 80%

Full text — tagged

Automatic diagnosis of the 12-lead ECGterm·dom using a deep neural networkterm·key.

Abstract The role of automatic electrocardiogram (ECGterm·key) analysis in clinical practice is limited by the accuracy of existing models. Deep Neural Networksterm·mth (DNNsterm·mth) are models composed of stacked transformations that learn tasks by examples. This technology has recently achieved striking success in a variety of task and there are great expectations on how it might improve clinical practice. Here we present a DNNterm·mth model trained in a dataset with more than 2 million labeled exams analyzed by the Telehealth Network of Minas Geraisterm·key and collected under the scope of the CODEterm·key (Clinical Outcomes in Digital Electrocardiology) study. The DNNterm·mth outperform cardiology resident medical doctors in recognizing 6 types of abnormalities in 12-lead ECGterm·dom recordings, with F1 scoresterm·var above 80% and specificity over 99%. These results indicate ECGterm·key analysis based on DNNsterm·mth, previously studied in a single-lead setup, generalizes well to 12-leadterm·key exams, taking the technology closer to the standard clinical practice.

Subject terms: Cardiology, Electrodiagnosis, Computational science Introduction Cardiovascular diseases are the leading cause of death worldwide 1 and the electrocardiogram (ECGterm·key) is a major tool in their diagnoses. As ECGsterm·key transitioned from analog to digital, automated computer analysis of standard 12-leadterm·key electrocardiograms gained importance in the process of medical diagnosis 2 , 3 . However, limited performance of classical algorithms 4 , 5 precludes its usage as a standalone diagnostic tool and relegates them to an ancillary role 3 , 6 .

Deep neural networksterm·mth (DNNsterm·mth) have recently achieved striking success in tasks such as image classification 7 and speech recognition 8 , and there are great expectations when it comes to how this technology may improve health care and clinical practice 9 – 11 . So far, the most successful applications used a supervised learningterm·key setup to automate diagnosis from exams. Supervised learning modelsterm·mth, which learn to map an input to an output based on example input−output pairs, have achieved better performance than a human specialist on their routine work-flow in diagnosing breast cancer 12 and detecting retinal diseases from three-dimensional optical coherence tomography scans 13 . While efficient, training DNNsterm·mth in this setup introduces the need for large quantities of labeled data which, for medical applications, introduce several challenges, including those related to confidentiality and security of personal health information 14 .

A convincing preliminary study of the use of DNNsterm·mth in ECGterm·key analysis was recently presented in ref. 15 . For single-lead ECGsterm·key, DNNsterm·mth could match state-of-the-art algorithms when trained in openly available datasets (e.g. 2017 PhysioNet Challenge data 16 ) and, for a large enough training dataset, present superior performance when compared to practicing cardiologists. However, as pointed out by the authors, it is still an open question if the application of this technology would be useful in a realistic clinical setting, where 12-lead ECGsterm·dom are the standard technique 15 .

The short-duration, standard, 12-lead ECGterm·dom (S12L-ECGterm·dom) is the most commonly used complementary exam for the evaluation of the heart, being employed across all clinical settings, from the primary care centers to the intensive care units. While long-term cardiac monitoring, such as in the Holter exam, provides information mostly about cardiac rhythm and repolarization, the S12L-ECGterm·dom can provide a full evaluation of the cardiac electrical activity. This includes arrhythmias, conduction disturbances, acute coronary syndromes, cardiac chamber hypertrophy and enlargement and even the effects of drugs and electrolyte disturbances. Thus, a deep learning approach that allows for accurate interpretation of S12L-ECGsterm·dom would have the greatest impact.

S12L-ECGsterm·dom are often performed in settings, such as in primary care centers and emergency units, where there are no specialists to analyze and interpret the ECGterm·key tracings. Primary care and emergency department health professionals have limited diagnostic abilities in interpreting S12-ECGs 17 , 18 . The need for an accurate automatic interpretation is most acute in low and middle-income countries, which are responsible for more than 75% of deaths related to cardiovascular disease 19 , and where the population, often, do not have access to cardiologists with full expertise in ECGterm·key diagnosis.

The use of DNNsterm·mth for S12L-ECGterm·dom is still largely unexplored. A contributing factor for this is the shortage of full digital S12L-ECGterm·dom databases, since most recordings are still registered only on paper, archived as images, or stored in PDF format 20 . Most available databases comprise a few hundreds of tracings and no systematic annotation of the full list of ECGterm·key diagnoses 21 , limiting their usefulness as training datasets in a supervised learningterm·key setting. This lack of systematically annotated data is unfortunate, as training an accurate automatic method of diagnosis from S12L-ECGterm·dom would be greatly beneficial.

In this paper, we demonstrate the effectiveness of DNNsterm·mth for automatic S12L-ECGterm·dom classification. We build a large-scale dataset of labeled S12L-ECGterm·dom exams for clinical and prognostic studies (the CODEterm·key—Clinical Outcomes in Digital Electrocardiology study) and use it to develop a DNNterm·mth to classify six types of ECGterm·key abnormalities considered representative of both rhythmic and morphologic ECGterm·key abnormalities.

Results Model specification and training We collected a dataset consisting of 2,322,513 ECGterm·key records from 1,676,384 different patients of 811 counties in the state of Minas Gerais/Brazil from the Telehealth Network of Minas Geraisterm·key (TNMG) 22 . The dataset characteristics are summarized in Table 1 . The acquisition and annotation procedures of this dataset are described in Methods. We split this dataset into a training set and a validation set. The training set contains 98% of the data. The validation set consists of the remaining 2% (~50,000 exams) of the dataset and it was used for hyperparameter tuning.

(Dataset summary) Patient characteristics and abnormalities prevalence, n (%).

We train a DNNterm·mth to detect: 1st degree AV block (1dAVb), right bundle branch block (RBBB), left bundle branch block (LBBB), sinus bradycardia (SB), atrial fibrillationterm·key (AF) and sinus tachycardia (ST). These six abnormalities are displayed in Fig. 1 .

Abnormalities examples. A list of all the abnormalities the model classifies. We show only three representative leads (DII, V1 and V6).

We used a DNNterm·mth architecture known as the residual networkterm·mth 23 , commonly used for images, which we here have adapted to unidimensional signals. A similar architecture has been successfully employed for detecting abnormalities in single-lead ECGterm·key signals 15 . Furthermore, in the 2017 Physionet challenge 16 , algorithms for detecting AF have been compared in an open dataset of single-lead ECGsterm·key and both the architecture described in ref. 15 and other convolutional architectures 24 , 25 have achieved top scores.

The DNNterm·mth parameters were learned using the training dataset and our design choices were made in order to maximize the performance on the validation dataset. We should highlight that, despite using a significantly larger training dataset, we got the best validation results with an architecture with, roughly, one quarter the number of layers and parameters of the network employed in ref. 15 .

Testing and performance evaluation For testing the model we employed a dataset consisting of 827 tracings from distinct patients annotated by three different cardiologists with experience in electrocardiography (see Methods). The test dataset characteristics are summarized in Table 1 . Table 2 shows the performance of the DNNterm·mth on the test set. High-performance measures were obtained for all ECGterm·key abnormalities, with F1 scoresterm·var above 80% and specificity indexes over 99%. We consider our model to have predicted the abnormality when its output—a number between 0 and 1—is above a threshold. Figure 2 shows the precision-recall curve for our model, for different values of this threshold.

(Performance indexes) Scores of our DNNterm·mth are compared on the test set with the average performance of: (i) 4th year cardiology resident (cardio.); (ii) 3rd year emergency resident (emerg.); and (iii) 5th year medical students (stud.).

Precision-recall curve. Show precision-recall curve for our nominal prediction model on the test set (strong line) with regard to each ECGterm·key abnormalities. The shaded region shows the range between maximum and minimum precision for neural networks trained with the same configuration and different initialization. Points corresponding to the performance of resident medical doctors and students are also displayed, together with the point corresponding to the DNNterm·mth performance for the same threshold used for generating Table 2 . Gray dashed curves in the background correspond to iso- F 1 curves (i.e. curves in the precision-recall plane with constant F1 scoreterm·key).

Neural networks are initialized randomly, and different initialization usually yield different results. In order to show the stability of the method, we have trained ten neural networks with the same set of hyperparameters and different initializations. The range between the maximum and minimum precision among these realizations, for different values of threshold, are the shaded regions displayed in Fig. 2 . These realizations have micro average precision (mAP) between 0.946 and 0.961; we choose the one with mAP immediately above the median value of all executions (the one with mAP = 0.951) (We couldn’t choose the model with mAP equal to the median value because 10 is an even number; hence, there is no single middle value.). All the analyses from now on will be for this realization of the neural network, which correspond both to the strong line in Fig. 2 and to the scores presented in Table 2 . For this model, Fig. 2 shows the point corresponding to the maximum F1 scoreterm·key for each abnormality. The threshold corresponding to this point is used for producing the DNNterm·mth scores displayed in Table 2 .

The same dataset was evaluated by: (i) two 4th year cardiology residents; (ii) two 3rd year emergency residents; and (iii) two 5th year medical students. Each one annotated half of the exams in the test set. Their average performances are given, together with the DNNterm·mth results, in Table 2 and their precision-recall scores are plotted in Fig. 2 . Considering the F1 scoreterm·key, the DNNterm·mth matches or outperforms the medical residents and students for all abnormalities. The confusion matrices and the inter-rater agreement (kappa coefficients) for the DNNterm·mth, the resident medical doctors and students are provided, respectively, in Supplementary Tables 1 and 2(a) . Additionally, in Supplementary Table 2(b) , we compare the inter-rater agreement between the neural network and the certified cardiologists that annotated the test set.

A trained cardiologist reviewed all the mistakes made by the DNNterm·mth, the medical residents and the students, trying to explain the source of the error. The cardiologist had meetings with the residents and students where they together agreed on which was the source of the error. The results of this analysis are given in Table 3 .

(Error analysis) Present the analysis of misclassified exams.

In order to compare the performance difference between the DNNterm·mth and resident medical doctors and students, we compute empirical distributions for the precision (PPV), recall (sensitivity), specificity and F1 scoreterm·key using bootstrapping 26 . The boxplots corresponding to these bootstrapped distributions are presented in Supplementary Fig. 1 . We have also applied the McNemar test 27 to compare the misclassification distribution of the DNNterm·mth, the medical residents and the students. Supplementary Table 3 shows the p values of the statistical test. Both analyses do not indicate a statistically significant difference in performance among the DNNterm·mth and the medical residents and students for most of the classes.

Finally, to assess the effect of how we structure our problem, we have considered alternative scenarios where we use the 2,322,513 ECGterm·key records in 90%5%-5% splits, stratified randomly, by patient or in chronological order. Being the splits used, respectively, for training, validation and as a second larger test set. The results indicate no statistically significant difference between the original DNNterm·mth used in our analysis and the alternative models developed in the 90%-5%-5% splits. The exception is the model developed using the chronologically order split, for which the changes along time in the telehealth center operation have affected the splits (cf. Supplementary Fig. 2 ).

Discussion This paper demonstrates the effectiveness of “end-to-endterm·key” automatic S12L-ECGterm·dom classification. This presents a paradigm shift from the classical ECGterm·key automatic analysis methods 28 . These classical methods, such as the University of Glasgow ECGterm·key analysis program 29 , first extract the main features of the ECGterm·key signal using traditional signal processing techniques and then use these features as inputs to a classifier. End-to-end learningterm·mth presents an alternative to these two-step approachesterm·mth, where the raw signal itself is used as an input to the classifier which learns, by itself, to extract the features. This approach has presented, in a emergency room setting, performance superior to commercial ECGterm·key software based on traditional signal processing techniques 30 .

Neural networks have previously been used for classification of ECGsterm·key both in a classical—feature-based—setup 31 , 32 and in an end-to-endterm·key learn setup 15 , 33 , 34 . Hybrid methods combining the two paradigms are also available: the classification may be done using a combination of handcrafted and learned features 35 or by using a two-stage training, obtaining one neural network to learn the features and another to classify the exam according to these learned features 36 .

The paradigm shift towards end-to-end learningterm·mth had a significant impact on the size of the datasets used for training the models. Many results using classical methods 28 , 34 , 36 train their models on datasets with few examples, such as the MIT-BIH arrhythmia database 37 , with only 47 unique patients. The most convincing papers using end-to-endterm·key deep learning or mixed approaches, on the other hand, have constructed large datasets, ranging from 3000 to 100,000 unique patients, for training their models 15 , 16 , 30 , 35 .

Large datasets from previous work 15 , 16 , 35 , however, either were obtained from cardiac monitors and Holter exams, where patients are usually monitored for several hours and the recordings are restricted to one or two leads or consist of 12-lead ECGsterm·dom obtained in an emergency room setting 30 , 38 . Our dataset with well over 2 million entries, on the other hand, consists of short duration (7−10 s) S12L-ECGterm·dom tracings obtained from in-clinic exams and is orders of magnitude larger than those used in previous studies. It encompasses not only rhythm disorders, like AF, SB and ST, as in previous studies 15 , but also conduction disturbances, such as 1dAVb, RBBB and LBBB. Instead of beat-to-beat classification, as in the MIT-BIH arrhythmia database, our dataset provides annotation for S12L-ECGterm·dom exams, which are the most common in clinical practice.

The availability of such a large database of S12L-ECGterm·dom tracings, with annotation for the whole spectrum of ECGterm·key abnormalities, opens up the possibility of extending initial results of end-to-endterm·key DNNterm·mth in ECGterm·key automatic analysis 15 to a system with applicability in a wide range of clinical settings. The development of such technologies may yield high-accuracy automatic ECGterm·key classification systems that could save clinicians considerable time and prevent wrong diagnoses. Millions of S12L-ECGsterm·dom are performed every year, many times in places where there is a shortage of qualified medical doctors to interpret them. An accurate classification system could help to detect wrong diagnoses and improve the access of patients from deprived and remote locations to this essential diagnostic tool of cardiovascular diseases.

The error analysis shows that most of the DNNterm·mth mistakes were related to measurements of ECGterm·key intervals. Most of those were borderline cases, where the diagnosis relies on a consensus definitions 39 that can only be ascertained when a measurement is above a sharp cutoff point. The mistakes can be explained by the DNNterm·mth failing to encode these very sharp thresholds. For example, the DNNterm·mth wrongly detecting an SB with a heart rate slightly above 50 bpm or an ST with a heart rate slightly below 100 bpm. Supplementary Fig. 3 illustrates this effect. Noise and interference in the baseline are established causes of error 40 and affected both automatic and manual diagnosis of ECGterm·key abnormalities. Nevertheless, the DNNterm·mth seems to be more robust to noise and it made fewer mistakes of this type compared to the medical residents and students. Conceptual errors (where our reviewer suggested that the doctor failed to understand the definitions of each abnormality) were more frequent for emergency residents and medical students than for cardiology residents. Attention errors (where we believe that the error could have been avoided if the manual reviewer were more careful) were present at a similar ratio for cardiology residents, emergency residents and medical students.

Interestingly, the performance of the emergency residents is worse than the medical students for many abnormalities. This might seem counter-intuitive because they have less years of medical training. It might, however, be justified by the fact that emergency residents, unlike cardiology residents, do not have to interpret these exams on a daily basis, while medical students still have these concepts fresh from their studies.

Our work is perhaps best understood in the context of its limitations. While we obtained the highest F1 scoresterm·var for the DNNterm·mth, the McNemar statistical test and bootstrapping suggest that we do not have confidence enough to assert that the DNNterm·mth is actually better than the medical residents and students with statistical significance. We attribute this lack of confidence in the comparison to the presence of relatively infrequent classes, where a few erroneous classifications may significantly affect the scores. Furthermore, we did not test the accuracy of the DNNterm·mth in the diagnosis of other classes of abnormalities, like those related to acute coronary syndromes or cardiac chamber enlargements and we cannot extend our results to these untested clinical situations. Indeed, the real clinical setting is more complex than the experimental situation tested in this study and, in complex and borderline situations, ECGterm·key interpretation can be extremely difficult and may demand the input of highly specialized personnel. Thus, even if a DNNterm·mth is able to recognize typical ECGterm·key abnormalities, further analysis by an experienced specialist will continue to be necessary to these complex exams.

This proof-of-concept study, showing that a DNNterm·mth can accurately recognize ECGterm·key rhythm and morphological abnormalities in clinical S12L-ECGterm·dom exams, opens a series of perspectives for future research and clinical applications. A next step would be to prove that a DNNterm·mth can effectively diagnose multiple and complex ECGterm·key abnormalities, including myocardial infarction, cardiac chamber enlargement and hypertrophy and less common forms of arrhythmia, and to recognize a normal ECGterm·key. Subsequently, the algorithm should be tested in a controlled real-life situation, showing that accurate diagnosis could be achieved in real time, to be reviewed by clinical specialists with solid experience in ECGterm·key diagnosis. This real-time, continuous evaluation of the algorithm would provide rapid feedback that could be incorporated as further improvements of the DNNterm·mth, making it even more reliable.

The TNMG, the large telehealth service from which the dataset used was obtained 22 , is a natural laboratory for these next steps, since it performs more than 2000 ECGsterm·key a day and it is currently expanding its geographical coverage over a large part of a continental country (Brazil). An optimized system for ECGterm·key interpretation, where most of the classification decisions are made automatically, would imply that the cardiologists would only be needed for the more complex cases. If such a system is made widely available, it could be of striking utility to improve access to health care in low- and middle-income countries, where cardiovascular diseases are the leading cause of death and systems of care for cardiac diseases are lacking or not working well 41 .

In conclusion, we developed an end-to-endterm·key DNNterm·mth capable of accurately recognizing six ECGterm·key abnormalities in S12L-ECGterm·dom exams, with a diagnostic performance at least as good as medical residents and students. This study shows the potential of this technology, which, when fully developed, might lead to more reliable automatic diagnosis and improved clinical practice. Although expert review of complex and borderline cases seems to be necessary even in this future scenario, the development of such automatic interpretation by a DNNterm·mth algorithm may expand the access of the population to this basic and useful diagnostic exam.

Methods Dataset acquisition All S12L-ECGsterm·dom analyzed in this study were obtained by the Telehealth Network of Minas Geraisterm·key (TNMG), a public telehealth system assisting 811 out of the 853 municipalities in the state of Minas Gerais, Brazil 22 . Since September 2017, the TNMG has also provided telediagnostic services to other Brazilian states in the Amazonian and Northeast regions. The S12L-ECGterm·dom exam was performed mostly in primary care facilities using a tele-electrocardiograph manufactured by Tecnologia Eletrônica Brasileira (São Paulo, Brazil)—model TEB ECGPC—or Micromed Biotecnologia (Brasilia, Brazil)—model ErgoPC 13. The duration of the ECGterm·key recordings is between 7 and 10 s sampled at frequencies ranging from 300 to 600 Hz. A specific software developed in-house was used to capture the ECGterm·key tracings, to upload the exam together with the patient’s clinical history and to send it electronically to the TNMG analysis center. Once there, one cardiologist from the TNMG experienced team analyzes the exam and a report is made available to the health service that requested the exam through an online platform.

We have incorporated the University of Glasgow (Uni-G) ECGterm·key analysis program (release 28.5, issued in January 2014) in the in-house software since December 2017. The analysis program was used to automatically identify waves and to calculate axes, durations, amplitudes and intervals, to perform rhythm analysis and to give diagnostic interpretation 29 , 42 . The Uni-G analysis program also provides Minnesota codesterm·key 43 , a standard ECGterm·key classification used in epidemiological studies 44 . Since April 2018 the automatic measurements are being shown to the cardiologists that give the medical report. All clinical information, digital ECGsterm·key tracings and the cardiologist report were stored in a database. All previously stored data were also analyzed by Uni-G software in order to have measurements and automatic diagnosis for all exams available in the database, since the first recordings. The CODEterm·key study was established to standardize and consolidate this database for clinical and epidemiological studies. In the present study, the data (for patients above 16 years old) obtained between 2010 and 2016 were used in the training and validation set and, from April to September 2018, in the test set.

Labeling training data from text report For the training and validation sets, the cardiologist report is available only as a textual description of the abnormalities in the exam. We extract the label from this textual report using a three-step procedure. First, the text is preprocessed by removing stop-words and generating n-grams from the medical report. Then, the Lazy Associative Classifier (LAC) 45 , trained on a 2800-sample dictionary created from real diagnoses text reports, is applied to the n-grams. Finally, the text label is obtained using the LAC result in a rule-based classifier for class disambiguation. The classification model reported above was tested on 4557 medical reports manually labeled by a certified cardiologist who was presented with the free-text and was required to choose among the prespecified classes. The classification step recovered the true medical label with good results; the macro F1 scoreterm·key achieved were: 0.729 for 1dAVb; 0.849 for RBBB; 0.838 for LBBB; 0.991 for SB; 0.993 for AF; 0.974 for ST.

Training and validation set annotation To annotate the training and validation datasets, we used: (i) the Uni-G statements and Minnesota codesterm·key obtained by the Uni-G automatic analysis (automatic diagnosis); (ii) automatic measurements provided by the Uni-G software; and (iii) the text labels extracted from the expert text reports using the semi-supervised methodology (medical diagnosis). Both the automatic and medical diagnosis are subject to errors: automatic classification has limited accuracy 3 – 6 and text labels are subject to errors of both the practicing expert cardiologists and the labeling methodology. Hence, we combine the expert annotation with the automatic analysis to improve the quality of the dataset. The following procedure is used for obtaining the ground truth annotation: We: Accept a diagnosis (consider an abnormality to be present) if both the expert and either the Uni-G statement or the Minnesota codeterm·key provided by the automatic analysis indicated the same abnormality. Reject a diagnosis (consider an abnormality to be absent) if only one automatic classifier indicates the abnormality in disagreement with both the doctor and the other automatic classifier. After this initial step, there are two scenarios where we still need to accept or reject diagnoses. They are: (i) both classifiers indicate the abnormality, but the expert does not; or (ii) only the expert indicates the abnormality, whereas none of the classifiers indicates anything.

We used the following rules to reject some of the remaining diagnoses: Diagnoses of ST where the heart rate was below 100 (8376 medical diagnoses and 2 automatic diagnoses) were rejected. Diagnoses of SB where the heart rate was above 50 (7361 medical diagnoses and 16,427 automatic diagnosis) were rejected. Diagnoses of LBBB or RBBB where the duration of the QRS interval was below 115 ms (9313 medical diagnoses for RBBB and 8260 for LBBB) were rejected. Diagnoses of 1dAVb where the duration of the PR interval was below 190 ms (3987 automatic diagnoses) were rejected.

Then, using the sensitivity analysis of 100 manually reviewed exams per abnormality, we came up with the following rules to accept some of the remaining diagnoses: For RBBB, d1AVb, SB and ST, we accepted all medical diagnoses. 26,033, 13,645, 12,200 and 14,604 diagnoses were accepted in this fashion, respectively. For AF, we required not only that the exam was classified by the doctors as true, but also that the standard deviation of the NN intervals was higher than 646. 14,604 diagnoses were accepted using this rule. According to the sensitivity analysis, the number of false positives that would be introduced by this procedure was smaller than 3% of the total number of exams.

After this process, we were still left with 34,512 exams where the corresponding diagnoses could neither be accepted nor rejected. These were manually reviewed by medical students using the Telehealth ECGterm·key diagnostic system, under the supervision of a certified cardiologist with experience in ECGterm·key interpretation. The process of manually reviewing these ECGsterm·key took several months.

It should be stressed that information from previous medical reports and automatic measurements were used only for obtaining the ground truth for training and validation sets and not on later stages of the DNNterm·mth training.

Test set annotation The dataset used for testing the DNNterm·mth was also obtained from TNMG’s ECGterm·key system. It was independently annotated by two certified cardiologists with experience in electrocardiography. The kappa coefficients 46 indicate the inter-rater agreement for the two cardiologists and are: 0.741 for 1dAVb; 0.955 for RBBB; 0.964 for LBBB; 0.844 for SB; 0.831 for AF; 0.902 for ST. When they agreed, the common diagnosis was considered as ground truth. In cases where there was any disagreement, a third senior specialist, aware of the annotations from the other two, decided the diagnosis. The American Heart Association standardization 47 was used as the guideline for the classification.

It should be highlighted that the annotation was performed in an upgraded version of the TNMG software, in which the automatic measurements obtained by the Uni-G program are presented to the specialist, that has to choose the ECGterm·key diagnosis among a number of prespecified classes of abnormalities. Thus, the diagnosis was codified directly into our classes and there was no need to extract the label from a textual report, as it was done for the training and validation sets.

Neural network architecture and training We used a convolutional neural network similar to the residual networkterm·mth 23 , but adapted to unidimensional signals. This architecture allows DNNsterm·mth to be efficiently trained by including skip connections. We have adopted the modification in the residual block proposed in ref. 48 , which place the skip connection in the position displayed in Fig. 3 .

(DNNterm·mth architecture). The unidimensional residual neural network architecture used for ECGterm·key classification.

All ECGterm·key recordings are resampled to a 400 Hz sampling rate. The ECGterm·key recordings, which have between 7 and 10 s, are zero-padded resulting in a signal with 4096 samples for each lead. This signal is the input for the neural network.

The network consists of a convolutional layer (Conv) followed by four residual blocks with two convolutional layers per block. The output of the last block is fed into a fully connected layer (Dense) with a sigmoid activation function, σ , which was used because the classes are not mutually exclusive (i.e. two or more classes may occur in the same exam). The output of each convolutional layer is rescaled using batch normalization, (BN) 49 , and fed into a rectified linear activation unit (ReLU). Dropout 50 is applied after the nonlinearity.

The convolutional layers have filter length 16, starting with 4096 samples and 64 filters for the first layer and residual block and increasing the number of filters by 64 every second residual block and subsampling by a factor of 4 every residual block. Max Pooling 51 and convolutional layers with filter length 1 (1x1 Conv) are included in the skip connections to make the dimensions match those from the signals in the main branch.

The average cross-entropy is minimized using the Adam optimizer 52 with default parameters and learning rate lr = 0.001. The learning rate is reduced by a factor of 10 whenever the validation loss does not present any improvement for seven consecutive epochs. The neural network weights was initialized as in ref. 53 and the bias was initialized with zeros. The training runs for 50 epochs with the final model being the one with the best validation results during the optimization process.

Hyperparameter tuning This final architecture and configuration of hyperparameters was obtained after approximately 30 iterations of the procedure: (i) find the neural network weights in the training set; (ii) check the performance in the validation set; and (iii) manually choose new hyperparameters and architecture using insight from previous iterations. We started this procedure from the set of hyperparameters and architecture used in ref. 15 . It is also important to highlight that the choice of architecture and hyperparameters was done together with improvements in the dataset. Expert knowledge was used to take decision about how to incorporate, on the manual tuning procedure, information about previous iteration that were evaluated on slightly different versions of the dataset.

The hyperparameters were chosen among the following options: residual neural networks with {2, 4, 8, 16} residual blocks, kernel size {8, 16, 32}, batch size {16, 32, 64}, initial learning rate {0.01, 0.001, 0.0001}, optimization algorithms {SGD, ADAM}, activation functions {ReLU, ELU}, dropout rate {0, 0.5, 0.8}, number of epochs without improvement in plateaus between 5 and 10, that would result in a reduction in the learning rate between 0.1 and 0.5. Besides that, we also tried to: (i) use vectorcardiogram linear transformation to reduce the dimensionality of the input; (ii) include LSTM layer before convolutional layers; (iii) use residual networkterm·mth without the preactivation architecture proposed in ref. 48 ; (iv) use the convolutional architecture known as VGG; (v) switch the order of activation and batch normalization layer.

Statistical and empirical analysis of test results We computed the precision-recall curve to assess the model discrimination of each rhythm class. This curve shows the relationship between precision (PPV) and recall (sensitivity), calculated using binary decision thresholds for each rhythm class. For imbalanced classes, such as our test set, this plot is more informative than the ROC plot 54 . For the remaining analyses we fixed the DNNterm·mth threshold to the value that maximized the F1 scoreterm·key, which is the harmonic mean between precision and recall. The F1 scoreterm·key was chosen here due to its robustness to class imbalance 54 .

For the DNNterm·mth with a fixed threshold, and for the medical residents and students, we computed the precision, the recall, the specificity, the F1 scoreterm·key and, also, the confusion matrix. This was done for each class. Bootstrapping 26 was used to analyze the empirical distribution of each of the scores: we generated 1000 different sets by sampling with replacement from the test set, each set with the same number samples as in the test set, and computed the precision, the recall, the specificity and the F1 scoreterm·key for each. The resulting distributions are presented as a boxplot. We used the McNemar test 27 to compare the misclassification distribution of the DNNterm·mth and the medical residents and students on the test set and the kappa coefficient 46 to compare the inter-rater agreement.

All the misclassified exams were reviewed by an experienced cardiologist and, after an interview with the ECGterm·key reviewers, the errors were classified into: measurement errors, noise errors and unexplained errors (for the DNNterm·mth only) and conceptual and attention errors (for medical residents and students only).

We evaluate the F1 scoreterm·key for alternative scenarios where we use 90%-5%-5% splits of the 2,322,513 records, with the splits ordered randomly, by date, and stratified by patients. The neural networks developed in these alternative scenarios are evaluated on both the original test set ( n = 827) and the additional test splits (last 5% split). The distribution of the performance in each scenario is computed by a bootstrap analysis (with 1000 and 200 samples, respectively) and the resulting boxplots are displayed in the Supplementary Material.

Reporting summary Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Supplementary information Supplementary Information Peer Review File Reporting Summary Acknowledgements This research was partly supported by the Brazilian Agencies CNPq, CAPES, and FAPEMIG, by projects IATS, MASWeb, INCT-Cyber, Rede de Teleassistência de Minas Gerais and Atmosphere, and by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by Knut and Alice Wallenberg Foundation. We also thank NVIDIA for awarding our project with a Titan V GPU. A.L.P.R. receives unrestricted research scholarships from CNPq and FAPEMIG (PPM); W.M.Jr. receives an unrestricted research scholarship from CNPq; A.H.R. receives scholarships from CAPES and CNPq; and, M.H.R. and D.M.O. receive Google Latin America Research Award scholarships. None of the funding agencies had any role in the design, analysis or interpretation of the study. Open access funding provided by Uppsala University.

Source data Source Data Author contributions A.H.R., M.H.R., G.M.M.P., D.M.O., P.R.G., J.A.C, M.P.S.F and A.L.P.R were responsible for the study design. A.L.P.R conceived the project and acted as project leader. A.H.R., M.H.R and C.R.A. chose the architecture, implemented and tuned the deep neural networkterm·key. A.H.R did the statistical analysis of the test data and generated the figures and tables. M.H.R., G.M.M.P, J.A.C. were responsible for the preprocessing and annotating the datasets. G.M.M.P was responsible for the error analysis. D.M.O. implemented the semi-supervised methodology to extract the text label. P.R.G. implemented the user interface used to generate the dataset. P.R.G. and M.P.S.F were responsible for maintenance and extraction of the database. P.W.M., W.M.Jr., and T.B.S. helped in the interpretation of the data. A.H.R., M.H.R, P.W.M., T.B.S. and A.L.P.R. contributed to the writing and all authors revised it critically for important intellectual content. All authors read and approved the submitted manuscript.

Data availability The test dataset used in this study is openly available, and can be downloaded at (10.5281/zenodo.3625006). The weights of all deep neural networkterm·key models we developed for this paper are available at (10.5281/zenodo.3625017). Restrictions apply to the availability of the training set. Requests to access the training data will be considered on an individual basis by the Telehealth Network of Minas Geraisterm·key. Any data use will be restricted to noncommercial research purposes, and the data will only be made available on execution of appropriate data use agreements. The source data underlying Supplementary Figs. 1 and 2 are provided as a Source Data file.

Codeterm·key availability The codeterm·key for training and evaluating the DNNterm·mth model, and, also, for generating figures and tables in this paper is available at: https://github.com/antonior92/automatic-ecg-diagnosis .

Competing interests The authors declare no competing interests.

Ethics statement This study complies with all relevant ethical regulations and was approved by the Research Ethics Committee of the Universidade Federal de Minas Gerais, protocol 49368496317.7.0000.5149. Since this is a secondary analysis of anonymized data stored in the TNMG, informed consent was not required by the Research Ethics Committee for the present study. Researchers signed terms of confidentiality and data utilization.

Footnotes Contributor Information Antônio H. RibeiropersName, Email: antonio-ribeiro@ufmg.br Thomas B. SchönpersName, Email: thomas.schon@it.uu.se Antonio Luiz P. RibeiropersName, Email: antonio.ribeiropersName@ebserh.gov.br Supplementary information Supplementary information is available for this paper at 10.1038/s41467-020-15432-4.

References Associated Data Supplementary Materials Supplementary Information Peer Review File Reporting Summary Data Availability Statement The test dataset used in this study is openly available, and can be downloaded at (10.5281/zenodo.3625006). The weights of all deep neural networkterm·key models we developed for this paper are available at (10.5281/zenodo.3625017). Restrictions apply to the availability of the training set. Requests to access the training data will be considered on an individual basis by the Telehealth Network of Minas Geraisterm·key. Any data use will be restricted to noncommercial research purposes, and the data will only be made available on execution of appropriate data use agreements. The source data underlying Supplementary Figs. 1 and 2 are provided as a Source Data file.

The codeterm·key for training and evaluating the DNNterm·mth model, and, also, for generating figures and tables in this paper is available at: https://github.com/antonior92/automatic-ecg-diagnosis .