Defining an Optimal Cut-Point Value in ROC Analysis: An Alternative Approach. — TEI-tagged view

Ilker UnalpersName (2017date). “Defining an Optimal Cut-Point Value in ROC Analysis: An Alternative Approach..” Computational and Mathematical Methods in Medicinetitle·j.

Object of study — methods for identifying the optimal cut-pointterm·key value in ROC analysis

Research question — whether a cut-point rule based on the AUCterm estimates the true cut-point better than the standard selection methods

Contribution — the Index of Unionterm method, which picks the cut-point whose sensitivityterm·key and specificityterm·key are both closest to the AUCterm

Full text — tagged

The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.

Defining an Optimal Cut-Pointterm·key Value in ROC Analysis: An Alternative Approach.

Abstract ROC curveterm analysis is often applied to measure the diagnostic accuracy of a biomarker. The analysis results in two gains: diagnostic accuracy of the biomarker and the optimal cut-pointterm·key value. There are many methods proposed in the literature to obtain the optimal cut-pointterm·key value. In this study, a new approach, alternative to these methods, is proposed. The proposed approach is based on the value of the area under the ROC curveterm. This method defines the optimal cut-pointterm·key value as the value whose sensitivityterm·key and specificityterm·key are the closest to the value of the area under the ROC curveterm and the absolute value of the difference between the sensitivityterm·key and specificityterm·key values is minimum. This approach is very practical. In this study, the results of the proposed method are compared with those of the standard approaches, by using simulated data with different distribution and homogeneity conditions as well as a real data. According to the simulationterm·mth results, the use of the proposed method is advised for finding the true cut-point.

1. Introduction The ROC curveterm is a mapping of the sensitivityterm·key versus 1 − specificityterm·key for all possible values of the cut-point between cases and controls. To measure the diagnostic ability of a biomarker, it is common to use summary measures such as the area under the ROC curveterm (AUCterm) and/or the partial area under the ROC curveterm (pAUC) [ 1 ]. A biomarker with AUCterm = 1 discriminates individuals perfectly as diseased or healthy. Meanwhile, an AUCterm = 0.5 means that there is no apparent distributional difference between the biomarker values of the two groups [ 2 ].

ROC analysis provides two main outcomes: the diagnostic accuracy of the test and the optimal cut-pointterm·key value for the test. Cut-points dichotomize the test values, so this provides the diagnosis (diseased or not). The identification of the cut-point value requires a simultaneous assessment of sensitivityterm·key and specificityterm·key [ 3 ]. A cut-point will be referred to as optimal when the point classifies most of the individuals correctly [ 4 , 5 ].

AUCterm, sensitivityterm·key, and specificityterm·key values are useful for the evaluation of a marker; however they do not specify “optimal” cut-points directly. In the literature, related to the subject, there are many approaches using both sensitivityterm·key and specificityterm·key for cut-point selection [ 4 – 9 ]. One of the commonly used method is the Youden indexterm ( J ) method [ 5 ]. This method defines the optimal cut-pointterm·key as the point maximizing the Youden function which is the difference between true positive rate and false positive rate over all possible cut-point values [ 6 , 7 ]. Another approach is known as the point closest-to-(0,1) corner in the ROC plane (ER) which defines the optimal cut-pointterm·key as the point minimizing the Euclidean distance between the ROC curveterm and the (0,1) point [ 4 ]. A third approach is based on the maximum achievable value of the chi-square statistic (min⁡ P ) which is driven using the cross-tabulations of true disease status and categorized new variables that separate the biomarker into two categories according to all possible cut-point values [ 8 ]. A more recent approach was proposed by Liu [ 9 ], which defines the optimal cut-pointterm·key as the point maximizing the product of sensitivityterm·key and specificityterm·key (CZ). In the literature, there are studies comparing optimal metrics derived from the sensitivityterm·key, specificityterm·key, agreement, and distance [ 10 , 11 ]. In these studies, it is generally recommended that researchers should select one that is most clinically relevant.

In this study, a new approach is proposed for the identification of the optimal cut-pointterm·key value in ROC analysis. The approach is based on the area under the ROC curveterm (AUCterm), sensitivityterm·key, and specificityterm·key values. It defines the optimal cut-pointterm·key value as the point minimizing the summation of absolute values of the differences between AUCterm and sensitivityterm·key and AUCterm and specificityterm·key provided that the difference between sensitivityterm·key and specificityterm·key is minimum.

In the following section, first the background methodologies of previous methods are summarized, and, then, the proposed method is introduced. In Section 3 , in order to compare the performance of the previous methods with that of the proposed one, generated data under the assumption of normal distribution and gamma distribution models for the biomarker are used. Then, in Section 4 , using data from a real-world study of heart-failure patients [ 12 ], the cut-points for pulse pressure, plasma sodium, LVEF, and heart rate in prediction of mortality are calculated by applying the proposed and the previous methods. Finally, in Section 5 , conclusions are given.

2. Previous Methods and the Proposed Method 2.1. Minimum P Value Approach (min⁡ P ) Let X be a continuous biomarker that is assumed to be predictive of an event E (i.e., E = 1 for diseased or E = 0 for not diseased). At any given possible cut-point c of X , sensitivityterm·key (Se) and specificityterm·key (Sp) values are as follows: (1) S e c = P X > c ∣ E = 1 , S p c = P X ≤ c ∣ E = 0 . Cut-point c separates the data into two groups which forms a 2 × 2 table, as shown in Table 1 .

The minimum P value approach was proposed by Miller and Siegmund [ 8 ] and defines the optimal point as cut-point c ^ min ⁡ P that maximizes the standard chi-square statistic with one degree of freedom: (2) χ 1 2 c = N s v − u r 2 s + r u + v s + u r + v , where N = s + r + u + v . As it was shown by Rota and Antolini [ 11 ], it can be also written in terms of classification probabilities: (3) χ 1 2 c = S e c + S p c − 1 2 u + v S e c + s + r 1 − S p c / N 1 − u + v S e c + s + r 1 − S p c / N 1 / u + v + 1 / s + r .

2.2. Youden Indexterm ( J ) The Youden indexterm ( J ) is a measure for evaluating the biomarker effectiveness. This measure was first introduced to the medical literature by Youden [ 5 ]. J is a function of Se( c ) and Sp( c ), such that (4) J c = S e c + S p c − 1 = S e c − 1 − S p c over all cut-points c ; c ^ J denotes the cut-point corresponding to J . When the value of J is maximum, c ^ J is the “optimal” cut-point value [ 6 , 7 ].

2.3. The Closest to (0,1) Criteria (ER) In this criteria, the “optimal” cut-point is defined as the point closest to the point (0,1) on the ROC curveterm [ 3 , 4 ]. (5) E R c = ⁡ 1 − S e c 2 + 1 − S p c 2 . Mathematically, the point c ^ E R minimizing the ER( c ) function is called the “optimal” cut-point value.

2.4. Concordance Probability Methodterm (CZ) The concordance probability methodterm proposed by Liu [ 9 ] defines the optimal cut-pointterm·key as the point maximizing the product of sensitivityterm·key and specificityterm·key. (6) C Z c = S e c ∗ S p c . This product gets value between 0 and 1. The concordance probabilityterm·key of dichotomized measure at cut-point c can be expressed as the area of a rectangle associated with the ROC curveterm. Cut-point c ^ C Z maximizing CZ( c ) actually maximizes the area of the rectangle [ 9 ].

2.5. The Proposed Method: Index of Unionterm (IU) Perkins and Schisterman [ 4 ] stated that the “optimal” cut-point should be chosen as the point which classifies most of the individuals correctly and thus least of them incorrectly. From this point of view, in this study, the Index of Unionterm method is proposed. This method provides an “optimal” cut-point which has maximum sensitivityterm·key and specificityterm·key values at the same time. In order to find the highest sensitivityterm·key and specificityterm·key values at the same time, the AUCterm value is taken as the starting value of them. For example, let AUCterm value be 0.8. The next step is to look for a cut-point from the coordinates of ROC whose sensitivityterm·key and specificityterm·key values are simultaneously so close or equal to 0.8. This cut-point is then defined as the “optimal” cut-point. The above criteria correspond to the following equation: (7) I U c = S e c − A U C + S p c − A U C . The cut-point c ^ I U , which minimizes the IU( c ) function and the |Se( c ) − Sp( c )| difference, will be the “optimal” cut-point value.

In other words, the cut-point c ^ I U defined by the IU methodterm should satisfy two conditions: (1) sensitivityterm·key and specificityterm·key obtained at this cut-point should be simultaneously close to the AUCterm value; (2) the difference between sensitivityterm·key and specificityterm·key obtained at this cut-point should be minimum. The second condition is not compulsory, but it is an essential condition when multiple cut-points satisfy the equation.

In order to illustrate how the IU methodterm defines the “optimal” cut-point, the values obtained from an artificial data are used. Some of the cut-points (with their sensitivityterm·key and specificityterm·key values) provided by the artificial data are given in Table 2 . In this example, the AUCterm value is calculated as 0.918. For the sake of simplicity, instead of 1 − specificityterm·key values, specificityterm·key values are given in the table. By using IU methodterm, one can easily find that sensitivityterm·key (0.92) and specificityterm·key (0.92) values of the cut-point 1.985 are the nearest ones to the AUCterm value. Since also the difference between these two values is minimum, this cut-point will be called the “optimal” cut-point by the IU methodterm.

However, it should be noted that choosing such a cut-point as the “optimal” cut-point may sometimes fail. For example, let Se( c ) = Sp( c ) = AUCterm = 0.8. Then, the IU( c ) statistic given in ( 7 ) will be 0 and also the difference between Se( c ) and Sp( c ) will be 0. Thus according to the definition of optimality given in the IU methodterm, cut-point c will be accepted as the “optimal” cut-point. However, if there is a point c ∗ for which Se( c ∗ ) = 0.82 and Sp( c ∗ ) = 0.80, then the total misclassification rate will be 0.38 (which is smaller than that of the point c , i.e., 0.40). Hence, cut-point c ∗ is a better optimized point than cut-point c , based on the definition of optimality given by Perkins and Schisterman [ 4 ].

Geometrically, the idea behind the IU methodterm is very similar to the idea behind the ER methodterm. As it can be seen in Figure 1 , the IU methodterm also tries to find the closest point to a point, that is, the point (1 − AUCterm, AUCterm). In the ER methodterm, this point is taken as (0,1). However, instead of using the Euclidean distance as in the ER methodterm, the IU methodterm uses the absolute differences between the diagnostic accuracy measures and the AUCterm value. More specifically, the IU methodterm searches for the point that minimizes the half perimeter of the ABCD rectangle seen in Figure 1 . This rectangle is constructed by connecting the intersections points of the lines of x = 1 − AUCterm, y = AUCterm, x = 1 − Sp( c ), and y = Se( c ).

3. Simulationterm·mth Study As it was shown by Rota and Antolini [ 11 ] although some of these methods are mathematically related, they do not necessarily identify the same true cut-point. That is, depending on the design of the study (balanced or unbalanced), the methods may identify different cut-points. According to their results, in the balanced homoscedastic scenario, the methods identified the same point; in the remaining scenarios (i.e., unbalanced homoscedastic and balanced/unbalanced heteroscedastic scenarios), the methods identified different cut-points. These results emphasize the importance of correctly defining the true cut-point in all possible scenarios.

Let us assume that a specific biomarker ( X ) in diseased and nondiseased populations is normally distributed, X 1 ~ N ( μ 1 , σ 1 = 1) for diseased subjects and X 0 ~ N (0, σ 0 = 1) for nondiseased subjects. Under these assumptions, sensitivityterm·key and specificityterm·key can be written as (8) S e c = P X 1 ≥ c = Φ μ 1 − c , S p c = P X 0 ≤ c = Φ c , where Φ denotes the standard normal distribution function. The optimal cut-pointterm·key occurs at the intersection of the normal probability density functions of diseased and nondiseased subjects (i.e., c opt = μ 1 /2) [ 7 , 13 ]. For example, if μ 1 is taken as {0.51,1.05,1.68,2.56}, the corresponding true cut-points will be c opt = {0.25,0.52,0.84,1.28} [ 11 , 13 ]. These values of μ 1 guarantee a wide variety of classification accuracies, ranging from a poor to a high one [ 7 , 11 , 13 ]. The identification of the true theoretical cut-point for the IU methodterm under this scenario is given in the Appendix.

Now assume that X is gamma distributed with the following parameters: X 1 ~ G ( α 1 = 2.5, β 1 ) for diseased subjects and X 0 ~ G ( α 0 = 1.5, β 0 = 1) for nondiseased subjects. If, for instance, β 1 is taken as {0.79, 1.22, 1.97, 3.82}, the corresponding cut-points for each method will be different; that is, for min⁡ P approach, c min⁡ P = {0.80,1.73,2.54,3.51}, for Youden indexterm, c J = {1.12,1.79,2.45,3.42}, for the concordance probabilityterm·key, c CZ = {1.35,1.81,2.41,3.38}, and, for the point closest-to-(0,1) corner, c ER = {1.38,1.82,2.36,3.24} [ 11 ]. For the Index of Unionterm, the corresponding cut-points are estimated by the empirical estimation method given in Liu's work [ 9 ] as c IU = {1.42,1.78,2.41,3.30} ( Figure 2 ).

In order to compare the performance of the cut-point selection methods with the performance of the method proposed in this study, a simulationterm·mth study is conducted with different scenarios. These scenarios are the same as the ones given in Rota and Antolini's work [ 11 ]. The first scenario is normal homoscedastic scenario with balanced design where all of the methods theoretically identify the same true cut-point. The second one is the nonbalanced normal case, where all of the methods except the min⁡ P approach identify the same cut-point. The last scenario is gamma case where all of the methods identify different cut-points.

In all scenarios, 1000 samples were generated with sample sizes 50, 100, and 200 for each group and with sample size n 1 = 50, n 0 = 100; n 1 = 50, n 0 = 150; and n 1 = 50, n 0 = 200 ( n 1 is the number of diseased subjects and n 0 is the number of nondiseased subjects).

For each sample, the optimal cut-pointsterm·key c ^ min ⁡ P , c ^ J , c ^ C Z , c ^ E R , and c ^ I U for the minimum P value, the Youden indexterm, the concordance probabilityterm·key, the point closest-to-(0, 1) corner, and the Index of Unionterm are estimated, respectively. The relative bias and mean square errorterm (MSEterm) values of each method are computed by E c ^ - c / c and E c ^ - c 2 , respectively. ( c denotes the true cut-point and c ^ denotes the estimated cut-point by the method.) In order to estimate the standard deviation and the confidence intervalterm (CIterm) for the optimal cut-pointterm·key, the bootstrapterm·key resampling technique is applied [ 14 ]. To calculate the bootstrapterm·key estimate c ^ B , random sampling with replacement is used to draw 200 bootstrapterm·key samples within each of the 1000 generated samples. Moreover, to construct a 95% CIterm for the optimal cut-pointterm·key, the basic percentile method is applied by taking the 2.5 and 97.5 percentiles of the c ^ B bootstrapterm·key distribution.

The bootstrapterm·key estimator of the standard deviation (SD B ) for the estimated cut-point is calculated by taking the standard deviation of the 200 cut-point estimates. Within each of the simulationterm·mth scenarios, the CIsterm are subsequently evaluated by computing coverage probability and mean length.

All simulationsterm·mth are done by using R program with the version of 3.2.0. To determine the estimates for Youden indexterm and the point closest-to-(0,1) corner, the pROC library is used [ 15 ]. For defining the estimates of the rest of the methods, an R code is written by the author and it can be available upon request.

3.1. Simulationterm·mth Results Table 3 shows the results for the balanced design under normal homoscedastic distributions. The relative bias values of the previously proposed methods are similar to the results of Rota and Antolini's work [ 11 ] except the relative bias of Youden indexterm. In particular for poor classification accuracy scenarios (i.e., c opt = 0.25 and 0.52), Youden indexterm has worse performance in the estimation of the optimal cut-pointterm·key than their results. However, this discrepancy is not seen in the comparison of MSEsterm. That is, the MSEsterm of all methods are similar to that of Rota and Antolini's work [ 11 ].

When comparing the relative bias and MSEterm values of the IU methodterm with that of the other methods, it can be easily seen that the IU methodterm has mostly similar performance with the point closest-to-(0,1) corner method and has better performance than the other methods (i.e., lower relative bias and lower MSEterm values).

For the balanced design under normal homoscedastic distributions, the bootstrapterm·key standard deviation, coverage probability, and mean length of the 95% bootstrapterm·key CIterm for the cut-point are shown in Table 4 . As in Table 3 , the results given in Table 4 are similar to that of Rota and Antolini's work [ 11 ]. That is, the SD B of the minimum P value approach is still greater with respect to that of the other methods and the better classification accuracies provide the narrower 95% bootstrapterm·key CIsterm. The IU methodterm achieves the smallest SD B value and the narrowest CIsterm in most of the scenarios. The coverage probabilities are close to the nominal level for all methods.

The relative bias and MSEterm results for the unbalanced design under normal homoscedastic distributions are shown in Table 5 . Since the true cut-point for the minimum P value approach depends on the prevalence of the disease in the sample, different optimal cut-pointsterm·key are used for the comparisons [ 11 ]. The relative bias values of all methods are similar to those of Rota and Antolini's work [ 11 ], except for the minimum P value approach in the lowest classification accuracy scenario (i.e., c opt = 0.25). For this scenario the relative bias for the minimum P value approach is larger than the bias given in their work. For poor and poor-moderate classification accuracy (i.e., c opt = 0.25 and 0.52), the MSEterm is the lowest for the IU methodterm, and, for moderate-high and high classification accuracy (i.e., c opt = 0.84 and 1.28), both the point closest-to-(0,1) corner method and the IU methodterm get the lowest MSEterm values.

For the unbalanced design under normal homoscedastic distributions, the bootstrapterm·key standard deviation, coverage probability, and mean length of the 95% bootstrapterm·key CIterm for the cut-point are given in Table 6 . For this scenario, the lowest SD B and mean length of the 95% bootstrapterm·key CIterm values are obtained by the point closest-to-(0, 1) corner method and the IU methodterm. As in the comparison of the relative bias and MSEterm values of the methods ( Table 5 ), for poor and poor-moderate classification accuracy (i.e., c opt = 0.25 and 0.52), the IU methodterm gets the lowest SD B and mean length, and, for moderate-high and high classification accuracy (i.e., c opt = 0.84 and 1.28), both the point closest-to-(0,1) corner in the ROC plane and the IU methodterm get the lowest values. The coverage probabilities are close to the nominal level for all methods.

As it was shown in Rota and Antolini's work [ 11 ], under a gamma distribution assumption with a balanced design, the theoretical true cut-points c min⁡ P , c J , c CZ , and c ER are all different. For all classification accuracy scenarios, the theoretical true cut-points for the IU methodterm are obtained based on the idea given in the article of Liu [ 9 ] ( Figure 2 ). The relative bias values of all methods are similar to those of Rota and Antolini's work [ 11 ]. The MSEterm gets its lowest value in the point closest-to-(0,1) corner and the IU methodterm for all classification accuracy scenarios ( Table 7 ).

For this design (under gamma distributions), the SD B and mean length of 95% CIterm values for the point closest-to-(0,1) corner method and the IU methodterm are lower than the other investigated approaches ( Table 8 ). The coverage probabilities are close to the nominal level for all methods.

In all simulationterm·mth scenarios, the IU methodterm shows a better performance in the estimation of the optimal cut-pointterm·key with respect to the other methods. The bootstrapterm·key standard deviation and mean length of the 95% bootstrapterm·key CIterm values for the IU methodterm are also minimum among all methods. Thus, for all simulationterm·mth scenarios, although, in gamma scenarios, the methods do not lead to a common cut-point, in order to identify the optimal cut-pointterm·key, the IU methodterm is a better alternative than the previous proposed methods.

3.2. Cross-Validation of the Optimal Cut-Pointterm·key In order to evaluate the significance of the optimally selected cut-point, twofold cross-validation process [ 16 ] is used. The procedure is as follows: Generating data with the same properties given in this manuscript Applying all methods to the data and estimating cut-points for all methods Splitting data into two equal subsets, that is, subset I and subset II Applying all methods to subset I and estimating cut-points for all methods Assigning each observation in subset II to either one of two groups by using the cut-point obtained in the previous step Applying all methods to new subset II and estimating cut-points for all methods Assigning each observation in subset I to either one of two groups by using the cut-point obtained in the previous step Applying all methods to the combination of these two subsets and estimating cut-points for all methods Taking the difference between the cut-points obtained at the second step and at the last step This procedure is applied for 4 scenarios (2 normal and 2 gamma scenarios with the sample size n 0 = n 1 = 50) given in the manuscript. The results are shown in Figure 1 in Supplementary Material available online at https://doi.org/10.1155/2017/3762651 . According to the results, for each method, the difference between the optimal cut-pointsterm·key estimated before and after cross-validation is around 0 and the IU methodterm gets the smallest mean absolute difference in all four scenarios.

4. Application A real data obtained from a study in cardiology is used as an example. Yildiran et al. [ 12 ] investigated an association between pulse pressure and 2-year cardiovascular death in an entire heart-failure population. They prospectively enrolled 225 (188 male, 37 female) heart-failure patients with NYHA functional classes I–IV, mean age 56.5 [ 12 ].

They recorded detailed histories of the 225 patients, including demographic characteristics, cardiovascular (CV) risk factors, and medication usage. The patients were divided into 4 NYHA classes in accordance with their medical histories and the findings upon physical examination and then into 2 groups according to their NYHA class (mild heart failure [classes I-II] and advanced heart failure [classes III-IV]). Levels of serum lipids, glucose, high-sensitivity C-reactive protein, blood urea nitrogen, creatinine, sodium, and potassium were measured by routine laboratory methods. Blood pressures were measured by sphygmomanometer in accordance with published guidelines. Pulse pressure was calculated as the difference between systolic and diastolic blood pressure, and the patients were divided accordingly into 4 quartiles (PP of <35, 35–45, 46–55, or >55 mmHg) [ 12 ].

They used ROC analysis to define the cut-point values for pulse pressure, LVEF, plasma sodium value, and heart rate in predicting CV death. In this analysis, 170 patients who had all four measurements at the same time (55 patients' measurements were missing) were included. To get optimal cut-pointterm·key values, they used ER approach [ 12 ].

Supplementary Web-Only Table 1 reports some descriptive statistics of these four measurements. Pulse pressure, LVEF, and plasma sodium levels are significantly lower in dead patients ( n 1 = 43) than in alive patients ( n 0 = 117) and heart rate is significantly higher in dead patients than in alive patients. According to the results of the Shapiro-Wilk nonparametric normal distribution test, heart rate and plasma sodium are both normally distributed in both groups, LVEF is normally distributed in dead patients and is not normally distributed in alive patients, and pulse pressure is not normally distributed in both groups. For nonnormal distributed variables, the distribution of LVEF in alive patients is left-skewed and the distributions of pulse pressure in both groups are right-skewed. Since the numbers of patients in two groups are not close enough, the design is unbalanced and the ratio between the numbers of patients is similar to the 50 : 100 scenario in the simulationterm·mth protocol.

In this study, the data obtained from the study by Yildiran et al. [ 12 ] is used and all the methods including the IU methodterm are applied to this data. The corresponding results are given in Table 9 . The upper part of Table 9 shows the cut-points obtained by using the previously proposed methods. To define the cut-point with the IU methodterm, some of cut-points with their sensitivityterm·key and specificityterm·key values and AUCterm value are given. According to this table, the IU methodterm gives the same cut-points with the ER methodterm for different AUCterm values ( Figure 3 ).

5. Conclusions Defining the optimal cut-pointterm·key is very important when a continuous variable is considered as a diagnostic marker. Getting optimal classification level depends on the point chosen for diagnosis. The criteria for optimality can change according to the aim of the study. However, as a general rule, minimizing the total misclassification rates is a good approach. With IU methodterm, since the difference between sensitivityterm·key and specificityterm·key values is minimum, this condition is met most of the time.

According to the results given in the tables, the proposed IU methodterm can be a better alternative for defining the cut-point. When the definition of optimal point is stated as the point that minimizes the misclassification rates or the point that equalizes the values of sensitivityterm·key and specificityterm·key, the IU methodterm is better than the other methods in most of the comparison scenarios. This conclusion does not change with the distribution of biomarker or the homogeneity of variances of biomarkers. The changes in the sample size and the AUCterm values may affect but not alter the interpretation.

The IU methodterm uses the absolute difference between diagnostic accuracy measures and AUCterm value instead of using the Euclidean distance. The reason behind this idea is to provide the simplicity in defining the point as optimal. With the IU methodterm, one can easily identify the optimal cut-pointterm·key only by checking whether the sensitivityterm·key and specificityterm·key values are close enough to AUCterm value or not. That is, the complex calculations are not necessary for the IU methodterm.

When the relative bias and MSEterm values of the IU methodterm are compared with the previous methods, it is seen that the IU methodterm is better than the others. Thus this method can be used for defining the optimal cut-pointterm·key value especially when the sample sizes of the two groups are equal and the AUCterm value is high. (i.e., higher than 0.7).

A common practice is to select a cut-point which defines two risk groups for a continuously measured biomarker [ 16 ]. A cut-point for a biomarker is meaningful for the clinicians when it is clinically interpretable and understandable. Clinical meaning for a cut-point can be explained by using its accuracy, that is, true classification rate. Among all the methods, only two of them, the Youden indexterm and the concordance probabilityterm·key, are based on the maximization of this rate. Thus, these methods provide interpretable cut-points.

The point closest to (0,1) point on the ROC curveterm method involves a quadratic term and clinical meaning of this term is unknown. Despite the lack of clinical meaning, it is shown in the literature that this method is superior to the other methods in estimating the true cut-point [ 11 ].

The IU methodterm, like the Youden indexterm and the concordance probabilityterm·key, tries to minimize the misclassification rate. Hence, it also provides an interpretable cut-point. In this study, it is shown that the IU methodterm performs better than (or equal to) the point closest to (0,1) point on the ROC curveterm method. Therefore, the use of the IU methodterm is advised to get more interpretable and better optimized cut-point.

The IU methodterm provides a cut-point whose sensitivityterm·key and specificityterm·key are equally high. This means that, in a cut-point determination process, if sensitivityterm·key and specificityterm·key are valued equally, the IU methodterm seems to be the best option among all other methods.

Supplementary Material Supplementary Table 1: Descriptive statistics of the application example of cut-point finding for Pulse pressure, LVEF, Plasma sodium level and Heart rate in prediction of mortality, from Yildiran et al. (2010).

Supplementary Figure 1: The difference between the optimal cut-pointsterm·key estimated before and after cross-validation is around 0 and the IU methodterm gets the smallest mean absolute difference in all four scenarios.