Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer — TEI-tagged view

Godfred O. BoatengpersName, Torsten B. NeilandspersName, Edward A. FrongillopersName, Hugo R. Melgar-QuiñonezpersName, Sera L. YoungpersName (2018date). “Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer.” Frontiers in Public Healthtitle·j.

Object of study — the concepts, procedures, diagnostics, and evidence used to develop and validate health, social, and behavioral scales

Research question — What best-practice sequence should researchers use to generate itemsterm, constructterm scales, and evaluate dimensionality, reliabilityterm, and validityterm?

Contribution — a non-systematic primer synthesizing a three-phase, nine-step scale-development and validation workflow

Full text — tagged

Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer Abstract Scale development and validationterm·mth are critical to much of the work in the health, social, and behavioral sciencesterm·dom. However, the constellation of techniques required for scale developmentterm·mth and evaluation can be onerous, jargon-filled, unfamiliar, and resource-intensive. Further, it is often not a part of graduate training. Therefore, our goal was to concisely reviewterm·mth the process of scale developmentterm·mth in as straightforward a manner as possible, both to facilitate the development of new, valid, and reliable scales, and to help improve existing ones. To do this, we have created a primer for best practices for scale developmentterm·mth in measuring complex phenomenaterm·dom. This is not a systematic reviewterm·mth, but rather the amalgamation of technical literatureterm and lessons learned from our experiences spent creating or adapting a number of scales over the past several decades. We identified three phasesterm that span nine stepsterm. In the first phase, itemsterm are generated and the validityterm of their content is assessed. In the second phase, the scale is constructed. Steps in scale constructionterm·mth include pre-testing the questionsterm, administering the surveyterm, reducing the number of itemsterm, and understanding how many factorsterm the scale captures. In the third phase, scale evaluationterm·mth, the number of dimensions is tested, reliabilityterm is tested, and validityterm is assessed. We have also added examples of best practices to each step. In sum, this primer will equip both scientists and practitioners to understand the ontology and methodology of scale development and validationterm·mth, thereby facilitating the advancement of our understanding of a range of health, social, and behavioral outcomes.

Keywords: scale developmentterm·mth, psychometric evaluation, content validityterm, item reductionterm, factor analysisterm, tests of dimensionalityterm, tests of reliabilityterm, tests of validityterm Introduction Scales are a manifestation of latent constructsterm; they measure behaviors, attitudes, and hypothetical scenarios we expect to exist as a result of our theoretical understanding of the world, but cannot assess directly ( 1 ). Scales are typically used to capture a behavior, a feeling, or an action that cannot be captured in a single variable or itemterm. The use of multiple itemsterm to measure an underlying latent constructterm can additionally account for, and isolate, item-specific measurement errorterm, which leads to more accurate research findings. Thousands of scales have been developed that can measure a range of social, psychological, and health behaviors and experiences.

As science advances and novel research questions are put forth, new scales become necessary. Scale developmentterm·mth is not, however, an obvious or a straightforward endeavor. There are many steps to scale developmentterm·mth, there is significant jargon within these techniques, the work can be costly and time consuming, and complex statistical analysis is often required. Further, many health and behavioral science degrees do not include training on scale developmentterm·mth. Despite the availability of a large amount of technical literatureterm on scale theory and development ( 1 – 7 ), there are a number of incomplete scales used to measure mental, physical, and behavioral attributes that are fundamental to our scientific inquiry ( 8 , 9 ).

Therefore, our goal is to describe the process for scale developmentterm·mth in as straightforward a manner as possible, both to facilitate the development of new, valid, and reliable scales, and to help improve existing ones. To do this, we have created a primer for best practices for scale developmentterm·mth. We anticipate this primer will be broadly applicable across many disciplines, especially for health, social, and behavioral sciencesterm·dom. This is not a systematic reviewterm·mth, but rather the amalgamation of technical literatureterm and lessons learned from our experiences spent creating or adapting a number of scales related to multiple disciplines ( 10 – 23 ).

First, we provide an overview of each of the nine stepsterm. Then, within each step, we define key concepts, describe the tasks required to achieve that step, share common pitfalls, and draw on examples in the health, social, and behavioral sciencesterm·dom to recommend best practices. We have tried to keep the material as straightforward as possible; references to the body of technical work have been the foundation of this primer.

Scale developmentterm·mth overview There are three phasesterm to creating a rigorous scale—item developmentterm·mth, scale developmentterm·mth, and scale evaluationterm·mth ( 24 ); these can be further broken down into nine stepsterm (Figure 1 ).

An overview of the three phasesterm and nine stepsterm of scale development and validationterm·mth.

Item developmentterm·mth, i.e., coming up with the initial set of questions for an eventual scale, is composed of: (1) identification of the domainterm(s) and itemterm generation, and (2) consideration of content validityterm. The second phase, scale developmentterm·mth, i.e., turning individual itemsterm into a harmonious and measuring constructterm, consists of (3) pre-testing questionsterm, (4) sampling and survey administrationterm, (5) item reductionterm, and (6) extraction of latent factorsterm. The last phase, scale evaluationterm·mth, requires: (7) tests of dimensionalityterm, (8) tests of reliabilityterm, and (9) tests of validityterm.

Phase 1: item developmentterm·mth Step 1: identification of the domainterm(s) and itemterm generation Domainterm identification The first step is to articulate the domainterm(s) that you are endeavoring to measure. A domainterm or constructterm refers to the concept, attribute, or unobserved behaviorterm that is the target of the study ( 25 ). Therefore, the domainterm being examined should be decided upon and defined before any itemterm activity ( 2 ). A well-defined domainterm will provide a working knowledge of the phenomenon under study, specify the boundaries of the domainterm, and ease the process of itemterm generation and content validation.

McCoach et al. outline a number of steps in scale developmentterm·mth; we find the first five to be suitable for the identification of domainterm ( 4 ). These are all based on thorough literature reviewterm and include (a) specifying the purpose of the domainterm or constructterm you seek to develop, and (b), confirming that there are no existing instruments that will adequately serve the same purpose. Where there is a similar instrument in existence, you need to justify why the development of a new instrument is appropriate and how it will differ from existing instruments. Then, (c) describe the domainterm and provide a preliminary conceptual definition and (d) specify, if any, the dimensions of the domainterm. Alternatively, you can let the number of dimensions forming the domainterm to be determined through statistical computation (cf. Steps 5, 6, and 7). Domainsterm are determined a priori if there is an established framework or theory guiding the study, but a posteriori if none exist. Finally, if domainsterm are identified a priori , (e) the final conceptual definition for each domainterm should be specified.

Itemterm generation Once the domainterm is delineated, the itemterm pool can then be identified. This process is also called “question development” ( 26 ) or “itemterm generation” ( 24 ). There are two ways to identify appropriate questions: deductive and inductive methodsterm ( 24 ).

The deductive methodterm, also known as “logical partitioningterm” or “classification from aboveterm” ( 27 ) is based on the description of the relevant domainterm and the identification of itemsterm. This can be done through literature reviewterm and assessment of existing scales and indicators of that domainterm ( 2 , 24 ). The inductive methodterm, also known as “groupingterm” or “classification from belowterm” ( 24 , 27 ) involves the generation of itemsterm from the responses of individuals ( 24 ). Qualitative data obtained through direct observations and exploratory research methodologies, such as focus groupsterm and individual interviewsterm, can be used to inductively identify domainterm itemsterm ( 5 ).

It is considered best practice to combine both deductive and inductive methodsterm to both define the domainterm and identify the questions to assess it. While the literature reviewterm provides the theoretical basis for defining the domainterm, the use of qualitative techniques moves the domainterm from an abstract point to the identification of its manifest formsterm. A scale or constructterm defined by theoretical underpinnings is better placed to make specific pragmatic decisions about the domainterm ( 28 ), as the constructterm will be based on accumulated knowledge of existing itemsterm.

It is recommended that the itemsterm identified using deductive and inductive approaches should be broader and more comprehensive than one's own theoretical view of the target ( 28 , 29 ). Further, content should be included that ultimately will be shown to be tangential or unrelated to the core constructterm. In other words, one should not hesitate to have itemsterm on the scale that do not perfectly fit the domainterm identified, as successive evaluation will eliminate undesirable itemsterm from the initial pool. Kline and Schinka et al. note that the initial pool of itemsterm developed should be at minimum twice as long as the desired final scale ( 26 , 30 ). Others have recommended the initial pool to be five times as large as the final version, to provide the requisite margin to select an optimum combination of itemsterm ( 30 ). We agree with Kline and Schinka et al. ( 26 , 30 ) that the number of itemsterm should be at least twice as long as the desired scale.

Further, in the development of itemsterm, the form of the itemsterm, the wording of the itemsterm , and the types of responses that the question is designed to induce should be taken into account. It also means questions should capture the lived experiences of the phenomenon by target population ( 30 ). Further, itemsterm should be worded simply and unambiguously. Itemsterm should not be offensive or potentially biased in terms of social identity, i.e., gender, religion, ethnicity, race, economic status, or sexual orientation ( 30 ).

Fowler identified five essential characteristics of itemsterm required to ensure the quality of constructterm measurement ( 31 ). These include (a) the need for itemsterm to be consistently understood; (b) the need for itemsterm to be consistently administered or communicated to respondents; (c) the consistent communication of what constitutes an adequate answer; (d) the need for all respondents to have access to the information needed to answer the question accurately; and (e) the willingness for respondents to provide the correct answers required by the question at all times.

These essentials are sometimes very difficult to achieve. Krosnick ( 32 ) suggests that respondents can be less thoughtful about the meaning of a question, search their memories less comprehensively, integrate retrieved information less carefully, or even select a less precise response choice. All this means that they are merely satisficing, i.e., providing merely satisfactory answers, rather than the most accurate ones. In order to combat this behavior, questions should be kept simple, straightforward, and should follow the conventions of normal conversation.

With regards to the type of responses to these questions, we recommend that questions with dichotomous response categories (e.g., true/false) should have no ambiguity. When a Likert-type response scaleterm·mth is used, the points on the scale should reflect the entire measurement continuum. Responses should be presented in an ordinal manner, i.e., in an ascending order without any overlap, and each point on the response scale should be meaningful and interpreted the same way by each participant to ensure data quality ( 33 ).

In terms of the number of points on the response scale, Krosnick and Presser ( 33 ) showed that responses with just two to three points have lower reliabilityterm than Likert-type response scalesterm·mth with five to seven points. However, the gain levels off after seven points. Therefore, response scales with five points are recommended for unipolar itemsterm, i.e., those reflecting relative degrees of a single itemterm response quality, e.g., not at all satisfied to very satisfied. Seven response itemsterm are recommended for bipolar itemsterm, i.e., those reflecting relative degrees of two qualities of an itemterm response scale, e.g., completely dissatisfied to completely satisfied. As an analytic aside, itemsterm with scale points fewer than five categories are best estimated using robust categorical methods. However, itemsterm with five to seven categories without strong floor or ceiling effects can be treated as continuous itemsterm in confirmatory factor analysisterm and structural equation modelingterm using maximum likelihood estimationsterm ( 34 ).

One pitfall in the identification of domainterm and itemterm generation is the improper conceptualization and definition of the domain(s)term. This can result in scales that may either be deficientterm because the definition of the domainterm is ambiguous or has been inadequately defined ( 35 ). It can also result in contaminationterm, i.e., the definition of the domainterm overlaps with other existing constructsterm in the same field ( 35 ).

Caution should also be taken to avoid construct underrepresentationterm, which is when a scale does not capture important aspects of a constructterm because its focus is too narrow ( 35 , 36 ). Further, construct-irrelevant varianceterm, which is the degree to which test scores are influenced by processes that have little to do with the intended constructterm and seem to be widely inclusive of non-related itemsterm ( 36 , 37 ), should be avoided. Both construct underrepresentationterm and irrelevant variance can lead to the invalidation of the scaleterm ( 36 ).

An example of best practice using the deductive approach to itemterm generation is found in the work of Dennis on breastfeeding self-efficacy ( 38 – 40 ). Dennis' breastfeeding self-efficacy scale itemsterm were first informed by Bandura's theory on self-efficacy, followed by content analysis of literature reviewterm, and empirical studies on breastfeeding-related confidence.

A valuable example for a rigorous inductive approach is found in the work of FrongillopersName and Nanama on the development and validation of an experience-based measure of household food insecurity in northern Burkina Faso ( 41 ). In order to generate itemsterm for the measure, they undertook in-depth interviews with 10 household heads and 26 women using interview guides. The data from these interviews were thematically analyzed, with the results informing the identification of itemsterm to be added or deleted from the initial questionnaire. Also, the interviews led to the development and revision of answer choices.

Step 2: content validityterm Content validityterm, also known as “theoretical analysisterm” ( 5 ), refers to the “adequacy with which a measure assesses the domainterm of interest” ( 24 ). The need for content adequacy is vital if the itemsterm are to measure what they are presumed to measure ( 1 ). Additionally, content validityterm specifies content relevanceterm and content representationsterm, i.e., that the itemsterm capture the relevant experience of the target population being examined ( 129 ).

Content validityterm entails the process of ensuring that only the phenomenon spelled out in the conceptual definition, but not other aspects that “might be related but are outside the investigator's intent for that particular [constructterm] are added” ( 1 ). Guion has proposed five conditions that must be satisfied in order for one to claim any form of content validityterm. We find these conditions to be broadly applicable to scale developmentterm·mth in any discipline. These include that (a) the behavioral content has a generally accepted meaning or definition; (b) the domainterm is unambiguously defined; (c) the content domainterm is relevant to the purposes of measurement; (d) qualified judges agree that the domainterm has been adequately sampled based on consensus; and (e) the response content must be reliably observed and evaluated ( 42 ). Therefore, content validityterm requires evidence of content relevanceterm, representativeness, and technical quality.

Content validityterm is mainly assessed through evaluation by expert and target population judgesterm.

Evaluation by experts Expert judgesterm are highly knowledgeable about the domainterm of interest and/or scale developmentterm·mth; target population judgesterm are potential users of the scale ( 1 , 5 ). Expert judgesterm seem to be used more often than target-population judges in scale developmentterm·mth work to date. Ideally, one should combine expert and target population judgment. When resources are constrained, however, we recommend at least the use of expert judgesterm.

Expert judgesterm evaluate each of the itemsterm to determine whether they represent the domainterm of interest. These expert judgesterm should be independent of those who developed the itemterm pool. Expert judgment can be done systematically to avoid bias in the assessment of itemsterm. Multiple judges have been used (typically ranging from 5 to 7) ( 25 ). Their assessments have been quantified using formalized scaling and statistical procedures such as the content validity ratioterm for quantifying consensus ( 43 ), content validity indexterm for measuring proportional agreement ( 44 ), or Cohen's coefficient kappaterm ( k ) for measuring inter-rater or expert agreement ( 45 ). Among the three procedures, we recommend Cohen's coefficient kappaterm, which has been found to be most efficient ( 46 ). Additionally, an increase in the number of experts has been found to increase the robustness of the ratings ( 25 , 44 ).

Another way by which content validityterm can be assessed through expert judgesterm is by using the Delphi method to come to a consensus on which questions are a reflection of the constructterm you want to measure. The Delphi method is a technique “for structuring group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem” ( 47 ).

A good example of evaluation of content validityterm using expert judgesterm is seen in the work of Augustine et al. on adolescent knowledge of micronutrients ( 48 ). After identifying a list of itemsterm to be validated, the authors consulted experts in the field of nutrition, psychology, medicine, and basic sciences. The itemsterm were then subjected to content analysis using expert judgesterm. Two independent reviewsterm·mth were carried out by a panel of five experts to select the questions that were appropriate, accurate, and interpretable. Itemsterm were either accepted, rejected, or modified based on majority opinion ( 48 ).

Evaluation by target population Target population judgesterm are experts at evaluating face validityterm, which is a component of content validityterm ( 25 ). Face validityterm is the “degree that respondents or end users [or lay persons] judge that the itemsterm of an assessment instrument are appropriate to the targeted constructterm and assessment objectives” ( 25 ). These end-users are able to tell whether the constructterm is a good measure of the domainterm through cognitive interviews, which we discuss in Step 3.

An example of the concurrent use of expert and target population judgesterm comes from BoatengpersName et al.'s work to develop a household-level water insecurity scale appropriate for use in western Kenya ( 20 ). We used the Delphi method to obtain three rounds of feedback from international experts including those in hydrology, geography, WASH and water-related programs, policy implementation, and food insecurity. Each of the three rounds was interspersed with focus groupterm discussions with our target population, i.e., people living in western Kenya. In each round, the questionnaires progressively became more closed ended, until consensus was attained on the definition of the domainterm we were studying and possible itemsterm we could use.

Phase 2: scale developmentterm·mth Step 3: pre-testing questionsterm Pre-testingterm helps to ensure that itemsterm are meaningful to the target population before the survey is actually administered, i.e., it minimizes misunderstanding and subsequent measurement errorterm. Because pre-testingterm eliminates poorly worded itemsterm and facilitates revision of phrasing to be maximally understood, it also serves to reduce the cognitive burdenterm on research participants. Finally, pre-testingterm represents an additional way in which members of the target population can participate in the research process by contributing their insights to the development of the survey.

Pre-testingterm has two components: the first is the examination of the extent to which the questions reflect the domainterm being studied. The second is the examination of the extent to which answers to the questions asked produce valid measurements ( 31 ).

Cognitive interviews To evaluate whether the questions reflect the domainterm of study and meet the requisite standards, techniques including cognitive interviews, focus groupterm discussion, and field pre-testingterm under realistic conditions can be used. We describe the most recommended, which is cognitive interviews.

Cognitive interviewingterm entails the administration of draft survey questions to target populations and then asking the respondents to verbalize the mental processterm entailed in providing such answers ( 49 ). Generally, cognitive interviews allow for questions to be modified, clarified, or augmented to fit the objectives of the study. This approach helps to determine whether the question is generating the information that the author intends by helping to ensure that respondents understand questions as developers intended and that respondents are able to answer in a manner that reflects their experience ( 49 , 50 ). This can be done on a sample outside of the study population or on a subset of study participants, but it must be explored before the questionnaire is finalized ( 51 , 52 ).

The sample used for cognitive interviewingterm should capture the range of demographics you anticipate surveying ( 49 ). A range of 5–15 interviews in two to three rounds, or until saturation, or relatively few new insights emerge is considered ideal for pre-testingterm ( 49 , 51 , 52 ).

In sum, cognitive interviews get to the heart of both assessing the appropriateness of the question to the target population and the strength of the responses ( 49 ). The advantages of using cognitive interviewingterm include: (a) it ensures questions are producing the intended data, (b) questions that are confusing to participants are identified and improved for clarity, (c) problematic questions or questions that are difficult to answer are identified, (d) it ensures response options are appropriate and adequate, (e) it reveals the thought process of participants on domainterm itemsterm, and (f) it can indicate problematic question order ( 52 , 53 ). Outcomes of cognitive interviews should always be reported, along with solutions used to remedy the situation.

An example of best practice in pre-testingterm is seen in the work of Morris et al. ( 54 ). They developed and validated a novel scale for measuring interpersonal factorsterm underlying injection drug use behaviors among injecting partners. After item developmentterm·mth and expert judgment, they conducted cognitive interviews with seven respondents with similar characteristics to the target population to refine and assess itemterm interpretation and to finalize itemterm structure. Eight itemsterm were dropped after cognitive interviews for lack of clarity or importance. They also made modifications to grammar, word choice, and answer options based on the feedback from cognitive interviews.

Step 4: survey administration and sample sizeterm Survey administration Collecting data with minimum measurement errorsterm from an adequate sample sizeterm is imperative. These data can be collected using paper and pen/pencil interviewing (PAPIterm) or Computer Assisted Personal Interviewing (CAPIterm) on devices like laptops, tablets, or phones. A number of software programs exist for building forms on devices. These include Computer Assisted Survey Information Collection (CASIC) Builder™ (West Portal Software Corporation, San Francisco, CA); Qualtrics Research Core™ ( www.qualtrics.com ); Open Data Kit (ODK, https://opendatakit.org/ ); Research Electronic Data Capture (REDCap) ( 55 ); SurveyCTO (Dobility, Inc. https://www.surveycto.com ); and Questionnaire Development System™ (QDS, www.novaresearch.com ), which allows the participant to report sensitive audio data.

Each approach has advantages and drawbacks. Using technology can reduce the errors associated with data entry, allow the collection of data from large samples with minimal cost, increase response rate, reduce enumerator errors, permit instant feedback, and increase monitoring of data collection and ability to get more confidential data ( 56 – 58 , 130 ). A subset of technology-based programs offers the option of attaching audio files to the survey questions so that questions may be recorded and read out loud to participants with low literacy via audio computer self-assisted interviewing (A-CASI) ( 131 ). Self-interviewing, whether via A-CASI or via computer-assisted personal interviewing, in which participants read and respond to questions on a computer without interviewer involvement, may increase reports of sensitive or stigmatized behaviors such as sexual behaviors and substance use, compared to when being asked by another human.

On the other hand, paper forms may avert the crisis of losing data if the software crashes, the devices are lost or stolen prior to being backed up, and may be more suitable in areas that have irregular electricity and/or internet. However, as sample sizesterm increase, the use of PAPIterm becomes more expensive, time and labor intensive, and the data are exposed in several ways to human errorterm ( 57 , 58 ). Based on the merits of CAPIterm over PAPIterm, we recommend researchers use CAPIterm in data collection for surveys when feasible.

Establishing the sample sizeterm The sample sizeterm to use for the development of a latent constructterm has often been contentious. It is recommended that potential scale itemsterm be tested on a heterogeneous sample, i.e., a sample that both reflects and captures the range of the target population ( 29 ). For example, when the scale is used in a clinical setting, Clark and Watson recommend using patient samples early on instead of a sample from the general population ( 29 ).

The necessary sample sizeterm is dependent on several aspects of any given study, including the level of variation between the variables, and the level of over-determination (i.e., the ratio of variables to number of factorsterm) of the factorsterm ( 59 ). The rule of thumb has been at least 10 participants for each scale itemterm, i.e., an ideal ratio of respondents to itemsterm is 10:1 ( 60 ). However, others have suggested sample sizesterm that are independent of the number of survey itemsterm. Clark and Watson ( 29 ) propose using 300 respondents after initial pre-testingterm. Others have recommended a range of 200–300 as appropriate for factor analysisterm ( 61 , 62 ). Based on their simulation study using different sample sizesterm, Guadagnoli and Velicer ( 61 ) suggested that a minimum of 300–450 is required to observe an acceptable comparability of patterns, and that replication is required if the sample sizeterm is < 300. Comrey and Lee suggest a graded scale of sample sizesterm for scale developmentterm·mth: 100 = poor, 200 = fair, 300 = good, 500 = very good, ≥1,000 = excellent ( 63 ). Additionally, item reductionterm procedures (described, below in Step 5), such as parallel analysis which requires bootstrappingterm (estimating statistical parameters from sample by means of resampling with replacement) ( 64 ), may require larger data sets.

In sum, there is no single item-ratio that works for all survey development scenarios. A larger sample sizeterm or respondent: itemterm ratio is always better, since a larger sample sizeterm implies lower measurement errorsterm and more, stable factor loadingsterm, replicable factorsterm, and generalizable results to the true population structure ( 59 , 65 ). A smaller sample sizeterm or respondent: itemterm ratio may mean more unstable loadings and factorsterm, random, non-replicable factorsterm, and non-generalizable results ( 59 , 65 ). Sample sizeterm is, however, always constrained by resources available, and more often than not, scale developmentterm·mth can be difficult to fund.

Determining the type of data to use The development of a scale minimally requires data from a single point in time. To fully test for the reliabilityterm of the scale (cf. Steps 8, 9), however, either an independent dataset or a subsequent time point is necessary. Data from longitudinal studies can be used for initial scale developmentterm·mth (e.g., from baseline), to conduct confirmatory factor analysisterm (using follow-up data, cf. Step 7), and to assess test–retest reliabilityterm (using baseline and follow-up data). The problem with using longitudinal data to test hypothesized latent structures is common error variance, since the same, potentially idiosyncratic, participants will be involved. To give the most credence to the reliabilityterm of scale, the ideal procedure is to develop the scale on sample A, whether cross-sectional or longitudinal, and then test it on an independent sample Bterm.

The work of Chesney et al. on the Coping Self-Efficacy scale provides an example of this best practice in the use of independent samples ( 132 ). This study sought to investigate the psychometric characteristics of the Coping Self-Efficacy (CSE) scale, and their samples came from two independent randomized clinical trials. As such, two independent samples with four different time points each (0, 3, 6, and 12 months) were used. The authors administered the 26-item scale to the sample from the first clinical trial and examined the covariance that existed between all the scale itemsterm (exploratory factor analysisterm) giving the hypothesized factor structureterm across time in that one trial. The obtained factorterm structure was then fitted to baseline data from the second randomized clinical trial to test the hypothesized factor structureterm generated in the first sample ( 132 ).

Step 5: item reduction analysisterm In scale developmentterm·mth, item reduction analysisterm is conducted to ensure that only parsimonious, functional, and internally consistent itemsterm are ultimately included ( 133 ). Therefore, the goal of this phase is to identify itemsterm that are not or are the least-related to the domainterm under study for deletion or modification.

Two theories, Classical Test Theory (CTTterm) and the Itemterm Response Theory (IRTterm), underpin scale developmentterm·mth ( 134 ). CTTterm is considered the traditional test theory and IRTterm the modern test theory; both function to produce latent constructsterm. Each theory may be used singly or in conjunction to complement the other's strengths ( 15 , 135 ). Whether the researcher is using CTTterm or IRTterm, the primary goal is to obtain functional itemsterm (i.e., itemsterm that are correlated with each other, discriminate between individual cases, underscore a single or multidimensional domainterm, and contribute significantly to the constructterm).

CTTterm allows the prediction of outcomes of constructsterm and the difficulty of itemsterm ( 136 ). CTTterm models assume that itemsterm forming constructsterm in their observed, manifest formsterm consist of a true scoreterm on the domainterm of interest and a random errorterm (which is the differences between the true scoreterm and a set of observed scores by an individual) ( 137 ). IRTterm seeks to model the way in which constructsterm manifest themselves in terms of observable itemterm response ( 138 ). Comparatively, the IRTterm approach to scale developmentterm·mth has the advantage of allowing the researcher to determine the effect of adding or deleting a given itemterm or set of itemsterm by examining the item informationterm and standard error functions for the itemterm pool ( 138 ).

Several techniques exist within the two theories to reduce the itemterm pool, depending on which test theory is driving the scale. The five major techniques used are: itemterm difficulty and itemterm discrimination indices, which are primarily for binary responses; inter-item and item-total correlations, which are mostly used for categorical itemsterm; and distractor efficiency analysis for itemsterm with multiple choice response options ( 1 , 2 ).

Item difficulty indexterm The item difficulty indexterm is both a CTTterm and an IRTterm parameter that can be traced largely to educational and psychological testing to assess the relative difficulties and discrimination abilities of test itemsterm ( 66 ). Subsequently, this approach has been applied to more attitudinal-type scales designed to measure latent constructsterm.

Under the CTTterm framework, the item difficulty indexterm, also called item easinessterm, is the proportion of correct answers on a given itemterm, e.g., the proportion of correct answers on a math test ( 1 , 2 ). It ranges between 0.0 and 1.0. A high difficulty score means a greater proportion of the sample answered the question correctly. A lower difficulty score means a smaller proportion of the sample understood the question and answered correctly. This may be due to the itemterm being coded wrongly, ambiguity with the itemterm, confusing language, or ambiguity with response options. A lower difficulty score suggests a need to modify the itemsterm or delete them from the pool of itemsterm.

Under the IRTterm framework, the itemterm difficulty parameter is the probability of a particular examinee correctly answering any given itemterm ( 67 ). This has the advantage of allowing the researcher to identify the different levels of individual performance on specific questions, as well as develop particular questions to specific subgroups or populations ( 67 ). Itemterm difficulty is estimated directly using logistic models instead of proportions.

Researchers must determine whether they need itemsterm with low, medium, or high difficulty. For instance, researchers interested in general purpose scales will focus on itemsterm with medium difficulty ( 68 ), i.e., the proportion with itemterm assertions ranging from 0.4 to 0.6 ( 2 , 68 ). The item difficulty indexterm can be calculated using existing commands in M plus , R, SAS, SPSS, or Stata.

Item discrimination indexterm The item discrimination indexterm (also called item-effectiveness testterm), is the degree to which an itemterm correctly differentiates between respondents or examinees on a constructterm of interest ( 69 ), and can be assessed under both CTTterm and IRTterm frameworks. It is a measure of the difference in performance between groups on a constructterm. The upper group represents participants with high scores and the lower group those with poor or low scores. The item discrimination indexterm is “calculated by subtracting the proportion of examinees in the lower group (lower %) from the proportion of examinees in the upper group (upper %) who got the itemterm correct or endorsed the itemterm in the expected manner” ( 69 ). It differentiates between the number of students in an upper group who get an itemterm correct and the number of students in a lower group who get the itemterm correct ( 70 ). The use of an item discrimination indexterm enables the identification of positively discriminating itemsterm (i.e., itemsterm that differentiate rightly between those who are knowledgeable about a subject and those who are not), negatively discriminating itemsterm (i.e., itemsterm which are poorly designed such that the more knowledgeable get them wrong and the less knowledgeable get them right), and non-discriminating itemterm (i.e., itemsterm that fail to differentiate between participants who are knowledgeable about a subject and those who are not) ( 70 ).

The item discrimination indexterm has been found to improve test itemsterm in at least three ways. First, non-discriminating itemsterm, which fail to discriminate between respondents because they may be too easy, too hard, or ambiguous, should be removed ( 71 ). Second, itemsterm which negatively discriminate, e.g., itemsterm which fail to differentiate rightly between medically diagnosed depressed and non-depressed respondents on a happiness scale, should be reexamined and modified ( 70 , 71 ). Third, itemsterm that positively discriminate should be retained, e.g., itemsterm that are correctly affirmed by a greater proportion of respondents who are medically free of depression, with very low affirmation by respondents diagnosed to be medically depressed ( 71 ). In some cases, it has been recommended that such positively discriminating itemsterm be considered for revision ( 70 ) as the differences could be due to the level of difficulty of the itemterm.

An item discrimination indexterm can be calculated through correlational analysis between the performance on an itemterm and an overall criterion ( 69 ) using either the point biserial correlation coefficient or the phi coefficient ( 72 ).

Itemterm discrimination under the IRTterm framework is a slope parameter that determines how steeply the probability of a correct response changes as the proficiency or trait increases ( 73 ). This allows differentiation between individuals with similar abilities and can also be estimated using a logistic model. Under certain conditions, the biserial correlation coefficient under the CTTterm framework has proven to be identical to the IRTterm itemterm discrimination parameter ( 67 , 74 , 75 ); thus, as the trait increases so does the probability of endorsing an itemterm. These parameters can be computed using existing commands in M plus , R, SAS, SPSS, or Stata. In both CTTterm and IRTterm, higher values are indicators of greater discrimination ( 73 ).

Inter-item and item-total correlations A third technique to support the deletion or modification of itemsterm is the estimation of inter-item and item-total correlations, which falls under CTTterm. These correlations often displayed in the form of a matrix are used to examine relationships that exist between individual itemsterm in a pool.

Inter-item correlationsterm (also known as polychoric correlationsterm for categorical variables and tetrachoric correlationsterm for binary itemsterm) examines the extent to which scores on one itemterm are related to scores on all other itemsterm in a scale ( 2 , 68 , 76 ). Also, it examines the extent to which itemsterm on a scale are assessing the same content ( 76 ). Itemsterm with very low correlations (< 0.30) are less desirable and could be a cue for potential deletion from the tentative scale.

Item-total correlations (also known as polyserial correlations for categorical variables and biserial correlations for binary itemsterm) aim at examining the relationship between each itemterm vs. the total score of scale itemsterm. However, the adjusted item-total correlation, which examines the correlation between the itemterm and the sum score of the rest of the itemsterm excluding itself is preferred ( 1 , 2 ). Itemsterm with very low adjusted item-total correlations (< 0.30) are less desirable and could be a cue for potential deletion from the tentative scale. Inter-item and itemterm total correlations can be calculated using M plus , R, SAS, SPSS, or Stata.

Distractor efficiency analysis The distractor efficiency analysis shows the distribution of incorrect options and how they contribute to the quality of a multiple-choice itemterm ( 77 ). The incorrect options, also known as distractors, are intentionally added in the response options to attract students who do not know the correct answer in a test question ( 78 ). To calculate this, respondents will be grouped into three groups—high, middle, and lower tertiles based on their total scores on a set of itemsterm. Itemsterm will be regarded as appropriate if 100% of those in the high group choose the correct response options, about 50% of those in the middle choose the correct option, and few or none in the lower group choose the correct option ( 78 ). This type of analysis is rarely used in the health sciences, as most multiple-choice itemsterm are on a Likert-type response scaleterm·mth and do not test respondent correct knowledge, but their experience or perception. However, distractor analysis can help to determine whether itemsterm are well-constructed, meaningful, and functional when researchers add response options to questions that do not fit a particular experience. It is expected that participants who are determined as having poor knowledge or experience on the constructterm will choose the distractors, while those with the right knowledge and experience will choose the correct response options ( 77 , 79 ). Where those with the right knowledge and experience are not able to differentiate between distractors and the right response, the question may have to be modified. Non-functional distractors identified need to be removed and replaced with efficient distractors ( 80 ).

Missing cases In addition to these techniques, some researchers opt to delete itemsterm with large numbers of cases that are missing, when other missing data-handling techniques cannot be used ( 81 ). For cases where modern missing data handling can be used, however, several techniques exist to solve the problem of missing cases. Two of the approaches have proven to be very useful for scale developmentterm·mth: full information maximum likelihoodterm (FIML) ( 82 ) and multiple imputationterm ( 83 ). Both methods can be applied using existing commands in statistical packages such as M plus , R, SAS, and Stata. When using multiple imputationterm to recover missing data in the context of survey research, the researcher can impute individual itemsterm prior to computing scale scoresterm or impute the scale scoresterm from other scale scoresterm ( 84 ). However, item-level imputationterm has been shown to produce more efficient estimates over scale-level imputation. Thus, imputing individual itemsterm before scale developmentterm·mth is a preferred approach to imputing newly developed scales for missing cases ( 84 ).

Step 6: extraction of factorsterm Factorterm extraction is the phase in which the optimal number of factorsterm, sometimes called domainsterm, that fit a set of itemsterm are determined. This is done using factor analysisterm. Factor analysisterm is a regression modelterm·mth in which observed standardized variables are regressed on unobserved (i.e., latent) factorsterm. Because the variables and factorsterm are standardized, the bivariate regression coefficients are also correlations, representing the loading of each observed variable on each factorterm. Thus, factor analysisterm is used to understand the latent (internal) structure of a set of itemsterm, and the extent to which the relationships between the itemsterm are internally consistent ( 4 ). This is done by extracting latent factorsterm which represent the shared varianceterm in responses among the multiple itemsterm ( 4 ). The emphasis is on the number of factorsterm, the salience of factor loadingterm estimates, and the relative magnitude of residual variancesterm ( 2 ).

A number of analytical processes have been used to determine the number of factorsterm to retain from a list of itemsterm, and it is beyond the scope of this paper to describe all of them. For scale developmentterm·mth, commonly available methods to determine the number of factorsterm to retain include a scree plot ( 85 ), the variance explained by the factorterm model, and the pattern of factor loadingsterm ( 2 ). Where feasible, researchers could also assess the optimal number of factorsterm to be drawn from the list of itemsterm using either parallel analysis ( 86 ), minimum average partial procedure ( 87 ), or the Hull method ( 88 , 89 ).

The extraction of factorsterm can also be used to reduce itemsterm. With factor analysisterm, itemsterm with factor loadingsterm or slope coefficients that are below 0.30 are considered inadequate as they contribute <10% variation of the latent constructterm measured. Hence, it is often recommended to retain itemsterm that have factor loadingsterm of 0.40 and above ( 2 , 60 ). Also, itemsterm with cross-loadingsterm or that appear not to load uniquely on individual factorsterm can be deleted. For single-factor models in which Rasch IRTterm modeling is used, itemsterm are selected as having a good fit based on mean-square residual summary statistics (infit and outfit) >0.4 and <1.6 ( 90 ).

A number of scales developed stop at this phase and jump to tests of reliabilityterm, but the factorsterm extracted at this point only provide a hypothetical structure of the scale. The dimensionality of these factorsterm need to be tested (cf. Step 7) before moving on to reliabilityterm (cf. Step 8) and validityterm (cf. Step 9) assessment.

Phase 3: scale evaluationterm·mth Step 7: tests of dimensionalityterm The test of dimensionalityterm is a test in which the hypothesized factorsterm or factorterm structure extracted from a previous model is tested at a different time point in a longitudinal study or, ideally, on a new sample ( 91 ). Tests of dimensionalityterm determine whether the measurement of itemsterm, their factorsterm, and function are the same across two independent samples or within the same sample at different time points. Such tests can be conducted using independent cluster model (ICM)-confirmatory factor analysisterm, bifactor modelingterm, or measurement invarianceterm.

Confirmatory factor analysisterm Confirmatory factor analysisterm is a form of psychometric assessment that allows for the systematic comparison of an alternative a priori factorterm structure based on systematic fit assessment procedures and estimates the relationship between latent constructsterm, which have been corrected for measurement errorsterm ( 92 ). Morin et al. ( 92 ) note that it relies on a highly restrictive ICM, in which cross-loadingsterm between itemsterm and non-target factorsterm are assumed to be exactly zero. The systematic fit assessment procedures are determined by meaningful satisfactory thresholds; Table 2 contains the most common techniques for testing dimensionality. These techniques include the chi-square test of exact fit, Root Mean Square Error of Approximation (RMSEA ≤ 0.06), Tucker Lewis Index (TLI ≥ 0.95), Comparative Fit Index (CFI ≥ 0.95), Standardized Root Mean Square Residual (SRMR ≤ 0.08), and Weighted Root Mean Square Residual (WRMR ≤ 1.0) ( 90 , 92 – 101 ).

Description of model fit indices and thresholds for evaluating scales developed for health, social, and behavioral research.

Bifactor modelingterm Bifactor modelingterm, also referred to as nested factor modelingterm, is a form of itemterm response theory used in testing dimensionality of a scale ( 102 , 103 ). This method can be used when the hypothesized factor structureterm from the previous model produces partially overlapping dimensions so that one could be seeing most of the itemsterm loading onto one factorterm and a few itemsterm loading onto a second and/or a third factorterm. The bifactor model allows researchers to estimate a unidimensional constructterm while recognizing the multidimensionality of the constructterm ( 104 , 105 ). The bifactor model assumes each itemterm loads onto two dimensions, i.e., itemsterm forming the constructterm may be associated with more than one source of true scoreterm variance ( 92 ). The first is a general latent factorterm that underlies all the scale itemsterm and the second, a group factorterm (subscale). A “bifactor model is based on the assumption that a f -factor solution exists for a set of n itemsterm with one [general]/Global (G) factorterm and f – 1 Specific (S) factorsterm also called group factorsterm” ( 92 ). This approach allows researchers to examine any distortion that may occur when unidimensional IRTterm models are fit to multidimensional data ( 104 , 105 ). To determine whether to retain a constructterm as unidimensional or multidimensional, the factor loadingsterm from the general factorterm are then compared to those from the group factorsterm ( 103 , 106 ). Where the factor loadingsterm on the general factorterm are significantly larger than the group factorsterm, a unidimensional scale is implied ( 103 , 104 ). This method is assessed based on meaningful satisfactory thresholds. Alternatively, one can test for the coexistence of a general factorterm that underlies the constructterm and multiple group factorsterm that explain the remaining variance not explained by the general factorterm ( 92 ). Each of these methods can be done using statistical software such as M plus , R, SAS, SPSS, or Stata.

Measurement invarianceterm Another method to test dimensionality is measurement invarianceterm, also referred to as factorial invarianceterm or measurement equivalenceterm ( 107 ). Measurement invarianceterm concerns the extent to which the psychometric propertiesterm of the observed indicators are transportable (generalizable) across groups or over time ( 108 ). These properties include the hypothesized factor structureterm, regression slopes, intercept, and residual variancesterm. Measurement invarianceterm is tested sequentially at five levels—configural, metric, scalar, strict (residual), and structural ( 107 , 109 ). Of key significance to the test of dimensionalityterm is configural invariance, which is concerned with whether the hypothesized factor structureterm is the same across groups. This assumption has to be met in order for subsequent tests to be meaningful ( 107 , 109 ). For example, a hypothesized unidimensional structure, when tested across multiple countries, should be the same. This can be tested in CTTterm, using multigroup confirmatory factor analysisterm ( 110 – 112 ).

An alternative approach to measurement invarianceterm in the testing of unidimensionality under itemterm response theory is the Rasch measurement model for binary itemsterm and polytomous IRTterm models for categorical itemsterm. Here, emphasis is on testing the differential itemterm functioning (DIF)—an indicator of whether “a group of respondents is scoring better than another group of respondents on an itemterm or a test after adjusting for the overall ability scores of the respondents” ( 108 , 113 ). This is analogous to the conditions underpinning measurement invarianceterm in a multi-group CFA ( 108 , 113 ).

Whether the hypothesized structure is bidimensional or multidimensional, each dimension in the structure needs to be tested again to confirm its unidimensionality. This can also be done using confirmatory factor analysisterm. Appropriate model fit indices and the strength of factor loadingsterm (cf. Table 2 ) are the basis on which the latent structure of the itemsterm can be judged.

One commonly encountered pitfall is a lack of satisfactory global model fit in confirmatory factor analysisterm conducted on a new sample following a satisfactory initial factor analysisterm performed on a previous sample. Lack of satisfactory fit offers the opportunity to identify additional underperforming itemsterm for removal. Itemsterm with very poor loadings (≤0.3) can be considered for removal. Also, modification indices, produced by M plus and other structural equation modelingterm (SEM) programs, can help identify itemsterm that need to be modified. Sometimes a higher-order factorterm structure, where correlations among the original factorsterm can be explained by one or more higher-order factorsterm, is needed. This can also be assessed using statistical software such as M plus , R, SAS, SPSS, or Stata.

A good example of best practice is seen in the work of Pushpanathan et al. on the appropriateness of using a traditional confirmatory factor analysisterm or a bifactor model ( 114 ) in assessing whether the Parkinson's Disease Sleep Scale-Revised was better used as a unidimensional scale, a tri-dimensional scale, or a scale that has an underlying general factorterm and three group factorsterm (sub-scales). They tested this using three different models—a unidimensional model (1-factor CFA); a 3-factor model (3 factorterm CFA) consisting of sub-scales measuring insomnia, motor symptoms and obstructive sleep apnea, and REM sleep behavior disorder; and a confirmatory bifactor model having a general factorterm and the same three sub-scales combined. The results of this study suggested that only the bifactor model with a general factorterm and the three sub-scales combined achieved satisfactory model fitness. Based on these results, the authors cautioned against the use of a unidimensional total scale scoresterm as a cardinal indicator of sleep in Parkinson's disease, but encouraged the examination of its multidimensional subscales ( 114 ).

Scoring scale itemsterm Finalized itemsterm from the tests of dimensionalityterm can be used to create scale scoresterm for substantive analysis including tests of reliabilityterm and validityterm. Scale scoresterm can be calculated by using unweighted or weighted procedures. The unweighted approachterm involves summing standardized itemterm scores or raw itemterm scores, or computing the mean for raw itemterm scores ( 115 ). The weighted approachterm in calculating scale scoresterm can be produced via statistical software programs such as M plus , R, SAS, SPSS, or Stata. For instance, in using confirmatory factor analysisterm, structural equation models, or exploratory factor analysisterm, each factorterm produced reveals a statistically independent source of variation among a set of itemsterm ( 115 ). The contribution of each individual itemterm to this factorterm is considered a weight, with the factor loadingterm value representing the weight. The scores associated with each factorterm in a model then represents a composite scale scoreterm based on a weighted sum of the individual itemsterm using factor loadingsterm ( 115 ). In general, it does not make much difference in the performance of the scale if scales are computed as unweighted itemsterm (e.g., mean or sum scores) or weighted itemsterm (e.g., factorterm scores).

Step 8: tests of reliabilityterm Reliabilityterm is the degree of consistencyterm exhibited when a measurement is repeated under identical conditions ( 116 ). A number of standard statistics have been developed to assess reliabilityterm of a scale, including Cronbach's alphaterm ( 117 ), ordinal alpha ( 118 , 119 ) specific to binary and ordinal scale itemsterm, test–retest reliabilityterm (coefficient of stabilityterm) ( 1 , 2 ), McDonald's Omega ( 120 ), Raykov's rho ( 2 ) or Revelle's beta ( 121 , 122 ), split-half estimates, Spearman-Brown formula, alternate form method (coefficient of equivalence), and inter-observer reliabilityterm ( 1 , 2 ). Of these statistics, Cronbach's alphaterm and test–retest reliabilityterm are predominantly used to assess reliabilityterm of scales ( 2 , 117 ).

Cronbach's alphaterm Cronbach's alphaterm assesses the internal consistency of the scale itemsterm, i.e., the degree to which the set of itemsterm in the scale co-vary, relative to their sum score ( 1 , 2 , 117 ). An alpha coefficient of 0.70 has often been regarded as an acceptable threshold for reliabilityterm; however, 0.80 and 0.95 is preferred for the psychometric quality of scales ( 60 , 117 , 123 ). Cronbach's alphaterm has been the most common and seems to have received general approval; however, reliabilityterm statistics such as Raykov's rho, ordinal alpha, and Revelle's beta, which are debated to have improvements over Cronbach's alphaterm, are beginning to gain acceptance.

Test–retest reliabilityterm An additional approach in testing reliabilityterm is the test–retest reliabilityterm. The test–retest reliabilityterm, also known as the coefficient of stabilityterm, is used to assess the degree to which the participants' performance is repeatable, i.e., how consistent their sum scores are across time ( 2 ). Researchers vary in how they assess test–retest reliabilityterm. While some prefer to use intra class correlation coefficient ( 124 ), others use the Pearson product-moment correlation ( 125 ). In both cases, the higher the correlation, the higher the test–retest reliabilityterm, with values close to zero indicating low reliabilityterm. In addition, study conditions could change values on the constructterm being measured over time (as in an intervention study, for example), which could lower the test-retest reliabilityterm.

The work of Johnson et al. ( 16 ) on the validation of the HIV Treatment Adherence Self-Efficacy Scale (ASES) is a good example of the test of reliabilityterm. As part of testing for reliabilityterm, the authors tested for the internal consistency reliabilityterm values for the ASES and its subscales using Raykov's rho (produces a coefficient similar to alpha but with fewer assumptions and with confidence intervals); they then tested for the temporal consistency of the ASES' factorterm structure. This was then followed by test–retest reliabilityterm assessment among the latent factorsterm. The different approaches provided support for the reliabilityterm of the ASES scale.

Other approaches found to be useful and support scale reliabilityterm include split-half estimates, Spearman-Brown formula, alternate form method (coefficient of equivalence), and inter-observer reliabilityterm ( 1 , 2 ).

Step 9: tests of validityterm Scale validityterm is the extent to which “an instrument indeed measures the latent dimension or constructterm it was developed to evaluate” ( 2 ). Although it is discussed at length here in Step 9, validation is an ongoing process that starts with the identification and definition of the domainterm of study (Step 1) and continues to its generalizability with other constructsterm (Step 9) ( 36 ). The validityterm of an instrument can be examined in numerous ways; the most common tests of validityterm are content validityterm (described in Step 2), which can be done prior to the instrument being administered to the target population, and criterion (predictive and concurrent) and construct validityterm (convergent, discriminant, differentiation by known groups, correlations), which occurs after survey administration.

Criterion validityterm Criterion validityterm is the “degree to which there is a relationship between a given test score and performance on another measure of particular relevance, typically referred to as criterion” ( 1 , 2 ). There are two forms of criterion validityterm: predictive (criterion) validityterm and concurrent (criterion) validityterm. Predictive validityterm is “the extent to which a measure predicts the answers to some other question or a result to which it ought to be related with” ( 31 ). Thus, the scale should be able to predict a behavior in the future. An example is the ability for an exclusive breastfeeding social support scale to predict exclusive breastfeeding ( 10 ). Here, the mother's willingness to exclusively breastfeed occurs after social support has been given, i.e., it should predict the behavior. Predictive validityterm can be estimated by examining the association between the scale scoresterm and the criterion in question.

Concurrent criterion validityterm is the extent to which test scores have a stronger relationship with criterion (gold standard) measurement made at the time of test administration or shortly afterward ( 2 ). This can be estimated using Pearson product-moment correlation or latent variable modeling. The work of Greca and Stone on the psychometric evaluation of the revised version of a social anxiety scale for children (SASC-R) provides a good example for the evaluation of concurrent validityterm ( 140 ). In this study, the authors collected data on an earlier validated version of the SASC scale consisting of 10 itemsterm, as well as the revised version, SASC-R, which had additional 16 itemsterm making a 26-item scale. The SASC consisted of two sub scales [fear of negative evaluation (FNE), social avoidance and distress (SAD)] and the SASC-R produced three new subscales (FNE, SAD-New, and SAD-General). Using a Pearson product-moment correlation, the authors examined the inter-correlations between the common subscales for FNE, and between SAD and SAD-New. With a validityterm coefficient of 0.94 and 0.88, respectively, the authors found evidence of concurrent validityterm.

A limitation of concurrent validityterm is that this strategy for validityterm does not work with small sample sizesterm because of their large sampling errors. Secondly, appropriate criterion variables or “gold standards” may not be available ( 2 ). This reason may account for its omission in most validation studies.

Construct validityterm Construct validityterm is the “extent to which an instrument assesses a constructterm of concern and is associated with evidence that measures other constructsterm in that domainterm and measures specific real-world criteria” ( 2 ). Four indicators of construct validityterm are relevant to scale developmentterm·mth: convergent validityterm, discriminant validityterm, differentiation by known groups, and correlation analysisterm.

Convergent validityterm is the extent to which a constructterm measured in different ways yields similar results. Specifically, it is the “degree to which scores on a studied instrument are related to measures of other constructsterm that can be expected on theoretical grounds to be close to the one tapped into by this instrument” ( 2 , 37 , 126 ). This is best estimated through the multi-trait multi-method matrix ( 2 ), although in some cases researchers have used either latent variable modeling or Pearson product-moment correlation based on Fisher's Z transformation. Evidence of convergent validityterm of a constructterm can be provided by the extent to which the newly developed scale correlates highly with other variables designed to measure the same constructterm ( 2 , 126 ). It can be invalidated by too low or weak correlations with other tests which are intended to measure the same constructterm.

Discriminant validityterm is the extent to which a measure is novel and not simply a reflection of some other constructterm ( 126 ). Specifically, it is the “degree to which scores on a studied instrument are differentiated from behavioral manifestations of other constructsterm, which on theoretical grounds can be expected not to be related to the constructterm underlying the instrument under investigation” ( 2 ). This is best estimated through the multi-trait multi method matrix ( 2 ). Discriminant validityterm is indicated by predictably low or weak correlations between the measure of interest and other measures that are supposedly not measuring the same variable or concept ( 126 ). The newly developed constructterm can be invalidated by too high correlations with other tests which are intended to differ in their measurements ( 37 ). This approach is critical in differentiating the newly developed constructterm from other rival alternatives ( 36 ).

Differentiation or comparison between known groups examines the distribution of a newly developed scale scoreterm over known binary itemsterm ( 126 ). This is premised on previous theoretical and empirical knowledge of the performance of the binary groups. An example of best practice is seen in the work of BoatengpersName et al. on the validation of a household water insecurity scale in Kenya. In this study, we compared the mean household water insecurity scores over households with or without E. coli present in their drinking water. Consistent with what we knew from the extant literature, we found households with E. coli present in their drinking water had higher mean water insecurity scores than households that had no E. coli in drinking water. This suggested our scale could discriminate between particular known groups.

Although correlational analysis is frequently used by several scholars, bivariate regression analysisterm is preferred to correlational analysis for quantifying validityterm ( 127 , 128 ). Regression analysis between scale scoresterm and an indicator of the domainterm examined has a number of important advantages over correlational analysis. First, regression analysis quantifies the association in meaningful units, facilitating judgment of validityterm. Second, regression analysis avoids confounding validityterm with the underlying variation in the sample and therefore the results from one sample are more applicable to other samples in which the underlying variation may differ. Third, regression analysis is preferred because the regression modelterm·mth can be used to examine discriminant validityterm by adding potential alternative measures. In addition to regression analysis, alternative techniques such as analysis of standard deviations of the differences between scores and the examination of intraclass correlation coefficients (ICC) have been recommended as viable options ( 128 ).

Taken together, these methods make it possible to assess the validityterm of an adapted or a newly developed scale. In addition to predictive validityterm, existing studies in fields such as health, social, and behavioral sciencesterm·dom have shown that scale validityterm is supported if at least two of the different forms of construct validityterm discussed in this section have been examined. Further information about establishing validityterm and constructing indictors from scales can be found in FrongillopersName et al. ( 141 ).

Conclusions In sum, we have sought to give an overview of the key steps in scale development and validationterm·mth (Figure 1 ) as well as to help the reader understand how one might approach each step (Table 1 ). We have also given a basic introduction to the conceptual and methodological underpinnings of each step.

The three phasesterm and nine stepsterm of scale development and validationterm·mth.

Because scale developmentterm·mth is so complicated, this should be considered a primer, i.e., a “jumping off point” for anyone interested in scale developmentterm·mth. The technical literatureterm and examples of rigorous scale developmentterm·mth mentioned throughout will be important for readers to pursue. There are a number of matters not addressed here, including how to interpret scale output, the designation of cut-offs, when indices, rather than scales, are more appropriate, and principles for re-testing scales in new populations. Also, this reviewterm·mth leans more toward the classical test theory approach to scale developmentterm·mth; a comprehensive reviewterm·mth on IRTterm modeling will be complementary. We hope this reviewterm·mth helps to ease readers into the literature, but space precludes consideration of all these topics.

The necessity of the nine stepsterm that we have outlined here (Table 1 , Figure 1 ) will vary from study to study. While studies focusing on developing scales de novo may use all nine stepsterm, others, e.g., those that set out to validate existing scales, may end up using only the last four steps. Resource constraints, including time, money, and participant attention and patience are very real, and must be acknowledged as additional limits to rigorous scale developmentterm·mth. We cannot state which steps are the most important; difficult decisions about which steps to approach less rigorously can only be made by each scale developer, based on the purpose of the research, the proposed end-users of the scale, and resources available. It is our hope, however, that by outlining the general shape of the phases and steps in scale developmentterm·mth, researchers will be able to purposively choose the steps that they will include, rather than omitting a step out of lack of knowledge.

Well-designed scales are the foundation of much of our understanding of a range of phenomena, but ensuring that we accurately quantify what we purport to measure is not a simple matter. By making scale developmentterm·mth more approachable and transparent, we hope to facilitate the advancement of our understanding of a range of health, social, and behavioral outcomes.

Author contributions GB and SY developed the first draft of the scale development and validationterm·mth manuscript. All authors participated in the editing and critical revision of the manuscript and approved the final version of the manuscript for publication.

Conflict of interest statement The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments We would like to acknowledge the importance of the works of several scholars of scale development and validationterm·mth used in developing this primer, particularly Robert DeVellis, Tenko Raykov, George Marcoulides, David Streiner, and Betsy McCoach. We would also like to acknowledge the help of Josh Miller of Northwestern University for assisting with design of Figure 1 and development of Table 1 , and we thank Zeina Jamuladdine for helpful comments on tests of unidimensionality.

Glossary audio computer self-assisted interviewing adherence self-efficacy scale computer assisted personal interviewing confirmatory factor analysisterm computer assisted survey information collection builder comparative fit index classical test theory differential itemterm functioning exploratory factor analysisterm full information maximum likelihoodterm fear of negative evaluation global factorterm intraclass correlation coefficient Independent cluster model itemterm response theory Open Data Kit paper and pen/pencil interviewing Questionnaire Development System root mean square error of approximation social avoidance and distress statistical analysis systems social anxiety scale for children revised structural equation model statistical package for the social sciences statistics and data standardized root mean square residual of approximation Tucker Lewis Index water, sanitation, and hygiene weighted root mean square residual.

Footnotes References