10⁹ into 109. The quotes are reproduced exactly as the extraction left them and have NOT been repaired — the meaning of such a code point belongs to the one PDF it came from, and guessing it would produce a clean, confident, unverifiable quote.standOff. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote shown on the same row, so the borrowing can be checked against the thing borrowed from without leaving this page. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.| relation to other work | claim_relations | how this paper stands to another: agrees, contradicts, extends, qualifies |
| claim type | claim_type | the shape of the claim: existence, mechanism, method, critique, prediction, null |
| concept edge | concept_edges | an asserted relation between two concepts, with the sentence asserting it |
| concept | concepts | an idea the paper works with, quoted where the paper uses it |
| content tag | content_tags | a finding, definition, quantity, statistic or variable named in the paper |
| contribution | contribution | what the paper claims to add |
| control | controls | an experiment run to eliminate a rival explanation |
| limitation | limitations | what the paper says its evidence cannot support |
| mechanism | mechanisms | how the reported effect is said to come about |
| method | method | how the paper produced its evidence |
| object of study | object_of_study | what the paper is about — the thing it investigates |
| phenomenon | phenomenon | the effect or regularity under study |
| research question | research_question | the question the paper sets itself |
| sense | senses | a distinct use of a term, quoted where the paper fixes it |
| unrecordable relation | unhoused_relations | a relation the paper asserts that this vocabulary cannot express, with the reason |
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
How good are my data and what is the resolution? Evans Philip R. a * Murshudov Garib N. a a MRC Laboratory of Molecular Biology, Hills Road, Cambridge CB2 0QH, England Correspondence e-mail: pre@mrc-lmb.cam.ac.uk 01 7 2013 13 6 2013 69 Pt 7 224279 d130700 1204 1214 07 9 2012 02 1 2013 13 06 2013 28 06 2013 13 05 2024 © Evans & Murshudov 2013 2013 https://creativecommons.org/licenses/by/2.0/uk/ This is an open-access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are cited. A full version of this article is available from Crystallography Journals Online. concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗The new scaling program AIMLESS is described and tests of refinements at different resolutions are compared with analyses from the scaling step.
Following integration of the observed diffraction spots, the process of ‘data reduction’ initially aims to determine the point-group symmetry of the data and the likely space group. This can be performed with the program POINTLESS . The scaling program then puts all the measurements on a common scale, averages measurements of symmetry-related reflections (using the symmetry determined previously) and produces many statistics that provide the first important measures of data quality. A new scaling program, AIMLESS , implements scaling models similar to those in SCALA but adds some additional analyses. From the analyses, a number of decisions can be made about the quality of the data and whether some measurements should be discarded. contributionsensescontent_tags↗The effective ‘resolution’ of a data set is a difficult and possibly contentious question (particularly with referees of papers) and this is discussed in the light of tests comparing the data-processing statistics with trials of refinement against observed and simulated data, and automated model-building and comparison of maps calculated with different resolution limits. These trials show that adding weak high-resolution data beyond the commonly used limits may make some improvement and does no harm.
data reduction data scaling software data statistics Data collection and processing pmc-status-qastatus 0 pmc-status-live yes pmc-status-embargo no pmc-status-released yes pmc-prop-open-access yes pmc-prop-olf no pmc-prop-manuscript no pmc-prop-legally-suppressed no pmc-prop-has-pdf yes pmc-prop-has-supplement no pmc-prop-pdf-only no pmc-prop-suppress-copyright no pmc-prop-is-real-version no pmc-prop-is-scanned-article no pmc-prop-preprint no pmc-prop-in-epmc yes pmc-license-ref CC BY 1. Introduction
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concept_edgesconcept_edgesFollowing integration of the spots on a set of X-ray diffraction images to produce a list of reflection intensities, a series of operations are performed on the data, usually referred to as ‘data reduction’. These processes include determination of the point group and if possible the space group, checking for consistent indexing where there are alternatives, putting all of the data on a common scale, deciding whether to reject parts of the collected data or to cut the resolution and estimating the structure amplitude | F | from intensity. Statistics on the internal consistency of the data also provide a good indication of the overall quality of the data set. Algorithms and methods for data reduction have been well documented in many papers ( e.g. Fox & Holmes, 1966 ▶ ; Otwinowski et al. , 2003 ▶ ; Evans, 2006 ▶ , 2011 ▶ ; Kabsch, 2010 ▶ ) and the details will not be repeated here. Scaling attempts to correct for contributions to the measured intensities arising from experimental conditions which vary during data collection, such as variations in the incident-beam intensity, the volume of the crystal illuminated, absorption in the primary or secondary beam and average radiation damage. This is performed by trying to make all symmetry-related or replicated measurements of a reflection intensity equal, i.e. to make the data as internally consistent as possible.
This paper describes a new data-scaling program, AIMLESS , and discusses criteria for deciding the ‘resolution’ of a measured data set. In the CCP 4 context (Winn et al. , 2011 ▶ ), AIMLESS is used immediately after the program unhoused_relationsPOINTLESS , which determines the likely point group and possible space group, as well as optionally combining data from multiple files each containing a ‘sweep’ of data and putting them on a common indexing system if necessary (Evans, 2011 ▶ ). 1 AIMLESS is then followed by unhoused_relationsCTRUNCATE , which calculates the structure amplitudes | F | from the intensities and outputs various intensity statistics mainly to detect twinning.
2. The scaling program AIMLESS
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationsconcept_edgesAIMLESS is a new implementation in C++ of a classic scaling method, designed to make it easy to add new scaling models and algorithms. It is a replacement for the earlier CCP 4 program SCALA (Evans, 2006 ▶ , 2011 ▶ ) and at present uses a very similar scale model. The function minimized is where I h l is the l th observation of reflection h , g h l is its associated inverse scale, w h l = 1/σ 2 ( I h l ) and 〈 I h 〉 is the weighted average intensity for all observations l of reflection h or its symmetry mates. The inverse scale g h l comes from the refined scale model and is a function of the crystal rotation angle ϕ as a proxy for primary beam direction s 1 and for radiation dose (or time) and the secondary beam direction s 2 : g = g primary (ϕ)exp[−2 B (ϕ)sin 2 θ/λ 2 ] g secondary ( s 2 ). unhoused_relationsThe relative B -factor term is largely an average radiation-damage correction. The smoothed primary scale factors g primary (ϕ) and relative B factors B (ϕ) are determined at suitable intervals in ϕ and are interpolated using Gaussian weights. The secondary beam correction g secondary ( s 2 ) is determined as a sum of spherical harmonic terms. concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗Parameter restraint terms (ties) include a sphericity restraint on g secondary ( s 2 ), tying all spherical harmonic coefficients to zero and optional ties between adjacent primary scales and relative B factors.
AIMLESS iterates the scaling step with an optimization of the standard error estimates on each observation σ( I h l ) and outlier rejection. Note that the scaling process is generally hugely overdetermined (many more observations than parameters, e.g. 45 000 observations for 30 parameters) and that weak intensities do not contain much useful information about the scales, so that the scaling can be made faster (roughly linearly) by working with a selected subset of strong reflections. The principal steps in the process (in the current version) are as follows. (i) Read all observations into a reflection-list object ( SCALA does not store the observations but rereads them for each scaling cycle etc. , as storing large data sets was impracticable when SCALA was written). Sort symmetry-related observations and partial observations together if necessary.
(ii) Set up the scaling model depending on what data are present and explicit user-given control instructions (if any). For the smoothly varying scale and B factors, decide on suitable interval for the scales for each ‘run’ or ‘sweep’ of contiguous images, depending on the total length of the sweep and default or specified values.
(iii) Obtain initial rough scale estimates from average intensities.
(iv) First-round scaling with a sample of a few thousand strong reflections with I /σ( I ) greater than a suitable minimum value. At this stage the σ( I ) estimates read from the integration program may not be very accurate but are good enough for this selection.
(v) First outlier rejection, using an algorithm similar to that described in §A5 of Evans (2006 ▶ ).
(vi) For data from MOSFLM , which outputs two estimates of each intensity, optimize the level of intensity I mid at which to switch (smoothly) between using the profile-fitted value I prf for weak data and summation integration I sum for strong data, where I ′ is the summation integration intensity before application of the Lorentz and polarization corrections. The exact form of this weighted mean is not critical, as ideally the two estimates are the same.
(vii) First optimization of σ( I ) estimates (see § 2.1).
(viii) Main scaling with strong reflections chosen on normalized intensities E 2 (typically choosing only observations with 0.8 < E 2 < 5). This gives a subset of data distributed over all resolution ranges.
(ix) Second outlier rejection as before.
(x) Final optimization of σ( I ) estimates as before.
(xi) Final outlier rejection as before.
(xii) Accumulate and print statistics.
(xiii) Output merged or unmerged reflection lists to files.
2.1. Standard error estimates
mechanismscontrolsconcepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗content_tags↗Initial estimates of the standard error of each intensity observation, σ( I h l ), are generally underestimated by all integration programs, so AIMLESS , like SCALA , updates the σ( I ) estimates in an attempt to make the average standard error match the average scatter of observations as a function of intensity only. The mismatch arises partly from the unknown ‘gain’ of the detector ( i.e. detector units per photon), which scales the Poisson-statistic error estimates (as well as correcting for the detector point-spread function), and partly from a variety of instrumental instabilities which cause an increase in error with increasing intensity. If the error estimate were correct in explaining the observed discrepancies within the data set, then the normalized deviations (where n h is the number of observations of reflection h and here I h l is the scale-corrected value, i.e. I h l / g h l ) should be distributed with a mean of 0.0 and a standard deviation of 1.0. 2 The standard error estimate is then adjusted to optimizing the values of the ‘correction’ factors SdFac, SdB and SdAdd to make variance(δ h l ) equal to 1.0 over all intensity ranges. The value of the term inside the square root is (arbitrarily) set to a minimum value of 0.1σ( I h l ) 2 to avoid possible negative values (a rare possibility). A separate set of correction factors is determined for each ‘run’ (usually, although an option is available to use the same value for all runs) and separately for fully recorded and partially recorded observations (if relevant). This is performed by minimizing summed over all intensity ranges i , currently with equal weights w i on each intensity bin and loose restraints on SdB parameters to avoid extreme values. Final values of variance(δ h l ) as a function of intensity for each run are plotted in the output of the program as an indication of the success (or otherwise) of this ‘correction’. mechanismscontent_tagsSdfac can be identified primarily with the uncertainty in the detector gain and Sdadd with general instability factors which lead to an error proportional to intensity (see, for example, Diederichs, 2010 ▶ ), but SbB has no obvious physical interpretation. However, this factor helps to flatten the plot of variance(δ h l ) against intensity and may be justified as an empirical ‘correction’ factor. It should be noted that this ‘correction’ to the error estimates is a fairly crude approximation, as it assumes there are no major residual systematic errors (such as those arising from radiation damage, for example) and that the correction can be parameterized purely on intensity.
3. Analysis of data quality
mechanismsconcepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗sensescontent_tagsOnce we have a set of scale factors to put all the intensities on a common scale and improved estimates of the error on each intensity, then we can analyse the data for internal consistency and for signal-to-noise ratio. We can do this as a function of image number (equivalent to time or crystal rotation) to detect radiation damage and weak or inconsistent regions of data and against resolution to decide on a high-resolution cutoff (see §4 ). Note that internal consistency measures are likely to underestimate the true error, since symmetry-related observations may suffer from the same systematic error.
mechanismsconcepts↗concepts↗concepts↗sensesconcept_edgesconcept_edgesconcept_edgescontent_tagsInternal consistency has traditionally been measured by R factors relating an individual observation I h l (after scaling) to the (weighted) average of all symmetry-related or replicate observations of the unique reflection h , 〈 I h 〉. The multiplicity-weighted R meas is an improvement over R merge , as it is relatively insensitive to data multiplicity (Diederichs & Karplus, 1997 ▶ ; Weiss & Hilgenfeld, 1997 ▶ ; Weiss, 2001 ▶ ), whereas R merge tends to increase with increasing multiplicity, even though the averaged intensities are improving. R p.i.m. provides an estimate of data quality after merging multiple observations.
An alternative way of measuring internal consistency for the analysis against resolution is to split the observations randomly into halves and then calculate the linear correlation coefficient between the halves. This is arguably the most reliable measure and is discussed further below.
3.1. Analysis against ‘batch’ or image number
concepts↗concepts↗concepts↗concepts↗unhoused_relationsAnalysis of various parameters as a function of batch or image number, a proxy for crystal rotation, time or radiation dose, is useful to determine whether the crystal has suffered unduly from radiation damage or whether there are any other parts of the data which should be discarded. AIMLESS plots similar graphs of R merge and of scale and relative B factors to those produced by SCALA (see, for example, Fig. 2 of Evans, 2011 ▶ ). Increasing negative values of the relative B factor are an indicator of deterioration with dose, although the B factor is also affected by factors other than radiation damage. AIMLESS adds two new plots against batch number: a rough estimate of the maximum resolution for each image and a cumulative completeness (for all data and anomalous pairs; see Fig. 1 ▶ ). unhoused_relationsThe ‘maximum resolution’ is estimated from the point at which 〈 I /σ( I )〉 falls below 1.0 for each batch: note that this 〈 I /σ( I )〉 is without averaging multiple measurements (which would not generally occur on the same image), so will be smaller than the 〈 I /σ〉 after averaging (§ 3.2.1). The plot shows the noisy values for each batch, as well as a smoothed plot typically averaged over a 5° range. This is only a rough estimate of resolution (see § 4), but serves to illustrate any trends. The cumulative completeness plot helps in deciding whether cutting back data from the end because of radiation damage would compromise the completeness. Such decisions are more complicated if the data have been measured from multiple ‘sweeps’ or multiple crystals.
3.2. Analysis against resolution
sensesIn order to estimate the useful ‘resolution’ of the data, i.e. the resolution at which the data may be truncated without losing significant information, AIMLESS plots various measures of signal to noise and internal consistency against resolution. There is at present no general consensus on the optimum criteria for interpretation of these plots and how to estimate the point at which adding additional high-resolution data is not adding anything useful: the true ‘resolution’ of a data set has often been a point of contention with referees of papers.
3.2.1. Signal-to-noise ratio
concepts↗concepts↗concepts↗sensesOne obvious way to judge data significance is from the average signal-to-noise ratio of the merged intensities as a function of resolution. This is calculated as [after ‘correcting’ the σ′( I h l ) estimates; § 2.1], i.e. for each reflection h the average intensity over symmetry mates 〈 I h 〉 is divided by its estimated error σ(〈 I h 〉) and this ratio is averaged in resolution bins [reported as Mn(I/sd) in the program output]. Commonly used resolution-cutoff levels are typically in the range 1–2: even in a resolution bin with 〈 I /σ〉 = 1 a proportion of intensities are significantly above the noise level [∼5–7% I > 3σ, ∼20–25% negative]. 〈 I /σ〉 is a good criterion for resolution cutoff, but it does suffer from uncertainties in the estimation of σ( I ), both from inadequacies in the integration program and the necessary σ( I ) ‘correction’ (see § 2.1; Ian Tickle, in a private communication, has pointed out that the major correction applies to large intensities and therefore would not affect the weak high-resolution data relevant to determination of resolution cutoff). It should be noted that 〈 I /σ〉 is not independent of measures of internal consistency because the corrections to σ′( I ) are adjusted to match the scatter of observations. Thus, σ′( I ) estimates are still likely to be underestimates of the true standard deviations.
3.2.2. Measures of internal consistency
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗sensesThe traditional R factors measuring internal consistency, R merge or better R meas (§ 3), are not suitable measures for setting a resolution cutoff (Evans, 2011 ▶ ; Karplus & Diederichs, 2012 ▶ ), despite their apparent popularity with referees. As pointed out by Karplus and Diederichs, these R factors cannot be compared with the R factors in refinement, since R merge and R meas both tend to infinity as the data become weaker, while R cryst (either R work or R free ) tends to a constant (see Appendix A 1 and Luzzati, 1953 ▶ ). This means that there is no sensible way of setting a maximum acceptable level. Note that this is a difference between an R factor on intensity, which can be measured as zero or negative, and an R factor on amplitude | F |, which cannot be negative and will be biased positive if, for example, the TRUNCATE procedure of French & Wilson (1978 ▶ ) has been applied (or indeed any protocol which sets negative values to positive). R merge or R meas are useful metrics for monitoring variation with batch (§ 3.1), and their value for the strongest intensities (top intensity bin or low-resolution bin) is a good indicator of overall data quality. A large R factor for the strong data may indicate a problem [a serious cause for concern if R merge (strong) > 0.10; ideally it should be <0.05], but they are not good indicators for weak data. concept_edgesconcept_edgesIn general, since R factors are dependent on the distribution of the data (see, for example, Murshudov, 2011 ▶ ) they are not good indicators of model quality or internal consistency, whereas the correlation coefficients are indicators of the degree of linear dependence between data sets and are less dependent on the distribution of the data, so they may be better indicators (see Appendix A 1 ).
claim_typemechanismsconcepts↗concepts↗concepts↗unhoused_relationsunhoused_relationssensesclaim_relationscontent_tags↗content_tags↗A better measure for assessing the ‘resolution’ of a data set is the correlation coefficient between random half data sets, CC 1/2 (labelled ‘CC_Imean’ in older versions of SCALA ; Evans, 2006 ▶ , 2011 ▶ ; Karplus & Diederichs, 2012 ▶ ). This statistic is plotted against resolution in both SCALA and AIMLESS (Fig. 2 ▶ ). A related statistic, Fourier shell correlation, has been used for assessing the resolution of electron-microscopy reconstructions since the early 1980s (see, for example, Rosenthal & Henderson, 2003 ▶ ; Henderson et al. , 2012 ▶ ). The advantage of a correlation coefficient is that it has a well defined range: +1.0 for a good correlation and 0 for no correlation. CC 1/2 is generally close to 1 at low resolution and falls sharply to near zero at higher resolution as the intensities become weaker (Fig. 2 ▶ ).
3.2.3. Anisotropy
concepts↗concepts↗concepts↗concepts↗Most data sets are anisotropic to some extent, which complicates analysis and decision making. unhoused_relationsThe anisotropy of the data is analysed in AIMLESS using both CC 1/2 and 〈 I /σ〉. The two or three orthogonal principal directions of anisotropy are constrained by the lattice symmetry: no anisotropy for cubic symmetry, two principal directions, the unique c ( c *) axis and the ab plane, for tetragonal, hexagonal and trigonal, three orthogonal axes for orthorhombic, the unique b ( b *) axis and two orthogonal axes in the ac plane for monoclinic and three general orthogonal axes for the triclinic system. The general directions for triclinic and in the monoclinic ac plane are determined from the eigenvectors of a fitted anisotropic scale factor. Anisotropy is then analysed in cones around the principal axes (default semi-angle 20°) and within the same angle of a principal plane, or alternatively as projections onto the two or three principal directions. In the former case, observations are weighted with a cosine weight declining from 1 along the principal direction to 0 at the edge of the region. Plots of CC 1/2 and 〈 I /σ〉 then allow assessment of resolution in different directions in the same way as for the overall resolution and with the same difficulties.
4. Tests of resolution cutoffs
research_questionmechanismslimitationsconcepts↗concepts↗concepts↗sensesHow can we decide where to apply a resolution cutoff? On the one hand, using high-resolution data which are so weak as to be insignificant will add nothing useful, may add unwanted noise to maps and may lead us to overconfidence in the quality of our model (although the relationship between data quality and model quality is not clear). On the other hand, we do not want to exclude useful data which might aid structure solution and improve the final model. Anisotropy in the data complicates this decision, as it is not clear whether it is better to include data based on the best direction, the worst, or something in between. concept_edgesAnisotropic cutoffs are likely to cause artefacts in map calculation. The problem of anisotropic data and how to deal with them in refinement and map calculation is an open question, and future work needs to address this problem, with the goal of developing clear protocols.
To examine these questions, a number of tests were carried out using data integrated beyond what would normally be considered acceptable using the example data sets listed in Table 1 ▶ .
4.1. Comparison with simulated data
concepts↗concepts↗concepts↗concepts↗concepts↗By comparing refinement against measured data with refinement against simulated data, we can judge the resolution point at which the measured data become no better than simulated data and compare this with the data-processing statistics. Figs. 2 ▶ ( a )–2 ▶ ( d ) show various analyses for example 1, for which simulated data were generated beyond 2.4 Å resolution in several ways with simulations based on the observed intensity distribution but not on the structure itself (see Appendix A 2 for details of the simulation): (i) the expected value of | F | at the resolution and anisotropic position of each reflection, i.e. all | F |s close together in reciprocal space are the same (denoted F expected ), (ii) a number of data sets with random intensities around the expected intensities with the same distribution as the measured intensities at the same resolution. These simulated intensities were converted to amplitudes (denoted F random ) with the CCP 4 program CTRUNCATE using the same procedure as used for the experimental data (French & Wilson, 1978 ▶ ).
The model was then refined with REFMAC (Murshudov et al. , 2011 ▶ ) to 1.83 Å resolution against the observed data ( F obs ) and also against various simulated data sets. As shown in Fig. 2 ▶ ( a ), R free values (and R work ; not shown) for the experimental data increase with increasing resolution as expected, while against the simulated data there is a sudden increase at 2.4 Å resolution where the refinement switches from the experimental data, but beyond about 1.95 Å resolution R free for F obs is no better than that for F expected , while refinement against random data sets (with different errors) gives R free values which converge towards the values from F obs and F expected at higher resolution at around R = 0.42, the expected value (see Appendix A 1 ). mechanismscontrolsclaim_relationscontent_tagscontent_tagscontent_tagscontent_tagsWe could conclude from this that there is no gain in including data beyond about 1.95 Å resolution, at which point CC 1/2 is 0.27 and 〈 I /σ〉 is 0.9: a more conventional cutoff point at 〈 I /σ〉 = 2.0 would be at 2.06 Å resolution. This data set is significantly anisotropic, with orthogonal principal axes along d1 = 0.97 h + 0.23 l , d2 = k , d3 = −0.75 h + 0.67 l , with the resolutions at which 〈 I /σ〉 = 2.0 along d1 , d2 and d3 being 2.23, 1.98 and 2.05 Å, respectively (Fig. 2 ▶ b ). Analysis of R factors in cones in the same way (§ 3.2.3 and Fig. 2 ▶ b ) shows a similar pattern to the overall values, with a convergence of R factors against F obs and F expected in the range 2.08–1.91 Å.
concepts↗The TRUNCATE procedure for inferring | F | from experimentally measured I produces a positive bias for weak intensities, which complicates the comparison with simulated data. To test the effect of this, refinement was also carried out against observed and simulated intensities instead of against amplitudes: R free values against simulated data were larger from 2.4 Å resolution as before, but had converged by ∼1.98 Å resolution (Fig. 2 ▶ c ). Following refinement against intensities, REFMAC calculates R factors in a way that mimics R factors on F (Murshudov et al. , 2011 ▶ ), but these R factors rise sharply with resolution (ultimately to infinity), rather than flattening out as do those on | F |.
concepts↗concepts↗Another estimate of the significance of the data is shown by the maximum-likelihood scale factor D . The best electron density after refinement is calculated using coefficients 2 mF o − DF c , where m is dependent on D . The values of D within resolution shells indicate how much these reflections contribute to electron density, with a value close to zero indicating that these reflections make little contribution. Fig. 2 ▶ ( d ) shows that (for example 1) D falls with resolution but is still greater than the values for simulated data even at the resolution edge.
method↗mechanismscontent_tagsSimilar tests were carried out for examples 4 and 5 (see Figs. 2 ▶ e and 2 ▶ f ), with similar conclusions that by the time CC 1/2 has fallen to around 0.2–0.4, or 〈 I /σ〉 to around 0.5–1.5, there is little information remaining, but that it would be hard to make a definite rule. With increasing resolution the R factors and D values for experimental data converge towards the values from simulated data, but the point at which they coalesce relative to the data-processing scores varies between different scores and different data sets.
4.2. Tests with automated model building
limitationscontrolsconcepts↗concepts↗concepts↗concepts↗concepts↗concepts↗There have been anecdotal reports that extending the resolution to include weak data may help automated model-building procedures. Unfortunately, this has been hard to prove: the examples tried here either worked at all resolutions or largely failed at all resolutions. Example 2 was used to test model building from a rather poor map with experimental phases to 2.5 Å resolution and model building with the Buccaneer / REFMAC pipeline (Cowtan, 2012 ▶ ) tested at resolutions of 2.3, 2.4, 2.5 and 2.6 Å. Models built at all of these resolutions had some correct and some incorrect parts, with the assigned sequence being largely wrong, but there was no consistent pattern over the different resolutions in the number of residues built or sequence assignment or in the correctness of the models. Another test was performed on example 3, a 2:1 complex, starting with a molecular-replacement model with the two large molecules (595 residues each) and building the smaller 91-residue molecule. In this case the smaller component was built more or less consistently at any resolution between 2.57 and 3.3 Å, although perhaps with fewer errors away from these extremes at between 2.7 and 3.1 Å. Similar results were obtained with example 4, omitting the last domain (207 residues): this could be rebuilt more or less successfully at resolutions between 1.8 and 2.8 Å, although maybe slightly more correctly at higher resolutions. However, extending the resolution at least seemed to do no harm. It might indicate that the Buccaneer / REFMAC pipeline for model building at least is more dependent on phase error rather than quality of structure-factor amplitudes (and hence resolution). In cases of difficulty, it may be worth trying model building with data to different resolutions.
concepts↗4.3. Electron density in OMIT maps
concepts↗concepts↗concepts↗The effect of resolution on the visual appearance of difference maps was tested on regions of the model which were not included in refinement. This is important in manual building and completion of structures. Part of a structure was omitted, the remaining coordinates were perturbed by a random shift of up to 0.3 Å to reduce model bias and the structure was refined in REFMAC with different resolution cutoffs. Fig. 3 ▶ shows two examples of maximum-likelihood difference maps at different resolutions for examples 4 and 1. There is not much difference between the maps, although there may be a slight improvement in sharpness in example 1 (Fig. 3 ▶ b ) on extending from 2.4 to 2.0 Å resolution, consistent with the refinement results in § 4.1 (Fig. 2 ▶ a ). Again, as for the automated model building (§ 4.2), extending the resolution seems at least to do no harm.
5. Discussion
sensesThe program AIMLESS performs essentially the same task as SCALA and gives similar results. However, it is significantly faster (about three times) and is a better framework for adding new scaling models and analyses. In due course, the three programs POINTLESS , AIMLESS and CTRUNCATE may be combined into one.
The extensive statistics produced by AIMLESS are mostly similar to those produced by SCALA , so the questions and decisions for the user are as discussed in Evans (2011 ▶ ): (i) What is the point group (Laue group)? (ii) What is the space group? (iii) Is there radiation damage and should the most damaged regions of data be excluded? (iv) What resolution cutoff should be applied (see below)? (v) Is there a detectable anomalous signal? (vi) Are the data twinned? (vii) Is this data set better than those previously collected?
concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗With regard to this last point, one traditional way of improving weak or incomplete data is to merge data from different crystals, provided that they are isomorphous. With the advent of cryocooling, this has fallen out of fashion except for the most desperate cases, but recent work from Hendrickson’s group (Liu et al. , 2012 ▶ , 2013 ▶ ) makes a good case for unhoused_relationsmerging data from many crystals to enhance the very weak anomalous signal from sulfur. The current tools for checking isomorphism between crystals are rather undeveloped, but this technology is improving (see Giordano et al. , 2012 ▶ ). With the fast data collection on modern synchrotrons, it is common to collect several or many more-or-less equivalent data sets and merging them may be better than just choosing the best. limitationsconcept_edgesconcept_edgesAt the end, model quality depends on data quality and merging many data sets usually improves the signal-to-noise ratio in the data; however, it is not clear how to merge data when there is severe non-isomorphism or radiation damage. Blindly merging data may do more harm than good. It is necessary to analyse the joint distribution of all data sets and to merge using this distribution. In an ideal world all data sets would be used without merging, thus ensuring that the extraction of information from the data would be optimal at all stages of structure analysis: the distribution of the data and the amount of information available at each stage of analysis would define what needs to be used.
object_of_studyphenomenonmechanismsconcepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗concepts↗unhoused_relationsunhoused_relationssensessensesconcept_edgescontent_tagsA major cause of user indecision and conflicts with journal referees is the resolution cutoff. We cannot set definite rules for this, as it depends on what the data are to be used for. It is therefore a mistake to prematurely apply a harsh cutoff at the data-reduction stage: data can always be excluded later. It is also unclear how best to handle anisotropy: do you choose the best direction or the worst? It is probably best to include data to the limit in the best direction. Tests carried out here to relate the resolution statistics to final model building and refinement do suggest that extending the data somewhat beyond the traditional limits such as 〈 I /σ〉 = 2 may improve structure determination, as do the ‘paired-refinement’ tests of Karplus & Diederichs (2012 ▶ ). At the very least, adding these weak data seems to do no harm for the purposes of either automatic or manual model building. concept_edgesThe main problem is that we have become accustomed to using the nominal resolution as an indicator of model quality, and it is not a good indicator, particularly as important biological and chemical conclusions from a structure often depend on local rather than global correctness. Nor indeed can any global score can indicate the correctness of any local structural feature. Clearly, well measured data to 1.5 Å resolution contain more information than a data set to 3.5 Å resolution and are therefore likely to lead to a more correct structure, but nominal resolution in itself just tells us how many reflections were used, rather than their quality. From our limited tests here, it seems that changing the resolution cutoff over a considerable range ( e.g. from 2.2 to 1.9 Å) makes only a small difference, so the exact cutoff point is not a question to agonize over, but it seems sensible to set a generous limit so as not to exclude data containing real (if weak) information. There is no reason to suppose that cutting back the resolution of the data will improve the model. These tests were performed with current programs and our current procedures at all stages could be improved to extract the maximum information from weak noisy data.
1 For the naming of the programs POINTLESS and AIMLESS , see Gibbons (1932 ▶ ).
2 Note that (as pointed out by a referee) this is only strictly true if all weights used in the calculation of 〈 I h 〉 are equal: this needs further investigation, although some preliminary tests using different weighting schemes (including equal unit weights) showed only small differences.
We would like to thank Stephen Graham and Janet Deane for test data sets and Andrew Leslie for reading the manuscript. PRE was supported by MRC grant U105178845 and GNM by MRC grant MC_UP_A025_1012.
Appendix A Statistical methods
A1. Behaviour of crystallographic R values
Crystallographic R factors are calculated using the formula where the summation is over the reflections used to calculate the R value (in the case of twinning a generalization of this formula is used; see, for example, Murshudov, 2011 ▶ ). Obviously, the behaviour of this statistic depends on the statistical properties of the structure factors. Therefore, it can be expected that properties of the model, crystal and observed data such as (i) noisy data, (ii) twinning, (iii) modulation in crystals and (iv) model errors will affect the behaviour of the R value. Luzzati (1953 ▶ ) analysed the effect of model errors on R values in the absence of any other peculiarity and came to the conclusion that R values calculated for structure factors calculated from random atoms would be around 0.58 for acentric reflections. Murshudov (2011 ▶ ) carried out a similar analysis for cases of hemihedral twinning and demonstrated that R values for cases of hemihedral twinning are systematically lower than those for single crystals. Here, we analyse the effect of the procedure used to estimate the amplitude | F | from the measured intensity I when data are very noisy. Since weak intensities may be measured as negative, while the true amplitude | F | cannot be negative, | F | is generally estimated using the TRUNCATE procedure (French & Wilson, 1978 ▶ ). Under the assumption that true structure factors come from a crystal filled with atoms randomly distributed over the unit cell and the noise in the experimental intensities has a normal distribution, the TRUNCATE procedure estimates the amplitudes of structure factors using a Bayesian estimation with the formula where P ( J ; I o , crystal) is the conditional distribution of ideal intensities of structure factors ( J ) when observed intensities ( I o ) are known and it is known that the data came from a crystal. Noting that | F | = J 1/2 , and using the explicit form of P ( J ; I o , crystal), the conditional distribution for | F | can be written where ∊Σ = E (| F | 2 ), the expected value of | F | 2 , is estimated using the data in the resolution bins, I o is the experimental observed intensity and σ is its standard deviation. Thus, the expected values of the amplitudes of structure factors are estimated using the formula
Although this formulation has served the community well over the years, it has certain problems. These problems include the following. (i) It is assumed that Σ is a smooth function of resolution, which breaks down in the presence of pseudo-translation, DNA/RNA helices etc. (ii) It is assumed that the data are from single crystals. In cases of twinning the formulation may not work, although if the twinning fraction is known and there is no noncrystallographic rotation parallel to the twin operators then it is straightforward to account for twinning. However, in general not all properties of the crystal/data are known or possible to model. (iii) It is assumed that the observed data have a normal distribution. This assumption may break down, especially for weak reflections where profiles of neighbourhood spots are used for integration.
When the data are very noisy ( i.e. the standard deviation of the observation becomes very large) this procedure produces the expected value of Wilson’s distribution (Wilson, 1949 ▶ ),
Thus, in the limiting case if all estimation of parameters proceeds smoothly and there are no crystal-growth peculiarities, then the TRUNCATE procedure would give the expected value of the Wilson distribution. It is interesting to analyse the crystallographic R values for these cases. Let us assume that the R value is calculated in the resolution bins where Σ was estimated. Then, in the limiting case of noisy data the R value would have the following form (here, we assume that reciprocal-space points are sufficiently dense and the summation can be replaced by the integration),
Now if we use the fact that the distribution of | F c | is the same as the distribution of | F |, that is the Wilson distribution, and denote μ = E (| F |) = E (| F c |), then after some manipulation of integrals we can derive where erfc is the complementary error function,
Thus, we see that if structure factors are replaced by the expected value of the Wilson distribution then it can be expected that the calculated R values will be around 0.42. This is exactly what happens with observed values from the TRUNCATE procedure when errors in the experimental intensities become very large. It should be noted that this behaviour of R values is a property of data from single crystals with no other peculiarities.
Note that for perfect hemihedral twinning
If the perfect twinning assumption is used in the TRUNCATE procedure then for very noisy data we can obtain
A2. Generation of random intensities from Gaussian χ 2 distribution
Under the assumption that the ‘true’ structure factors came from the population with a Wilson distribution and experimental intensities have a normal distribution with mean value equal to the ‘true’ intensities and with experimental uncertainties equal to σ, the distribution of observed intensities can be written
To use this formula for random intensity generation it is necessary to estimate the unknown parameters (Σ). If we want to generate ‘data’ beyond the resolution of the observed data we need to parameterize Σ as a smooth function of resolution. We use the parameterization where k is the overall scale, B is the overall isotropic temperature factor, f ( s ) is the empirical intensity curve designed by Popov & Bourenkov (2003 ▶ ), s is a reciprocal-space vector, | s | is the length of the reciprocal-space vector and U is the overall anisotropic U value with properties where tr is the trace, R sym is the rotation part of a symmetry operator (here, we use the orthogonal version of the symmetry operators) and the superscript T denotes the transpose of the operator. The second condition in (22 ) is for all symmetry operators of the crystal. Since (21 ) is a continuous function of Σ, it can be used for any resolution including resolutions beyond the observed resolution.
The procedure for estimation of parameters and generation of random intensities is as follows. (i) Using E ( I o ) = ∊Σ, var( I o ) = (∊Σ) 2 , build a Gaussian approximation for the distribution of the observed intensities and estimate parameters of Σ as defined by (21 ).
(ii) Using the distribution (20 ), build the likelihood function and improve the estimation of the parameters of Σ using maximum-likelihood estimation.
(iii) Generate expected values of the amplitudes of structure factors for reflections within defined resolution using the formula E (| F |) = (π∊Σ) 1/2 /2.
(iv) Generate random intensity from the population defined by the distribution (20 ). For this, a two-stage procedure is used: ( a ) generate a random number from the exponential distribution with mean ∊Σ and denote it I exp and ( b ) generate a random number from the normal distribution with mean I exp and standard deviation σ. Σ is extrapolated to the given resolution and σ is taken from the observed data or from the defined signal-to-noise ratio.
(v) Generate random numbers from the population with Wilson distribution with the parameter ∊Σ.
Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; 13 of 13 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.
| facet | tag | field | conf. | whose words |
|---|---|---|---|---|
| object_of_study | WHERE TO CUT A DATA SET OFF - the resolution limit every crystal structure declares, tested against refinement, automated model building and simulated data, and then explicitly refused as a rule | object_of_study | the paper's words | |
| research_question | at what point weak high-resolution data stop carrying information, and whether any of the data-processing statistics marks that point | research_question | the paper's words | |
| contribution | the AIMLESS scaling program, and evidence that CC1/2 is a better resolution criterion than the traditional <I/sigma> - together with the finding that extending past the conventional limit 'does no harm' | contribution | the paper's words | |
| claim_type | method | claim_type | the paper's words | |
| phenomenon | a threshold whose value depends on the use the data is put to, and whose main friction is with referees | phenomenon | the paper's words | |
| method | experiment also in: experiment | method | the paper's words | |
| mechanisms | A THRESHOLD PAPER THAT REFUSES TO SET THE THRESHOLD, AND SAYS WHY: `A major cause of user indecision and conflicts with journal referees is the resolution cutoff. We cannot set definite rules for this, as it depends on what the data are to be used for. It is therefore a mistake to prematurely apply a harsh cutoff at the data-reduction stage: data can always be excluded later.` Three things at once - the friction is social, the value is purpose-dependent, and the advice is about WHEN to apply a criterion rather than where to put it | mechanisms[0] | 0.95 | the tagger's words |
| mechanisms | AND THE CONVENTION AND THE MEASUREMENT DISAGREE, IN ANGSTROMS: `there is NO GAIN in including data beyond about 1.95 A resolution, at which point CC1/2 IS 0.27 and <I/sigma> IS 0.9: A more conventional cutoff point at <I/sigma> = 2.0 WOULD BE AT 2.06 A RESOLUTION.` The traditional criterion throws away a shell of data the tests say is still useful - and the gap is 0.11 A on one data set | mechanisms[1] | 0.95 | the tagger's words |
| mechanisms | AND THE THRESHOLD IS NOT A SCALAR: the same data set gives `the resolutions at which <I/sigma> = 2.0 along d1, d2 and d3 being 2.23, 1.98 AND 2.05 A` - three answers depending on direction. And the paper declines to choose: `it is not clear whether it is better to include data based on the best direction, the worst, or something in between. Anisotropic cutoffs are likely to cause artefacts in map calculation` | mechanisms[2] | 0.95 | the tagger's words |
| mechanisms | AND THE EMPIRICAL RANGE IS GIVEN AND THEN REFUSED AS A RULE: `by the time CC1/2 has fallen to around 0.2-0.4, or <I/sigma> TO AROUND 0.5-1.5, there is little information remaining, but that it would be hard to make A DEFINITE RULE. ... the point at which they coalesce relative to the data-processing scores varies between different scores and different data sets.` A range published as a range, with the reason it cannot become a number | mechanisms[3] | 0.95 | the tagger's words |
| mechanisms | AND A QUALITY STATISTIC THAT GETS WORSE AS THE DATA GETS BETTER: `The multiplicity-weighted R_meas is an improvement over R_merge, as it is RELATIVELY INSENSITIVE TO DATA MULTIPLICITY, WHEREAS R_MERGE tends to increase with increasing multiplicity, even though the averaged intensities are improving.` The measure and the thing measured move in opposite directions, and the field used the bad one for decades | mechanisms[4] | 0.95 | the tagger's words |
| mechanisms | AND THE CONSISTENCY MEASURE CANNOT SEE THE ERROR THAT MATTERS MOST: `Note that internal consistency measures are likely to underestimate the true error, since symmetry-related observations may suffer from the same systematic error.` Agreement between replicates is evidence of precision and silent about accuracy - the measurement's blind spot is exactly the error class it is used to rule out | mechanisms[5] | 0.95 | the tagger's words |
| mechanisms | AND THE ERROR BARS ARE WRONG EVERYWHERE, BY DEFAULT: `Initial estimates of the standard error of each intensity observation are generally underestimated by all integration programs, so AIMLESS ... UPDATES THE sigma(I) ESTIMATES in an attempt to make the average standard error match the average scatter.` Every program in the field ships an underestimate, and the fix is a fitted correction | mechanisms[6] | 0.95 | the tagger's words |
| mechanisms | AND ONE OF THE CORRECTION TERMS HAS NO MEANING: `SdB has no obvious physical interpretation. However, this factor helps to flatten the plot of variance against intensity and may be justified as an empirical `CORRECTION` FACTOR. It should be noted that this `correction` to the error estimates is a fairly crude approximation, as it assumes there are no major residual systematic errors.` A parameter kept because it works, labelled as such, with the assumption it rests on printed beside it | mechanisms[7] | 0.95 | the tagger's words |
| mechanisms | AND THE PREFERRED STATISTIC WAS BORROWED FROM ANOTHER FIELD THIRTY YEARS EARLIER: `A better measure for assessing the `resolution` of a data set is the correlation coefficient between random half data sets, CC1/2 ... A related statistic, Fourier shell correlation, has been used for assessing the resolution of electron-microscopy reconstructions since the early 1980s.` The advantage given is definitional: `a correlation coefficient ... has a well defined range` | mechanisms[8] | 0.95 | the tagger's words |
| limitations | THE MODEL-BUILDING TEST COULD NOT SEPARATE ANYTHING: `There have been anecdotal reports that extending the resolution to include weak data may help automated model-building procedures. Unfortunately, this has been hard to prove: the examples tried here either worked at all resolutions or largely failed at all resolutions ... there was no consistent pattern over the different resolutions.` A negative result reported as a negative result, on the question the whole paper is about | limitations[0] | 0.95 | the tagger's words |
| limitations | AND MERGING MORE DATA CAN MAKE IT WORSE: `merging many data sets usually improves the signal-to-noise ratio in the data; however, it is not clear how to merge data when there is severe non-isomorphism or radiation damage.` More observations is not always more information, and the paper says where the boundary is unknown rather than guessing at it | limitations[1] | 0.95 | the tagger's words |
| limitations | and the relationship the cutoff is supposed to serve is itself unclear: `using high-resolution data which are so weak as to be insignificant will add nothing useful, may add unwanted noise to maps and may lead us to overconfidence in the quality of our model (although the relationship between data quality and model quality is not clear).` The parenthesis undercuts the sentence it is inside, and the authors leave it there | limitations[2] | 0.95 | the tagger's words |
| controls | refining against SIMULATED data generated from the observed intensity distribution but NOT from the structure itself, at the same resolutions as the measured data eliminates: mistaking model bias or noise-fitting for real information in the weak high-resolution shells - the point where measured and simulated converge is the information limit · outcome: held | controls[0] | 0.95 | the tagger's words |
| controls | omitting part of the model, perturbing the remaining coordinates by up to 0.3 A, and comparing difference maps at different cutoffs eliminates: model bias in the visual assessment of whether extra resolution helps · outcome: partial | controls[1] | 0.95 | the tagger's words |
| controls | updating the sigma(I) estimates until the normalised deviations have variance 1.0 across all intensity ranges eliminates: error bars that every integration program underestimates by default · outcome: partial | controls[2] | 0.95 | the tagger's words |
| concepts | R meas also in: R meas | concepts[0] | 0.95 | the paper's words |
| concepts | R merge also in: R merge | concepts[1] | 0.95 | the paper's words |
| concepts | multiplicity also in: multiplicity | concepts[2] | 0.95 | the paper's words |
| concepts | resolution cutoff also in: resolution cutoff | concepts[3] | 0.95 | the paper's words |
| concepts | CC 1/2 also in: CC 1/2 | concepts[4] | 0.95 | the paper's words |
| concepts | data reduction also in: data reduction | concepts[5] | 0.95 | the paper's words |
| concepts | data-scaling program also in: data-scaling program | concepts[6] | 0.95 | the paper's words |
| concepts | classic scaling method also in: classic scaling method | concepts[7] | 0.95 | the paper's words |
| concepts | POINTLESS also in: POINTLESS | concepts[8] | 0.95 | the paper's words |
| concepts | symmetry-related reflections also in: symmetry-related reflections | concepts[9] | 0.95 | the paper's words |
| concepts | data quality also in: data quality | concepts[10] | 0.95 | the paper's words |
| concepts | weak high-resolution data also in: weak high-resolution data | concepts[11] | 0.95 | the paper's words |
| concepts | AIMLESS also in: AIMLESS | concepts[12] | 0.95 | the paper's words |
| concepts | SCALA also in: SCALA | concepts[13] | 0.95 | the paper's words |
| concepts | resolution also in: resolution | concepts[14] | 0.95 | the paper's words |
| concepts | data-processing statistics also in: data-processing statistics | concepts[15] | 0.95 | the paper's words |
| concepts | automated model-building also in: automated model-building | concepts[16] | 0.95 | the paper's words |
| concepts | common scale also in: common scale | concepts[17] | 0.95 | the paper's words |
| concepts | X-ray diffraction images also in: X-ray diffraction images | concepts[18] | 0.95 | the paper's words |
| concepts | reflection intensities also in: reflection intensities | concepts[19] | 0.95 | the paper's words |
| concepts | point group also in: point group | concepts[20] | 0.95 | the paper's words |
| concepts | space group also in: space group | concepts[21] | 0.95 | the paper's words |
| concepts | consistent indexing also in: consistent indexing | concepts[22] | 0.95 | the paper's words |
| concepts | structure amplitudes | F | also in: structure amplitudes | F | | concepts[23] | 0.95 | the paper's words |
| concepts | internal consistency also in: internal consistency | concepts[24] | 0.95 | the paper's words |
| concepts | experimental conditions also in: experimental conditions | concepts[25] | 0.95 | the paper's words |
| concepts | incident-beam intensity also in: incident-beam intensity | concepts[26] | 0.95 | the paper's words |
| concepts | volume of the crystal illuminated also in: volume of the crystal illuminated | concepts[27] | 0.95 | the paper's words |
| concepts | absorption in the primary or secondary beam | concepts[28] | 0.95 | the paper's words |
| concepts | average radiation damage also in: average radiation damage | concepts[29] | 0.95 | the paper's words |
| concepts | scale factors also in: scale factors | concepts[30] | 0.95 | the paper's words |
| concepts | inverse scale also in: inverse scale | concepts[31] | 0.95 | the paper's words |
| concepts | scale model also in: scale model | concepts[32] | 0.95 | the paper's words |
| concepts | crystal rotation angle also in: crystal rotation angle | concepts[33] | 0.95 | the paper's words |
| concepts | radiation dose also in: radiation dose | concepts[34] | 0.95 | the paper's words |
| concepts | relative B factors also in: relative B factors | concepts[35] | 0.95 | the paper's words |
| concepts | radiation-damage correction also in: radiation-damage correction | concepts[36] | 0.95 | the paper's words |
| concepts | Gaussian weights also in: Gaussian weights | concepts[37] | 0.95 | the paper's words |
| concepts | secondary beam correction also in: secondary beam correction | concepts[38] | 0.95 | the paper's words |
| concepts | spherical harmonic terms also in: spherical harmonic terms | concepts[39] | 0.95 | the paper's words |
| concepts | Parameter restraint terms also in: Parameter restraint terms | concepts[40] | 0.95 | the paper's words |
| concepts | sphericity restraint also in: sphericity restraint | concepts[41] | 0.95 | the paper's words |
| concepts | standard error estimates also in: standard error estimates | concepts[42] | 0.95 | the paper's words |
| concepts | outlier rejection also in: outlier rejection | concepts[43] | 0.95 | the paper's words |
| concepts | strong reflections also in: strong reflections | concepts[44] | 0.95 | the paper's words |
| concepts | normalized intensities E 2 also in: normalized intensities E 2 | concepts[45] | 0.95 | the paper's words |
| concepts | merged reflection lists also in: merged reflection lists | concepts[46] | 0.95 | the paper's words |
| concepts | unmerged reflection lists also in: unmerged reflection lists | concepts[47] | 0.95 | the paper's words |
| concepts | detector gain also in: detector gain | concepts[48] | 0.95 | the paper's words |
| concepts | Poisson-statistic error estimates also in: Poisson-statistic error estimates | concepts[49] | 0.95 | the paper's words |
| concepts | instrumental instabilities also in: instrumental instabilities | concepts[50] | 0.95 | the paper's words |
| concepts | normalized deviations also in: normalized deviations | concepts[51] | 0.95 | the paper's words |
| concepts | SdFac also in: SdFac | concepts[52] | 0.95 | the paper's words |
| concepts | SdB also in: SdB | concepts[53] | 0.95 | the paper's words |
| concepts | SdAdd also in: SdAdd | concepts[54] | 0.95 | the paper's words |
| concepts | intensity bin also in: intensity bin | concepts[55] | 0.95 | the paper's words |
| concepts | signal-to-noise ratio also in: signal-to-noise ratio | concepts[56] | 0.95 | the paper's words |
| concepts | I /σ also in: I /σ | concepts[57] | 0.95 | the paper's words |
| concepts | image number also in: image number | concepts[58] | 0.95 | the paper's words |
| concepts | radiation damage also in: radiation damage | concepts[59] | 0.95 | the paper's words |
| concepts | weak or inconsistent regions of data also in: weak or inconsistent regions of data | concepts[60] | 0.95 | the paper's words |
| concepts | internal consistency measures also in: internal consistency measures | concepts[61] | 0.95 | the paper's words |
| concepts | R p.i.m. also in: R p.i.m. | concepts[62] | 0.95 | the paper's words |
| concepts | linear correlation coefficient also in: linear correlation coefficient | concepts[63] | 0.95 | the paper's words |
| concepts | batch number also in: batch number | concepts[64] | 0.95 | the paper's words |
| concepts | maximum resolution also in: maximum resolution | concepts[65] | 0.95 | the paper's words |
| concepts | cumulative completeness also in: cumulative completeness | concepts[66] | 0.95 | the paper's words |
| concepts | resolution bins also in: resolution bins | concepts[67] | 0.95 | the paper's words |
| concepts | noise level also in: noise level | concepts[68] | 0.95 | the paper's words |
| concepts | resolution-cutoff levels also in: resolution-cutoff levels | concepts[69] | 0.95 | the paper's words |
| concepts | R factors also in: R factors | concepts[70] | 0.95 | the paper's words |
| concepts | R cryst also in: R cryst | concepts[71] | 0.95 | the paper's words |
| concepts | R work also in: R work | concepts[72] | 0.95 | the paper's words |
| concepts | R free also in: R free | concepts[73] | 0.95 | the paper's words |
| concepts | correlation coefficients also in: correlation coefficients | concepts[74] | 0.95 | the paper's words |
| concepts | correlation coefficient also in: correlation coefficient | concepts[75] | 0.95 | the paper's words |
| concepts | distribution of the data also in: distribution of the data | concepts[76] | 0.95 | the paper's words |
| concepts | Fourier shell correlation also in: Fourier shell correlation | concepts[77] | 0.95 | the paper's words |
| concepts | anisotropy also in: anisotropy | concepts[78] | 0.95 | the paper's words |
| concepts | principal directions of anisotropy also in: principal directions of anisotropy | concepts[79] | 0.95 | the paper's words |
| concepts | fitted anisotropic scale factor also in: fitted anisotropic scale factor | concepts[80] | 0.95 | the paper's words |
| concepts | cones around the principal axes also in: cones around the principal axes | concepts[81] | 0.95 | the paper's words |
| concepts | weak data also in: weak data | concepts[82] | 0.95 | the paper's words |
| concepts | structure determination also in: structure determination | concepts[83] | 0.95 | the paper's words |
| concepts | unwanted noise also in: unwanted noise | concepts[84] | 0.95 | the paper's words |
| concepts | Anisotropic cutoffs also in: Anisotropic cutoffs | concepts[85] | 0.95 | the paper's words |
| concepts | artefacts in map calculation also in: artefacts in map calculation | concepts[86] | 0.95 | the paper's words |
| concepts | simulated data also in: simulated data | concepts[87] | 0.95 | the paper's words |
| concepts | F expected also in: F expected | concepts[88] | 0.95 | the paper's words |
| concepts | F random also in: F random | concepts[89] | 0.95 | the paper's words |
| concepts | CTRUNCATE also in: CTRUNCATE | concepts[90] | 0.95 | the paper's words |
| concepts | REFMAC also in: REFMAC | concepts[91] | 0.95 | the paper's words |
| concepts | positive bias also in: positive bias | concepts[92] | 0.95 | the paper's words |
| concepts | maximum-likelihood scale factor D also in: maximum-likelihood scale factor D | concepts[93] | 0.95 | the paper's words |
| concepts | electron density also in: electron density | concepts[94] | 0.95 | the paper's words |
| concepts | automated model-building procedures also in: automated model-building procedures | concepts[95] | 0.95 | the paper's words |
| concepts | Buccaneer / REFMAC pipeline also in: Buccaneer / REFMAC pipeline | concepts[96] | 0.95 | the paper's words |
| concepts | experimental phases also in: experimental phases | concepts[97] | 0.95 | the paper's words |
| concepts | molecular-replacement model also in: molecular-replacement model | concepts[98] | 0.95 | the paper's words |
| concepts | phase error also in: phase error | concepts[99] | 0.95 | the paper's words |
| concepts | structure-factor amplitudes also in: structure-factor amplitudes | concepts[100] | 0.95 | the paper's words |
| concepts | OMIT maps also in: OMIT maps | concepts[101] | 0.95 | the paper's words |
| concepts | difference maps also in: difference maps | concepts[102] | 0.95 | the paper's words |
| concepts | model bias also in: model bias | concepts[103] | 0.95 | the paper's words |
| concepts | maximum-likelihood difference maps also in: maximum-likelihood difference maps | concepts[104] | 0.95 | the paper's words |
| concepts | cryocooling also in: cryocooling | concepts[105] | 0.95 | the paper's words |
| concepts | isomorphism also in: isomorphism | concepts[106] | 0.95 | the paper's words |
| concepts | very weak anomalous signal from sulfur | concepts[107] | 0.95 | the paper's words |
| concepts | fast data collection also in: fast data collection | concepts[108] | 0.95 | the paper's words |
| concepts | non-isomorphism also in: non-isomorphism | concepts[109] | 0.95 | the paper's words |
| concepts | joint distribution of all data sets also in: joint distribution of all data sets | concepts[110] | 0.95 | the paper's words |
| concepts | merging many data sets also in: merging many data sets | concepts[111] | 0.95 | the paper's words |
| concepts | nominal resolution also in: nominal resolution | concepts[112] | 0.95 | the paper's words |
| concepts | what the data are to be used for also in: what the data are to be used for | concepts[113] | 0.95 | the paper's words |
| concepts | model quality also in: model quality | concepts[114] | 0.95 | the paper's words |
| concepts | local structural feature also in: local structural feature | concepts[115] | 0.95 | the paper's words |
| concepts | paired-refinement also in: paired-refinement | concepts[116] | 0.95 | the paper's words |
| concepts | generous limit also in: generous limit | concepts[117] | 0.95 | the paper's words |
| concepts | weak noisy data also in: weak noisy data | concepts[118] | 0.95 | the paper's words |
| concepts | merging data from many crystals also in: merging data from many crystals | concepts[119] | 0.95 | the paper's words |
| unhoused_relations | AIMLESS → SCALA AIMLESS replaces the earlier SCALA program; replaces is absent from the eight governed types, and the relation clause lacks a differentiation cue for contrasted-with. | unhoused_relations[0] | 0.95 | the tagger's words |
| unhoused_relations | relative B factors → radiation-damage correction The relative B-factor term is described as largely an average radiation-damage correction; the intervening hedge prevents the gate's strict is-a cue, so the relation is filed rather than forced. | unhoused_relations[1] | 0.95 | the tagger's words |
| unhoused_relations | maximum resolution → I /σ Per-batch maximum resolution is estimated from where I/sigma falls below 1.0; estimated-from is absent from the eight governed types and is not forced through the operationalisation cue gate. | unhoused_relations[2] | 0.95 | the tagger's words |
| unhoused_relations | CC 1/2 → correlation coefficient CC 1/2 is described as the correlation coefficient between random half data sets; the definite-article class statement is not admitted by the strict is-a cue, so it is filed rather than weakened. | unhoused_relations[3] | 0.95 | the tagger's words |
| unhoused_relations | POINTLESS → point group POINTLESS determines the likely point group; determines is absent from the eight governed types. | unhoused_relations[4] | 0.95 | the tagger's words |
| unhoused_relations | CTRUNCATE → structure amplitudes | F | CTRUNCATE calculates structure amplitudes from intensities; calculates/derives is absent from the eight governed types. | unhoused_relations[5] | 0.95 | the tagger's words |
| unhoused_relations | image number → radiation dose Image or batch number is a proxy for radiation dose; proxy-for is absent and is not a measurement definition strong enough for operationalised-as. | unhoused_relations[6] | 0.95 | the tagger's words |
| unhoused_relations | anisotropy → CC 1/2 AIMLESS analyses anisotropy using CC 1/2; analysed-using is absent from the eight governed types. | unhoused_relations[7] | 0.95 | the tagger's words |
| unhoused_relations | Fourier shell correlation → CC 1/2 Fourier shell correlation is presented as a related statistic to CC 1/2; related-to is absent from the eight governed types. | unhoused_relations[8] | 0.95 | the tagger's words |
| unhoused_relations | resolution cutoff → what the data are to be used for The appropriate resolution cutoff depends on what the data will be used for; use-dependent-choice is absent from the eight governed types. | unhoused_relations[9] | 0.95 | the tagger's words |
| unhoused_relations | very weak anomalous signal from sulfur → merging data from many crystals Merging many crystals is used to enhance the weak sulfur anomalous signal; enhances is absent and the paper does not support a clean intervention-level causes edge here. | unhoused_relations[10] | 0.95 | the tagger's words |
| unhoused_relations | nominal resolution → local structural feature No global score, including nominal resolution, can certify the correctness of a local structural feature; indicator-of/certifies is absent from the eight governed types. | unhoused_relations[11] | 0.9 | the tagger's words |
| senses | `resolution` (of a data set) - `a difficult and possibly contentious question (particularly with referees of papers)`, with no definite rule because it `depends on what the data are to be used for` | senses[0] | 0.95 | the tagger's words |
| senses | `CC1/2` - the correlation coefficient between two random half data sets; preferred because a correlation coefficient `has a well defined range`, and a descendant of Fourier shell correlation from 1980s electron microscopy | senses[1] | 0.95 | the tagger's words |
| senses | `R_merge` vs `R_meas` vs `R_p.i.m.` - three R factors for one property, the first of which RISES as the data improves | senses[2] | 0.95 | the tagger's words |
| senses | `internal consistency` - agreement among symmetry-related observations, which `LIKELY UNDERESTIMATES THE TRUE ERROR` because those observations share their systematic errors | senses[3] | 0.95 | the tagger's words |
| senses | AIMLESS scaling - a faster replacement for SCALA that preserves a similar scale model while making new models and analyses easier to add | senses[4] | 0.95 | the tagger's words |
| senses | Resolution as usable information - a use-dependent truncation decision rather than a fixed property determined by one universal threshold | senses[5] | 0.95 | the tagger's words |
| senses | Signal-to-noise cutoff - average merged intensity divided by estimated error, conventionally cut around one to two but sensitive to error calibration | senses[6] | 0.95 | the tagger's words |
| senses | R-factor mismatch - merging R factors diverge as observations weaken whereas refinement R factors tend toward a constant | senses[7] | 0.95 | the tagger's words |
| senses | Directional resolution - anisotropy makes one global cutoff inadequate and directional cutoffs potentially artefactual | senses[8] | 0.95 | the tagger's words |
| senses | Weak-data recommendation - retain data somewhat beyond traditional limits because trials show small possible gains and little observed harm | senses[9] | 0.95 | the tagger's words |
| senses | Nominal resolution boundary - the number of included reflections is not a global certificate of model quality or local structural correctness | senses[10] | 0.95 | the tagger's words |
| claim_relations | extends → Karplus & Diederichs (2012), who proposed CC1/2 as a resolution criterion - and, further back, Fourier shell correlation, used for electron-microscopy resolution since the early 1980s | claim_relations[0] | 0.95 | the tagger's words |
| claim_relations | critiques → the conventional <I/sigma> = 2.0 resolution cutoff, which on the worked example sits at 2.06 A where the tests put the information limit at 1.95 A | claim_relations[1] | 0.95 | the tagger's words |
| concept_edges | R meas —distinct-from→ R merge the relation itself is separately evidenced · strength of the relation: 0.95 · note: 'The multiplicity-weighted R_MEAS is an improvement over R_MERGE, AS IT IS RELATIVELY INSENSITIVE TO DATA MULTIPLICITY, WHEREAS R_MERGE tends to increase with increasing multiplicity, even though the averaged intensities are improving.' Two names for one property, and only one of them tracks it · the relation stated: R_meas is insensitive to multiplicity and R_merge is not - which is why the older statistic degrades as the data improves | concept_edges[0] | 0.95 | the tagger's words |
| concept_edges | multiplicity —causes→ R merge direction on the measure: increases · the relation itself is separately evidenced · strength of the relation: 0.95 · note: 'R_MERGE tends to increase with increasing multiplicity, even though the averaged intensities are improving.' The clearest case in this corpus of a quality statistic that moves opposite to quality - and the field reported it for decades before R_meas. The 'even though' is the paper doing the work a cue list cannot: it marks the causal claim AND its perversity in three words · the relation stated: increasing multiplicity INCREASES R_merge, while simultaneously improving the averaged intensities it is supposed to report on also in: multiplicity · R merge | concept_edges[1] | 0.95 | the tagger's words |
| concept_edges | AIMLESS —is-a→ classic scaling method the relation itself is separately evidenced · strength of the relation: 1 · note: The paper directly states that AIMLESS is a new implementation of a classic scaling method. also in: AIMLESS · classic scaling method | concept_edges[2] | 0.95 | the tagger's words |
| concept_edges | point group —part-of→ data reduction the relation itself is separately evidenced · strength of the relation: 1 · note: Determination of the point group is explicitly included among data-reduction processes. also in: point group · data reduction | concept_edges[3] | 0.95 | the tagger's words |
| concept_edges | space group —part-of→ data reduction the relation itself is separately evidenced · strength of the relation: 1 · note: Determination of the possible space group is explicitly included among data-reduction processes. also in: space group · data reduction | concept_edges[4] | 0.95 | the tagger's words |
| concept_edges | internal consistency —operationalised-as→ R factors the relation itself is separately evidenced · strength of the relation: 1 · note: The paper explicitly says internal consistency has traditionally been measured by R factors. also in: internal consistency · R factors | concept_edges[5] | 0.95 | the tagger's words |
| concept_edges | R factors —contrasted-with→ correlation coefficients the relation itself is separately evidenced · strength of the relation: 0.9 · note: The paper contrasts distribution-dependent R factors with less distribution-dependent correlations, while hedging that correlations may be better. also in: R factors · correlation coefficients | concept_edges[6] | 0.95 | the tagger's words |
| concept_edges | Anisotropic cutoffs —causes→ artefacts in map calculation status: methodological risk stated as likely, with the issue left open · direction on the measure: increases · the relation itself is separately evidenced · strength of the relation: 0.7 · status (stated by the tagger, no quotation): methodological risk stated as likely, with the issue left open · note: The weight reflects the paper's 'likely to cause' hedge.
Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but anisotropic cutoffs 'are likely to cause' artefacts; hedged. also in: Anisotropic cutoffs · artefacts in map calculation | concept_edges[7] | 0.95 | the tagger's words |
| concept_edges | distribution of the data —causes→ R factors status: mathematical dependence of a statistic on its data distribution · the relation itself is separately evidenced · strength of the relation: 1 · status (stated by the tagger, no quotation): mathematical dependence of a statistic on its data distribution · note: The paper states that R factors are dependent on the distribution of the data. also in: distribution of the data · R factors | concept_edges[8] | 0.95 | the tagger's words |
| concept_edges | data quality —causes→ model quality status: general dependency claim, not an isolated intervention · direction on the measure: increases · the relation itself is separately evidenced · strength of the relation: 0.85 · status (stated by the tagger, no quotation): general dependency claim, not an isolated intervention · note: The authors state that model quality depends on data quality. also in: data quality · model quality | concept_edges[9] | 0.95 | the tagger's words |
| concept_edges | merging many data sets —causes→ signal-to-noise ratio status: general empirical-methodological claim qualified by 'usually' · direction on the measure: increases · the relation itself is separately evidenced · strength of the relation: 0.9 · status (stated by the tagger, no quotation): general empirical-methodological claim qualified by 'usually' · note: The paper says merging many data sets usually improves signal-to-noise. also in: merging many data sets · signal-to-noise ratio | concept_edges[10] | 0.95 | the tagger's words |
| concept_edges | weak data —causes→ structure determination status: limited refinement and model-building tests; the paper says 'may improve' and 'seems' harmless · direction on the measure: increases · the relation itself is separately evidenced · strength of the relation: 0.6 — the record marks this weak · status (stated by the tagger, no quotation): limited refinement and model-building tests; the paper says 'may improve' and 'seems' harmless · note: A deliberately low weight preserves the paper's hedging and limited-test basis.
Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but extending the data 'may improve' structure determination; hedged. also in: weak data · structure determination | concept_edges[11] | 0.95 | the tagger's words |
| concept_edges | nominal resolution —contrasted-with→ model quality the relation itself is separately evidenced · strength of the relation: 1 · note: The paper explicitly rejects nominal resolution as a good indicator of model quality. also in: nominal resolution · model quality | concept_edges[12] | 0.95 | the tagger's words |
| content_tags | A THRESHOLD PAPER THAT REFUSES TO SET THE THRESHOLD: `We cannot set definite rules for this, as it depends on what the data are to be used for. It is therefore a mistake to prematurely apply a harsh cutoff ... data can always be excluded later` | content_tags.findings[0] | 0.95 | the tagger's words |
| content_tags | AND THE CONVENTION AND THE MEASUREMENT DISAGREE BY 0.11 ANGSTROM: information limit at 1.95 A where `CC1/2 IS 0.27 AND <I/sigma> IS 0.9`, against `A more conventional cutoff point at <I/sigma> = 2.0 ... AT 2.06 A` | content_tags.findings[1] | 0.95 | the tagger's words |
| content_tags | AND THE THRESHOLD IS NOT A SCALAR: `the resolutions at which <I/sigma> = 2.0 along d1, d2 and d3 being 2.23, 1.98 AND 2.05 A` - one data set, three answers by direction | content_tags.findings[2] | 0.95 | the tagger's words |
| content_tags | AND A QUALITY STATISTIC THAT GETS WORSE AS THE DATA GETS BETTER: `R_MERGE tends to increase with increasing multiplicity, even though the averaged intensities are improving` | content_tags.findings[3] | 0.95 | the tagger's words |
| content_tags | AND THE CONSISTENCY MEASURE IS BLIND TO SHARED SYSTEMATIC ERROR: `internal consistency measures are likely to underestimate the true error, since symmetry-related observations may suffer from the same systematic error` | content_tags.findings[4] | 0.95 | the tagger's words |
| content_tags | AND A CORRECTION TERM KEPT WITHOUT A MEANING: `SdB has no obvious physical interpretation ... may be justified as an empirical `CORRECTION` FACTOR ... this `correction` ... is a fairly crude approximation` | content_tags.findings[5] | 0.95 | the tagger's words |
| content_tags | `the correlation coefficient between random half data sets` - preferred because `The advantage of a correlation coefficient is that it has a well defined range: +1.0 for a good correlation and 0 for no correlation` also in: CC 1/2 | content_tags.definitions[0] | 0.95 | the tagger's words |
| content_tags | `a difficult and possibly contentious question (particularly with referees of papers)` - and `We cannot set definite rules for this, as it depends on what the data are to be used for` also in: resolution | content_tags.definitions[1] | 0.95 | the tagger's words |
| content_tags | the traditional criterion discards a shell the tests say still carries information | content_tags.quantities[0] | 0.95 | the tagger's words |
| content_tags | the threshold is anisotropic and the paper declines to say which direction wins | content_tags.quantities[1] | 0.95 | the tagger's words |
| content_tags | `BUT IT WOULD BE HARD TO MAKE A DEFINITE RULE` - the point of coalescence `varies between different scores and different data sets` | content_tags.quantities[2] | 0.95 | the tagger's words |
| content_tags | CC1/2, preferred over the R factors because `it has a well defined range` and is `less dependent on the distribution of the data` - a statistic chosen for the properties of its SCALE, not for what it measures | content_tags.statistics[0] | 0.95 | the tagger's words |
| content_tags | the per-observation sigma the signal-to-noise criterion divides by - and it is `generally underestimated by all integration programs`, so the traditional cutoff of I/sigma = 2.0 is a threshold on a quantity every program gets wrong by default also in: standard error | content_tags.variables[0] | 0.95 | the tagger's words |
Tags that could not be placed in the text:
What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.