How good are my data and what is the resolution? — TEI-tagged view

Philip R. EvanspersName, Garib N. MurshudovpersName (2013date). “How good are my data and what is the resolution?.” Acta Crystallographica Section Dtitle·venue.

Object of study — WHERE TO CUT A DATA SET OFF - the resolutionterm limit every crystal structure declares, tested against refinement, automated model buildingterm and simulated dataterm, and then explicitly refused as a rule

Research question — at what point weak high-resolution dataterm stop carrying information, and whether any of the data-processing statisticsterm marks that point

Contribution — the AIMLESSterm scaling program, and evidence that CC1/2 is a better resolutionterm criterion than the traditional <I/sigma> - together with the finding that extending past the conventional limit 'does no harm'

Full text — tagged

The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.

How good are my data and what is the resolutionterm? EvanspersName Philip R. a * MurshudovpersName Garib N. a a MRC Laboratory of Molecular Biology, Hills Road, Cambridge CB2 0QH, England Correspondence e-mail: pre@mrc-lmb.cam.ac.uk 01 7 2013 13 6 2013 69 Pt 7 224279 d130700 1204 1214 07 9 2012 02 1 2013 13 06 2013 28 06 2013 13 05 2024 © EvanspersName & MurshudovpersName 2013 2013 https://creativecommons.org/licenses/by/2.0/uk/ This is an open-access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are cited. A full version of this article is available from Crystallography Journals Online. The new scaling program AIMLESSterm is described and tests of refinements at different resolutionsterm are compared with analyses from the scaling step.

Following integration of the observed diffraction spots, the process of ‘data reductionterm’ initially aims to determine the point-group symmetry of the data and the likely space groupterm. This can be performed with the program POINTLESSterm . The scaling program then puts all the measurements on a common scaleterm, averages measurements of symmetry-related reflectionsterm (using the symmetry determined previously) and produces many statistics that provide the first important measures of data qualityterm. A new scaling program, AIMLESSterm , implements scaling models similar to those in SCALAterm but adds some additional analyses. From the analyses, a number of decisions can be made about the quality of the data and whether some measurements should be discarded. The effective ‘resolutionterm’ of a data set is a difficult and possibly contentious question (particularly with referees of papers) and this is discussed in the light of tests comparing the data-processing statisticsterm with trials of refinement against observed and simulated dataterm, and automated model-buildingterm and comparison of maps calculated with different resolutionterm limits. These trials show that adding weak high-resolution dataterm beyond the commonly used limits may make some improvement and does no harm.

data reductionterm data scaling software data statistics Data collection and processing pmc-status-qastatus 0 pmc-status-live yes pmc-status-embargo no pmc-status-released yes pmc-prop-open-access yes pmc-prop-olf no pmc-prop-manuscript no pmc-prop-legally-suppressed no pmc-prop-has-pdf yes pmc-prop-has-supplement no pmc-prop-pdf-only no pmc-prop-suppress-copyright no pmc-prop-is-real-version no pmc-prop-is-scanned-article no pmc-prop-preprint no pmc-prop-in-epmc yes pmc-license-ref CC BY 1. Introduction Following integration of the spots on a set of X-ray diffraction imagesterm to produce a list of reflection intensitiesterm, a series of operations are performed on the data, usually referred to as ‘data reductionterm’. These processes include determination of the point groupterm and if possible the space groupterm, checking for consistent indexingterm where there are alternatives, putting all of the data on a common scaleterm, deciding whether to reject parts of the collected data or to cut the resolutionterm and estimating the structure amplitude | F | from intensity. Statistics on the internal consistencyterm of the data also provide a good indication of the overall quality of the data set. Algorithms and methods for data reductionterm have been well documented in many papers ( e.g. Fox & Holmes, 1966 ▶ ; Otwinowski et al. , 2003 ▶ ; EvanspersName, 2006 ▶ , 2011 ▶ ; Kabsch, 2010 ▶ ) and the details will not be repeated here. Scaling attempts to correct for contributions to the measured intensities arising from experimental conditionsterm which vary during data collection, such as variations in the incident-beam intensityterm, the volume of the crystal illuminatedterm, absorption in the primary or secondary beamterm and average radiation damageterm. This is performed by trying to make all symmetry-related or replicated measurements of a reflection intensity equal, i.e. to make the data as internally consistent as possible.

This paper describes a new data-scaling programterm, AIMLESSterm , and discusses criteria for deciding the ‘resolutionterm’ of a measured data set. In the CCP 4 context (Winn et al. , 2011 ▶ ), AIMLESSterm is used immediately after the program POINTLESSterm , which determines the likely point groupterm and possible space groupterm, as well as optionally combining data from multiple files each containing a ‘sweep’ of data and putting them on a common indexing system if necessary (EvanspersName, 2011 ▶ ). 1 AIMLESSterm is then followed by CTRUNCATEterm , which calculates the structure amplitudes | F |term from the intensities and outputs various intensity statistics mainly to detect twinning.

2. The scaling program AIMLESSterm AIMLESSterm is a new implementation in C++ of a classic scaling methodterm, designed to make it easy to add new scaling models and algorithms. It is a replacement for the earlier CCP 4 program SCALAterm (EvanspersName, 2006 ▶ , 2011 ▶ ) and at present uses a very similar scale modelterm. The function minimized is where I h l is the l th observation of reflection h , g h l is its associated inverse scaleterm, w h l = 1/σ 2 ( I h l ) and 〈 I h 〉 is the weighted average intensity for all observations l of reflection h or its symmetry mates. The inverse scaleterm g h l comes from the refined scale modelterm and is a function of the crystal rotation angleterm ϕ as a proxy for primary beam direction s 1 and for radiation doseterm (or time) and the secondary beam direction s 2 : g = g primary (ϕ)exp[−2 B (ϕ)sin 2 θ/λ 2 ] g secondary ( s 2 ). The relative B -­factor term is largely an average radiation-damage correctionterm. The smoothed primary scale factorsterm g primary (ϕ) and relative B factorsterm B (ϕ) are determined at suitable intervals in ϕ and are interpolated using Gaussian weightsterm. The secondary beam correctionterm g secondary ( s 2 ) is determined as a sum of spherical harmonic termsterm. Parameter restraint termsterm (ties) include a sphericity restraintterm on g secondary ( s 2 ), tying all spheri­cal harmonic coefficients to zero and optional ties between adjacent primary scales and relative B factorsterm.

AIMLESSterm iterates the scaling step with an optimization of the standard error estimatesterm on each observation σ( I h l ) and outlier rejectionterm. Note that the scaling process is generally hugely overdetermined (many more observations than parameters, e.g. 45 000 observations for 30 parameters) and that weak intensities do not contain much useful information about the scales, so that the scaling can be made faster (roughly linearly) by working with a selected subset of strong reflectionsterm. The principal steps in the process (in the current version) are as follows. (i) Read all observations into a reflection-list object ( SCALAterm does not store the observations but rereads them for each scaling cycle etc. , as storing large data sets was impracticable when SCALAterm was written). Sort symmetry-related observations and partial observations together if necessary.

(ii) Set up the scaling model depending on what data are present and explicit user-given control instructions (if any). For the smoothly varying scale and B factors, decide on suitable interval for the scales for each ‘run’ or ‘sweep’ of contiguous images, depending on the total length of the sweep and default or specified values.

(iii) Obtain initial rough scale estimates from average intensities.

(iv) First-round scaling with a sample of a few thousand strong reflectionsterm with I /σterm( I ) greater than a suitable minimum value. At this stage the σ( I ) estimates read from the integration program may not be very accurate but are good enough for this selection.

(v) First outlier rejectionterm, using an algorithm similar to that described in §A5 of EvanspersName (2006 ▶ ).

(vi) For data from MOSFLM , which outputs two estimates of each intensity, optimize the level of intensity I mid at which to switch (smoothly) between using the profile-fitted value I prf for weak dataterm and summation integration I sum for strong data, where I ′ is the summation integration intensity before application of the Lorentz and polarization corrections. The exact form of this weighted mean is not critical, as ideally the two estimates are the same.

(vii) First optimization of σ( I ) estimates (see § 2.1).

(viii) Main scaling with strong reflectionsterm chosen on normalized intensities E 2term (typically choosing only observations with 0.8 < E 2 < 5). This gives a subset of data distributed over all resolutionterm ranges.

(ix) Second outlier rejectionterm as before.

(x) Final optimization of σ( I ) estimates as before.

(xi) Final outlier rejectionterm as before.

(xii) Accumulate and print statistics.

(xiii) Output merged or unmerged reflection liststerm to files.

2.1. Standard error estimatesterm Initial estimates of the standard errorterm·var of each intensity observation, σ( I h l ), are generally underestimated by all integration programs, so AIMLESSterm , like SCALAterm , updates the σ( I ) estimates in an attempt to make the average standard errorterm·var match the average scatter of observations as a function of intensity only. The mismatch arises partly from the unknown ‘gain’ of the detector ( i.e. detector units per photon), which scales the Poisson-statistic error estimatesterm (as well as correcting for the detector point-spread function), and partly from a variety of instrumental instabilitiesterm which cause an increase in error with increasing intensity. If the error estimate were correct in explaining the observed discrepancies within the data set, then the normalized deviationsterm (where n h is the number of observations of reflection h and here I h l is the scale-corrected value, i.e. I h l / g h l ) should be distributed with a mean of 0.0 and a standard deviation of 1.0. 2 The standard error estimateterm is then adjusted to optimizing the values of the ‘correction’ factors SdFacterm, SdBterm and SdAddterm to make variance(δ h l ) equal to 1.0 over all intensity ranges. The value of the term inside the square root is (arbitrarily) set to a minimum value of 0.1σ( I h l ) 2 to avoid possible negative values (a rare possibility). A separate set of correction factors is determined for each ‘run’ (usually, although an option is available to use the same value for all runs) and separately for fully recorded and partially recorded observations (if relevant). This is performed by minimizing summed over all intensity ranges i , currently with equal weights w i on each intensity binterm and loose restraints on SdBterm parameters to avoid extreme values. Final values of variance(δ h l ) as a function of intensity for each run are plotted in the output of the program as an indication of the success (or otherwise) of this ‘correction’. Sdfacterm can be identified primarily with the uncertainty in the detector gainterm and Sdaddterm with general instability factors which lead to an error proportional to intensity (see, for example, Diederichs, 2010 ▶ ), but SbB has no obvious physical interpretation. However, this factor helps to flatten the plot of variance(δ h l ) against intensity and may be justified as an empirical ‘correction’ factor. It should be noted that this ‘correction’ to the error estimates is a fairly crude approximation, as it assumes there are no major residual systematic errors (such as those arising from radiation damageterm, for example) and that the correction can be parameterized purely on intensity.

3. Analysis of data qualityterm Once we have a set of scale factorsterm to put all the intensities on a common scaleterm and improved estimates of the error on each intensity, then we can analyse the data for internal consistencyterm and for signal-to-noise ratioterm. We can do this as a function of image numberterm (equivalent to time or crystal rotation) to detect radiation damageterm and weak or inconsistent regions of dataterm and against resolutionterm to decide on a high-resolution cutoff (see §4 ). Note that internal consistency measuresterm are likely to underestimate the true error, since symmetry-related observations may suffer from the same systematic error.

Internal consistencyterm has traditionally been measured by R factorsterm relating an individual observation I h l (after scaling) to the (weighted) average of all symmetry-related or replicate observations of the unique reflection h , 〈 I h 〉. The multiplicity-weighted R measterm is an improvement over R mergeterm , as it is relatively insensitive to data multiplicityterm (Diederichs & Karplus, 1997 ▶ ; Weiss & Hilgenfeld, 1997 ▶ ; Weiss, 2001 ▶ ), whereas R mergeterm tends to increase with increasing multiplicityterm, even though the averaged intensities are improving. R p.i.m.term provides an estimate of data qualityterm after merging multiple observations.

An alternative way of measuring internal consistencyterm for the analysis against resolutionterm is to split the observations randomly into halves and then calculate the linear correlation coefficientterm between the halves. This is arguably the most reliable measure and is discussed further below.

3.1. Analysis against ‘batch’ or image numberterm Analysis of various parameters as a function of batch or image numberterm, a proxy for crystal rotation, time or radiation doseterm, is useful to determine whether the crystal has suffered unduly from radiation damageterm or whether there are any other parts of the data which should be discarded. AIMLESSterm plots similar graphs of R mergeterm and of scale and relative B factorsterm to those produced by SCALAterm (see, for example, Fig. 2 of EvanspersName, 2011 ▶ ). Increasing negative values of the relative B factorterm are an indicator of deterioration with dose, although the B factor is also affected by factors other than radiation damageterm. AIMLESSterm adds two new plots against batch numberterm: a rough estimate of the maximum resolutionterm for each image and a cumulative completenessterm (for all data and anomalous pairs; see Fig. 1 ▶ ). The ‘maximum resolutionterm’ is estimated from the point at which 〈 I /σterm( I )〉 falls below 1.0 for each batch: note that this 〈 I /σterm( I )〉 is without averaging multiple measurements (which would not generally occur on the same image), so will be smaller than the 〈 I /σterm〉 after averaging (§ 3.2.1). The plot shows the noisy values for each batch, as well as a smoothed plot typically averaged over a 5° range. This is only a rough estimate of resolutionterm (see § 4), but serves to illustrate any trends. The cumulative completenessterm plot helps in deciding whether cutting back data from the end because of radiation damageterm would compromise the completeness. Such decisions are more complicated if the data have been measured from multiple ‘sweeps’ or multiple crystals.

3.2. Analysis against resolutionterm In order to estimate the useful ‘resolutionterm’ of the data, i.e. the resolutionterm at which the data may be truncated without losing significant information, AIMLESSterm plots various measures of signal to noise and internal consistencyterm against resolutionterm. There is at present no general consensus on the optimum criteria for interpretation of these plots and how to estimate the point at which adding additional high-resolution data is not adding anything useful: the true ‘resolutionterm’ of a data set has often been a point of contention with referees of papers.

3.2.1. Signal-to-noise ratioterm One obvious way to judge data significance is from the average signal-to-noise ratioterm of the merged intensities as a function of resolutionterm. This is calculated as [after ‘correcting’ the σ′( I h l ) estimates; § 2.1], i.e. for each reflection h the average intensity over symmetry mates 〈 I h 〉 is divided by its estimated error σ(〈 I h 〉) and this ratio is averaged in resolution binsterm [reported as Mn(I/sd) in the program output]. Commonly used resolution-cutoff levelsterm are typically in the range 1–2: even in a resolution binterm with 〈 I /σterm〉 = 1 a proportion of intensities are significantly above the noise levelterm [∼5–7% I > 3σ, ∼20–25% negative]. 〈 I /σterm〉 is a good criterion for resolution cutoffterm, but it does suffer from uncertainties in the estimation of σ( I ), both from inadequacies in the integration program and the necessary σ( I ) ‘correction’ (see § 2.1; Ian Tickle, in a private communication, has pointed out that the major correction applies to large intensities and therefore would not affect the weak high-resolution dataterm relevant to determination of resolution cutoffterm). It should be noted that 〈 I /σterm〉 is not independent of measures of internal consistencyterm because the corrections to σ′( I ) are adjusted to match the scatter of observations. Thus, σ′( I ) estimates are still likely to be underestimates of the true standard deviations.

3.2.2. Measures of internal consistencyterm The traditional R factorsterm measuring internal consistencyterm, R mergeterm or better R measterm (§ 3), are not suitable measures for setting a resolution cutoffterm (EvanspersName, 2011 ▶ ; Karplus & Diederichs, 2012 ▶ ), despite their apparent popularity with referees. As pointed out by Karplus and Diederichs, these R factorsterm cannot be compared with the R factorsterm in refinement, since R mergeterm and R measterm both tend to infinity as the data become weaker, while R crystterm (either R workterm or R freeterm ) tends to a constant (see Appendix A 1 and Luzzati, 1953 ▶ ). This means that there is no sensible way of setting a maximum acceptable level. Note that this is a difference between an R factorterm on intensity, which can be measured as zero or negative, and an R factorterm on amplitude | F |, which cannot be negative and will be biased positive if, for example, the TRUNCATE procedure of French & Wilson (1978 ▶ ) has been applied (or indeed any protocol which sets negative values to positive). R mergeterm or R measterm are useful metrics for monitoring variation with batch (§ 3.1), and their value for the strongest intensities (top intensity binterm or low-resolution bin) is a good indicator of overall data qualityterm. A large R factorterm for the strong data may indicate a problem [a serious cause for concern if R mergeterm (strong) > 0.10; ideally it should be <0.05], but they are not good indicators for weak dataterm. In general, since R factorsterm are dependent on the distribution of the dataterm (see, for example, MurshudovpersName, 2011 ▶ ) they are not good indicators of model qualityterm or internal consistencyterm, whereas the correlation coefficientsterm are indicators of the degree of linear dependence between data sets and are less dependent on the distribution of the dataterm, so they may be better indicators (see Appendix A 1 ).

A better measure for assessing the ‘resolutionterm’ of a data set is the correlation coefficientterm between random half data sets, CC 1/2term (labelled ‘CC_Imean’ in older versions of SCALAterm ; EvanspersName, 2006 ▶ , 2011 ▶ ; Karplus & Diederichs, 2012 ▶ ). This statistic is plotted against resolutionterm in both SCALAterm and AIMLESSterm (Fig. 2 ▶ ). A related statistic, Fourier shell correlationterm, has been used for assessing the resolutionterm of electron-microscopy reconstructions since the early 1980s (see, for example, Rosenthal & Henderson, 2003 ▶ ; Henderson et al. , 2012 ▶ ). The advantage of a correlation coefficientterm is that it has a well defined range: +1.0 for a good correlation and 0 for no correlation. CC 1/2term is generally close to 1 at low resolutionterm and falls sharply to near zero at higher resolutionterm as the intensities become weaker (Fig. 2 ▶ ).

3.2.3. Anisotropyterm Most data sets are anisotropic to some extent, which complicates analysis and decision making. The anisotropyterm of the data is analysed in AIMLESSterm using both CC 1/2term and 〈 I /σterm〉. The two or three orthogonal principal directions of anisotropyterm are constrained by the lattice symmetry: no anisotropyterm for cubic symmetry, two principal directions, the unique c ( c *) axis and the ab plane, for tetragonal, hexagonal and trigonal, three orthogonal axes for orthorhombic, the unique b ( b *) axis and two orthogonal axes in the ac plane for monoclinic and three general orthogonal axes for the triclinic system. The general directions for triclinic and in the monoclinic ac plane are determined from the eigenvectors of a fitted anisotropic scale factorterm. Anisotropyterm is then analysed in cones around the principal axesterm (default semi-angle 20°) and within the same angle of a principal plane, or alternatively as projections onto the two or three principal directions. In the former case, observations are weighted with a cosine weight declining from 1 along the principal direction to 0 at the edge of the region. Plots of CC 1/2term and 〈 I /σterm〉 then allow assessment of resolutionterm in different directions in the same way as for the overall resolutionterm and with the same difficulties.

4. Tests of resolution cutoffsterm How can we decide where to apply a resolution cutoffterm? On the one hand, using high-resolution data which are so weak as to be insignificant will add nothing useful, may add unwanted noiseterm to maps and may lead us to overconfidence in the quality of our model (although the relationship between data qualityterm and model qualityterm is not clear). On the other hand, we do not want to exclude useful data which might aid structure solution and improve the final model. Anisotropyterm in the data complicates this decision, as it is not clear whether it is better to include data based on the best direction, the worst, or something in between. Anisotropic cutoffsterm are likely to cause artefacts in map calculationterm. The problem of anisotropic data and how to deal with them in refinement and map calculation is an open question, and future work needs to address this problem, with the goal of developing clear protocols.

To examine these questions, a number of tests were carried out using data integrated beyond what would normally be considered acceptable using the example data sets listed in Table 1 ▶ .

4.1. Comparison with simulated dataterm By comparing refinement against measured data with refinement against simulated dataterm, we can judge the resolutionterm point at which the measured data become no better than simulated dataterm and compare this with the data-processing statisticsterm. Figs. 2 ▶ ( a )–2 ▶ ( d ) show various analyses for example 1, for which simulated dataterm were generated beyond 2.4 Å resolutionterm in several ways with simulations based on the observed intensity distribution but not on the structure itself (see Appendix A 2 for details of the simulation): (i) the expected value of | F | at the resolutionterm and anisotropic position of each reflection, i.e. all | F |s close together in reciprocal space are the same (denoted F expectedterm ), (ii) a number of data sets with random intensities around the expected intensities with the same distribution as the measured intensities at the same resolutionterm. These simulated intensities were converted to amplitudes (denoted F randomterm ) with the CCP 4 program CTRUNCATEterm using the same procedure as used for the experimental data (French & Wilson, 1978 ▶ ).

The model was then refined with REFMACterm (MurshudovpersName et al. , 2011 ▶ ) to 1.83 Å resolutionterm against the observed data ( F obs ) and also against various simulated dataterm sets. As shown in Fig. 2 ▶ ( a ), R freeterm values (and R workterm ; not shown) for the experimental data increase with increasing resolutionterm as expected, while against the simulated dataterm there is a sudden increase at 2.4 Å resolutionterm where the refinement switches from the experimental data, but beyond about 1.95 Å resolutionterm R freeterm for F obs is no better than that for F expectedterm , while refinement against random data sets (with different errors) gives R freeterm values which converge towards the values from F obs and F expectedterm at higher resolutionterm at around R = 0.42, the expected value (see Appendix A 1 ). We could conclude from this that there is no gain in including data beyond about 1.95 Å resolutionterm, at which point CC 1/2term is 0.27 and 〈 I /σterm〉 is 0.9: a more conventional cutoff point at 〈 I /σterm〉 = 2.0 would be at 2.06 Å resolutionterm. This data set is significantly anisotropic, with orthogonal principal axes along d1 = 0.97 h + 0.23 l , d2 = k , d3 = −0.75 h + 0.67 l , with the resolutionsterm at which 〈 I /σterm〉 = 2.0 along d1 , d2 and d3 being 2.23, 1.98 and 2.05 Å, respectively (Fig. 2 ▶ b ). Analysis of R factorsterm in cones in the same way (§ 3.2.3 and Fig. 2 ▶ b ) shows a similar pattern to the overall values, with a convergence of R factorsterm against F obs and F expectedterm in the range 2.08–1.91 Å.

The TRUNCATE procedure for inferring | F | from experimentally measured I produces a positive biasterm for weak intensities, which complicates the comparison with simulated dataterm. To test the effect of this, refinement was also carried out against observed and simulated intensities instead of against amplitudes: R freeterm values against simulated dataterm were larger from 2.4 Å resolutionterm as before, but had converged by ∼1.98 Å resolutionterm (Fig. 2 ▶ c ). Following refinement against intensities, REFMACterm calculates R factorsterm in a way that mimics R factorsterm on F (MurshudovpersName et al. , 2011 ▶ ), but these R factorsterm rise sharply with resolutionterm (ultimately to infinity), rather than flattening out as do those on | F |.

Another estimate of the significance of the data is shown by the maximum-likelihood scale factor Dterm . The best electron densityterm after refinement is calculated using coefficients 2 mF o − DF c , where m is dependent on D . The values of D within resolutionterm shells indicate how much these reflections contribute to electron densityterm, with a value close to zero indicating that these reflections make little contribution. Fig. 2 ▶ ( d ) shows that (for example 1) D falls with resolutionterm but is still greater than the values for simulated dataterm even at the resolutionterm edge.

Similar tests were carried out for examples 4 and 5 (see Figs. 2 ▶ e and 2 ▶ f ), with similar conclusions that by the time CC 1/2term has fallen to around 0.2–0.4, or 〈 I /σterm〉 to around 0.5–1.5, there is little information remaining, but that it would be hard to make a definite rule. With increasing resolutionterm the R factorsterm and D values for experimental data converge towards the values from simulated dataterm, but the point at which they coalesce relative to the data-processing scores varies between different scores and different data sets.

4.2. Tests with automated model buildingterm There have been anecdotal reports that extending the resolutionterm to include weak dataterm may help automated model-building proceduresterm. Unfortunately, this has been hard to prove: the examples tried here either worked at all resolutionsterm or largely failed at all resolutionsterm. Example 2 was used to test model building from a rather poor map with experimental phasesterm to 2.5 Å resolutionterm and model building with the Buccaneer / REFMAC pipelineterm (Cowtan, 2012 ▶ ) tested at resolutionsterm of 2.3, 2.4, 2.5 and 2.6 Å. Models built at all of these resolutionsterm had some correct and some incorrect parts, with the assigned sequence being largely wrong, but there was no consistent pattern over the different resolutionsterm in the number of residues built or sequence assignment or in the correctness of the models. Another test was performed on example 3, a 2:1 complex, starting with a molecular-replacement modelterm with the two large molecules (595 residues each) and building the smaller 91-residue molecule. In this case the smaller component was built more or less consistently at any resolutionterm between 2.57 and 3.3 Å, although perhaps with fewer errors away from these extremes at between 2.7 and 3.1 Å. Similar results were obtained with example 4, omitting the last domain (207 residues): this could be rebuilt more or less successfully at resolutionsterm between 1.8 and 2.8 Å, although maybe slightly more correctly at higher resolutionsterm. However, extending the resolutionterm at least seemed to do no harm. It might indicate that the Buccaneer / REFMAC pipelineterm for model building at least is more dependent on phase errorterm rather than quality of structure-factor amplitudesterm (and hence resolutionterm). In cases of difficulty, it may be worth trying model building with data to different resolutionsterm.

4.3. Electron densityterm in OMIT mapsterm The effect of resolutionterm on the visual appearance of difference mapsterm was tested on regions of the model which were not included in refinement. This is important in manual building and completion of structures. Part of a structure was omitted, the remaining coordinates were perturbed by a random shift of up to 0.3 Å to reduce model biasterm and the structure was refined in REFMACterm with different resolution cutoffsterm. Fig. 3 ▶ shows two examples of maximum-likelihood difference mapsterm at different resolutionsterm for examples 4 and 1. There is not much difference between the maps, although there may be a slight improvement in sharpness in example 1 (Fig. 3 ▶ b ) on extending from 2.4 to 2.0 Å resolutionterm, consistent with the refinement results in § 4.1 (Fig. 2 ▶ a ). Again, as for the automated model buildingterm (§ 4.2), extending the resolutionterm seems at least to do no harm.

5. Discussion The program AIMLESSterm performs essentially the same task as SCALAterm and gives similar results. However, it is significantly faster (about three times) and is a better framework for adding new scaling models and analyses. In due course, the three programs POINTLESSterm , AIMLESSterm and CTRUNCATEterm may be combined into one.

The extensive statistics produced by AIMLESSterm are mostly similar to those produced by SCALAterm , so the questions and decisions for the user are as discussed in EvanspersName (2011 ▶ ): (i) What is the point groupterm (Laue group)? (ii) What is the space groupterm? (iii) Is there radiation damageterm and should the most damaged regions of data be excluded? (iv) What resolution cutoffterm should be applied (see below)? (v) Is there a detectable anomalous signal? (vi) Are the data twinned? (vii) Is this data set better than those previously collected?

With regard to this last point, one traditional way of improving weak or incomplete data is to merge data from different crystals, provided that they are isomorphous. With the advent of cryocoolingterm, this has fallen out of fashion except for the most desperate cases, but recent work from Hendrickson’s group (Liu et al. , 2012 ▶ , 2013 ▶ ) makes a good case for merging data from many crystalsterm to enhance the very weak anomalous signal from sulfurterm. The current tools for checking isomorphismterm between crystals are rather undeveloped, but this technology is improving (see Giordano et al. , 2012 ▶ ). With the fast data collectionterm on modern synchrotrons, it is common to collect several or many more-or-less equivalent data sets and merging them may be better than just choosing the best. At the end, model qualityterm depends on data qualityterm and merging many data setsterm usually improves the signal-to-noise ratioterm in the data; however, it is not clear how to merge data when there is severe non-isomorphismterm or radiation damageterm. Blindly merging data may do more harm than good. It is necessary to analyse the joint distribution of all data setsterm and to merge using this distribution. In an ideal world all data sets would be used without merging, thus ensuring that the extraction of information from the data would be optimal at all stages of structure analysis: the distribution of the dataterm and the amount of information available at each stage of analysis would define what needs to be used.

A major cause of user indecision and conflicts with journal referees is the resolution cutoffterm. We cannot set definite rules for this, as it depends on what the data are to be used forterm. It is therefore a mistake to prematurely apply a harsh cutoff at the data-reduction stage: data can always be excluded later. It is also unclear how best to handle anisotropyterm: do you choose the best direction or the worst? It is probably best to include data to the limit in the best direction. Tests carried out here to relate the resolutionterm statistics to final model building and refinement do suggest that extending the data somewhat beyond the traditional limits such as 〈 I /σterm〉 = 2 may improve structure determinationterm, as do the ‘paired-refinementterm’ tests of Karplus & Diederichs (2012 ▶ ). At the very least, adding these weak dataterm seems to do no harm for the purposes of either automatic or manual model building. The main problem is that we have become accustomed to using the nominal resolutionterm as an indicator of model qualityterm, and it is not a good indicator, particularly as important biological and chemical conclusions from a structure often depend on local rather than global correctness. Nor indeed can any global score can indicate the correctness of any local structural featureterm. Clearly, well measured data to 1.5 Å resolutionterm contain more information than a data set to 3.5 Å resolutionterm and are therefore likely to lead to a more correct structure, but nominal resolutionterm in itself just tells us how many reflections were used, rather than their quality. From our limited tests here, it seems that changing the resolution cutoffterm over a considerable range ( e.g. from 2.2 to 1.9 Å) makes only a small difference, so the exact cutoff point is not a question to agonize over, but it seems sensible to set a generous limitterm so as not to exclude data containing real (if weak) information. There is no reason to suppose that cutting back the resolutionterm of the data will improve the model. These tests were performed with current programs and our current procedures at all stages could be improved to extract the maximum information from weak noisy dataterm.

1 For the naming of the programs POINTLESSterm and AIMLESSterm , see Gibbons (1932 ▶ ).

2 Note that (as pointed out by a referee) this is only strictly true if all weights used in the calculation of 〈 I h 〉 are equal: this needs further investigation, although some preliminary tests using different weighting schemes (including equal unit weights) showed only small differences.

We would like to thank Stephen Graham and Janet Deane for test data sets and Andrew Leslie for reading the manuscript. PRE was supported by MRC grant U105178845 and GNM by MRC grant MC_UP_A025_1012.

Appendix A Statistical methods A1. Behaviour of crystallographic R values Crystallographic R factorsterm are calculated using the formula where the summation is over the reflections used to calculate the R value (in the case of twinning a generalization of this formula is used; see, for example, MurshudovpersName, 2011 ▶ ). Obviously, the behaviour of this statistic depends on the statistical properties of the structure factors. Therefore, it can be expected that properties of the model, crystal and observed data such as (i) noisy data, (ii) twinning, (iii) modulation in crystals and (iv) model errors will affect the behaviour of the R value. Luzzati (1953 ▶ ) analysed the effect of model errors on R values in the absence of any other peculiarity and came to the conclusion that R values calculated for structure factors calculated from random atoms would be around 0.58 for acentric reflections. MurshudovpersName (2011 ▶ ) carried out a similar analysis for cases of hemihedral twinning and demonstrated that R values for cases of hemihedral twinning are systematically lower than those for single crystals. Here, we analyse the effect of the procedure used to estimate the amplitude | F | from the measured intensity I when data are very noisy. Since weak intensities may be measured as negative, while the true amplitude | F | cannot be negative, | F | is generally estimated using the TRUNCATE procedure (French & Wilson, 1978 ▶ ). Under the assumption that true structure factors come from a crystal filled with atoms randomly distributed over the unit cell and the noise in the experimental intensities has a normal distribution, the TRUNCATE procedure estimates the amplitudes of structure factors using a Bayesian estimation with the formula where P ( J ; I o , crystal) is the conditional distribution of ideal intensities of structure factors ( J ) when observed intensities ( I o ) are known and it is known that the data came from a crystal. Noting that | F | = J 1/2 , and using the explicit form of P ( J ; I o , crystal), the conditional distribution for | F | can be written where ∊Σ = E (| F | 2 ), the expected value of | F | 2 , is estimated using the data in the resolution binsterm, I o is the experimental observed intensity and σ is its standard deviation. Thus, the expected values of the amplitudes of structure factors are estimated using the formula Although this formulation has served the community well over the years, it has certain problems. These problems include the following. (i) It is assumed that Σ is a smooth function of resolutionterm, which breaks down in the presence of pseudo-translation, DNA/RNA helices etc. (ii) It is assumed that the data are from single crystals. In cases of twinning the formulation may not work, although if the twinning fraction is known and there is no noncrystallographic rotation parallel to the twin operators then it is straightforward to account for twinning. However, in general not all properties of the crystal/data are known or possible to model. (iii) It is assumed that the observed data have a normal distribution. This assumption may break down, especially for weak reflections where profiles of neighbourhood spots are used for integration.

When the data are very noisy ( i.e. the standard deviation of the observation becomes very large) this procedure produces the expected value of Wilson’s distribution (Wilson, 1949 ▶ ), Thus, in the limiting case if all estimation of parameters proceeds smoothly and there are no crystal-growth peculi­arities, then the TRUNCATE procedure would give the expected value of the Wilson distribution. It is interesting to analyse the crystallographic R values for these cases. Let us assume that the R value is calculated in the resolution binsterm where Σ was estimated. Then, in the limiting case of noisy data the R value would have the following form (here, we assume that reciprocal-space points are sufficiently dense and the summation can be replaced by the integration), Now if we use the fact that the distribution of | F c | is the same as the distribution of | F |, that is the Wilson distribution, and denote μ = E (| F |) = E (| F c |), then after some manipulation of integrals we can derive where erfc is the complementary error function, Thus, we see that if structure factors are replaced by the expected value of the Wilson distribution then it can be expected that the calculated R values will be around 0.42. This is exactly what happens with observed values from the TRUNCATE procedure when errors in the experimental intensities become very large. It should be noted that this behaviour of R values is a property of data from single crystals with no other peculiarities.

Note that for perfect hemihedral twinning If the perfect twinning assumption is used in the TRUNCATE procedure then for very noisy data we can obtain A2. Generation of random intensities from Gaussian χ 2 distribution Under the assumption that the ‘true’ structure factors came from the population with a Wilson distribution and experimental intensities have a normal distribution with mean value equal to the ‘true’ intensities and with experimental uncertainties equal to σ, the distribution of observed intensities can be written To use this formula for random intensity generation it is necessary to estimate the unknown parameters (Σ). If we want to generate ‘data’ beyond the resolutionterm of the observed data we need to parameterize Σ as a smooth function of resolutionterm. We use the parameterization where k is the overall scale, B is the overall isotropic temperature factor, f ( s ) is the empirical intensity curve designed by Popov & Bourenkov (2003 ▶ ), s is a reciprocal-space vector, | s | is the length of the reciprocal-space vector and U is the overall anisotropic U value with properties where tr is the trace, R sym is the rotation part of a symmetry operator (here, we use the orthogonal version of the symmetry operators) and the superscript T denotes the transpose of the operator. The second condition in (22 ) is for all symmetry operators of the crystal. Since (21 ) is a continuous function of Σ, it can be used for any resolutionterm including resolutionsterm beyond the observed resolutionterm.

The procedure for estimation of parameters and generation of random intensities is as follows. (i) Using E ( I o ) = ∊Σ, var( I o ) = (∊Σ) 2 , build a Gaussian approximation for the distribution of the observed intensities and estimate parameters of Σ as defined by (21 ).

(ii) Using the distribution (20 ), build the likelihood function and improve the estimation of the parameters of Σ using maximum-likelihood estimation.

(iii) Generate expected values of the amplitudes of structure factors for reflections within defined resolutionterm using the formula E (| F |) = (π∊Σ) 1/2 /2.

(iv) Generate random intensity from the population defined by the distribution (20 ). For this, a two-stage procedure is used: ( a ) generate a random number from the exponential distribution with mean ∊Σ and denote it I exp and ( b ) generate a random number from the normal distribution with mean I exp and standard deviation σ. Σ is extrapolated to the given resolutionterm and σ is taken from the observed data or from the defined signal-to-noise ratioterm.

(v) Generate random numbers from the population with Wilson distribution with the parameter ∊Σ.