lDDT: a local superposition-free score for comparing protein structures and models using distance difference tests

2.2 Multireference lDDT

The lDDT can be computed simultaneously against multiple reference structures of the same protein at the same time. The set of reference distances L includes all pairs of corresponding atoms, which, in all reference structures, lie at a distance closer than the reference threshold R_o. For each atom pair, the minimum and the maximum distances observed across all the reference structures are compared with the distance between the corresponding atoms in the model M being evaluated. The distance is considered preserved if it falls within the interval defined by the minimum and the maximum reference distances or if it lies outside of the interval by less than the predefined length threshold. The fraction of preserved distances is computed like in the single reference case.

2.3 Stereochemical quality checks

To account for stereochemical quality and physical plausibility of the model being evaluated, the calculation of the lDDT can take violations of structure quality parameters into account. Here, stereochemical violations in the model are defined as bond lengths and angles with values that diverge from the respective average reference value derived from high-resolution experimental structures (Engh and Huber, 1991, 2006) by more than a predefined number of standard deviations (12σ by default; see Supplementary Material). Interatomic distances between pairs of non-bonded atoms in the model are considered clashing if the distance between them is smaller than the sum of their corresponding atomic van der Waals radii (Allen, 2002), within a predefined tolerance threshold (by default 1.5 Å). Tolerance thresholds can be defined for each pair of atomic elements independently.

In case where the side-chain atoms of a residue show stereochemical violations or steric clashes, all distances that include any side-chain atom of this residue are considered as not preserved for the lDDT calculation. In case the back-bone atoms are involved in stereochemical violations or steric clashes, all distances that include any atom of the residue are treated as not preserved.

2.4 Determination of the optimal inclusion radius R_o

To determine the optimum value of the inclusion radius parameter R_o for lDDT, an analysis of predictions of all multidomain targets evaluated during the CASP9 experiment (Kinch et al., 2011; Mariani et al., 2011) was carried out (see Supplementary Table S1 for a complete list). GDC-all scores for predictions covering >50% of the target protein sequence were computed based on the AUs definitions by the CASP9 assessors (Kinch et al., 2011). A weighted whole target GDC-all score was computed for each target as the average GDC-all scores of its AUs weighted by the AU size. GDC-all scores are an all-atom version of GDT with thresholds from 0.5 to 10 in steps of 0.5 Å. GDC-all scores were computed using LGA version 5/2009 (Zemla, 2003), using a 4Å cut-off for the sequence-dependent superposition.

lDDT scores were calculated for the whole targets by including all residues that are covered by any AU, and in an AU-based form using the same weighting scheme already applied to GDC-all scores. The inclusion radius parameter was varied in the range from 2 to 40 Å, and the correlation R² score between the distribution of weighted averaged GDC-all scores and the distribution of lDDT scores was computed and plotted against the value of the inclusion radius (Figs. 2 and ​and33).

Open in a separate window

Fig. 2.

Determination of the optimal inclusion radius parameter R_o. Pearson correlation (R²) between whole target lDDT scores (solid line) and domain-based weight-averaged lDDT score (dashed line) versus domain-based weight-averaged GDC-all scores for different values of the inclusion radius parameter R_o were computed over all CASP9 predictions for multidomain targets

Open in a separate window

Fig. 3.

Correlation between whole structure GDC-all and lDDT scores and domain-based weight-averaged GDC-all scores. For CASP9 predictions of multidomain targets, GDC-all scores (red dots) and lDDT scores (blue dots) were computed against the whole unsplit target structures. For the lDDT scores, the default value of 15 Å for the inclusion radius was used

2.5 Validation of baseline scores for different folds

To analyze the influence of the protein fold of the assessed structure on the lDDT score, pseudorandom models were created for different architectures in the CATH Protein Structure Classification system (Cuff et al., 2011) using the following procedure: representative domains longer than 50 residues were selected as evenly as possible among the topologies of the CATH classification. For each domain, side-chain coordinates were removed and then rebuilt using the SCWRL software package (with default parameters) (Krivov et al., 2009). Pseudorandom models representing threading errors were then generated by shifting all residues by one alignment position in a backbone-only model, and rebuilding the side-chains with SCWRL4, and computing the corresponding lDDT score. The procedure was repeated iteratively until a threading error of 50 residue positions was reached. This method is loosely based on the approach described in Shi et al. (2009). In Figure 4, we show the results for CATH Architecture entries 1.25 (Alpha Horseshoe) and 2.40 (Beta-barrel), each represented by 60 example structures.

Open in a separate window

Fig. 4.

Baseline lDDT scores for models with simulated threading errors. lDDT scores of pseudo-models with threading errors for two examples of different CATH Architectures are shown: Alpha Horseshoe (left) and Beta Barrel (right). The lDDT score is plotted as a function of the introduced threading error (top). The histograms (bottom) show the distribution of these ‘baseline’ scores for threading error offset >15 residues for the two architectures. The structure inlays show an example structure of the respective CATH Architecture. Peaks at large off-sets indicate repetitive structural elements with locally correct arrangement

For estimating lDDT scores of random protein pairs, 200 protein models with wrong fold were generated by selecting pairs of structures with different CATH topologies, generating models by rebuilding side chains on the backbone of the other protein, and computing lDDT scores for these decoy models. The median of the resulting distribution was 0.20, with a 0.04 mean absolute deviation.

3.1 Optimal choice of the inclusion radius parameter R₀ makes lDDT largely insensitive to domain movements

3.2 Validation of lDDT score baselines for different protein folds

Because lDDT scores express the percentage of inter-atomic distances present in the target structure that are also preserved in the model, a value of ‘0’ corresponds to 0 conserved distances, and ‘1’ to a perfect model. However, these extreme values are in practice rarely observed, even in extremely high and low-quality models. At the high-accuracy end, fluctuations in surface side chain conformations will result in values <1. For very low accuracy models, still some local inter-atomic distances will be preserved if the model has at least a stereochemically plausible structure and features some secondary structure elements. In the context of protein model assessment, two types of baseline values are of interest: the expected score when comparing two random structures, and scores for models with correct folds but including threading errors.

In principle, the first value could be estimated using Flory–Huggins polymer solution theory (Flory, 1969; Huggins, 1958), e.g. as done for the determination of RPF/DP values for NMR structures (Huang et al., 2012). However, because protein structures are rich in rigid structural elements like α-helices and β-sheets, where the relative local positions are restricted, they show in general a higher number of preserved local distances than random polymers. Based on these considerations, we decided to empirically derive lDDT baseline scores by comparing a reference structure with a set of well-defined decoy models. A comparison of the Flory–Huggins and decoy-based analysis can be found in the Supplementary Materials. The average lDDT score when comparing random structures, i.e. protein models with different architectures (see Materials and Methods), is 0.20 (±0.04). For estimating the effect of alignment shifts in models with otherwise correct fold and stereochemistry, we created pseudomodels starting from the original protein structure and introducing threading errors of increasing magnitude for different representative structure architectures from CATH (Cuff et al., 2011). We then compared the pseudomodels with the original structure, computing their lDDT scores against it.

Here, we show the results for CATH architecture entries 1.25 (Alpha Horseshoe) as example for proteins rich in α-helices, and 2.40 (Beta-barrel) as representative for a β-sheet protein (Fig. 4). The plots at the top of each panel show the value of the lDDT scores (on the y-axis) for 60 pseudomodels as a function of the magnitude of the threading error (residue offset) on the x-axis. For large threading errors, the lDDT scores converge to a ‘baseline’ range of scores, which appear to be largely independent of the threading error magnitude. We considered scores in this range to be typical lDDT scores for a low-quality model with the same architecture as the target structure. For models in the Alpha Horseshoe architecture, the average baseline lDDT score is ∼0.28, whereas for the Beta barrel class, the value of 0.22 is lower, illustrating the influence of the architecture of the protein. This indicates that the lower boundary of the lDDT score can vary as a function of the architecture of the target protein, which influences the comparison of absolute raw scores of models for different folds, but not of models of the same architecture. This is a common behavior of most structure comparison measures.

One interesting feature in Figure 4 is the presence of several peaks at larger threading errors (e.g. around residue 34) in the Alpha Horseshoe architecture example. These peaks correspond to internal repeats in the structure, which give rise to locally correct models when the threading shift coincides with the size of the repeat.

3.3 Local model accuracy assessment

Modeling errors are typically not homogenously distributed over the model, but are localized, e.g. in template-based models often in segments that had to be remodeled de novo. Residue-based lDDT scores quantify the model quality on the level of a residue’s environment. The low sensitivity of lDDT to relative domain movements also applies to per-residue scores. As shown in Figure 1, local lDDT scores are not dominated by different domain orientations between the target and the model structures, but correctly reflect the accuracy of the local atomic environment surrounding the residue under investigation in the model. Figure 1 shows a superposition of the structure of target T0542 (in light gray) with prediction by group TS236 (colored according to the full-length lDDT score). The models represent each of the two individual domains with high accuracy, but their relative domain orientation does not correspond to the target structure. Superposition-based scores would assign a high score to one of the domains but not to the other, or require scoring based on isolated domain. As illustrated on the right panel (Fig. 1), residues with low lDDT score correspond to regions of large local structural divergence between the two domain structures, irrespective of the domain movement between them. As expected, low local scores can also be detected at the interface between the two domains where the interactions cannot be modeled correctly without knowing their relative orientation in the target.

3.4 Stereochemical realism assessment

Although validation of the stereochemical plausibility of protein models is a routine procedure for experimental structure determination, e.g. in X-ray crystallography (Read et al., 2011), this is not a common practice in theoretical modeling. Depending on the applied method, models generated in silico may reveal rather unrealistic stereochemical properties. Typically, numerical scores applied in retrospective model assessment compute a measure for the average atomic dislocation between the reference structure and the model, without considering the stereochemical quality of the latter. Consequently, two models with similar average atomic displacements may nevertheless differ significantly in their stereochemical plausibility, and some models might include atomic arrangements that are physically impossible.

To address this question, lDDT incorporates a stereochemical plausibility check, which assesses two aspects of model quality: the lengths of chemical bonds and the widths of angles in the model structure. Bond and angle measurements are compared with a set of standard parameters derived from high-resolution crystal structures (Engh and Huber, 2006). A stereochemical violation is defined as a parameter deviating from the expected values by more than a specified number of standard deviations (default: 12σ; see Supplementary Material). Inter-atomic distances between non-bonded atoms in the model are compared with the sum of their Van der Waals radii (Allen, 2002), and a violation (‘clash’) is assigned if two atoms are closer than the sum of the Van der Waals radii, allowing a certain tolerance (default: 1.5 Å). When calculating the lDDT score, all distances involving side-chain atoms of a residue involved in any type of stereochemical violations in the model are considered as non-preserved. In cases where backbone atoms are involved in stereochemical violations, all distances involving this residue are considered non-preserved. This approach leads to the lowering of the final lDDT score of a model according to the extent of the structure’s stereochemical problems (Fig. 5).

Open in a separate window

Fig. 5.

Assessing stereochemical plausibility. This example illustrates the stereochemical quality checks on lDDT score for a model (TS276, left side as ribbon representation) for target T0570-D1 with unrealistic stereochemistry (close-up, right). Residues with too short (1) or too long (2) chemical bonds, as well as those with close atomic interactions (3) or impossible bond angles (4), result in lower scores during the lDDT computation

As an example, Figure 5 shows the CASP9 prediction TS276_1 for target T0570-D1. The backbone of the prediction can be superposed accurately to the backbone of the target structure (left panel), and the prediction has indeed a high GDT-HA score (0.814). Displacement-based all-atom scores do not immediately reveal the problems, with a GDC-all score of 0.705 and an lDDT score without stereochemical checks of 0.682. However, when the lDDT score includes stereochemical check, the lDDT score drops to 0.571. Panel B shows a close-up of the region around residue alanine 21, where several stereochemical violations are evident.

The authors gratefully acknowledge the financial support by the SIB Swiss Institute of Bioinformatics.

Conflict of Interest: none declared.