Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011;79 Suppl 10(Suppl 10):74-90.
doi: 10.1002/prot.23131. Epub 2011 Aug 30.

Assessment of protein structure refinement in CASP9

Justin L MacCallum  1 , Alberto Pérez, Michael J Schnieders, Lan Hua, Matthew P Jacobson, Ken A Dill
Affiliations

Assessment of protein structure refinement in CASP9

Justin L MacCallum et al. Proteins. 2011.

Abstract

We assess performance in the structure refinement category in CASP9. Two years after CASP8, the performance of the best groups has not improved. There are few groups that improve any of our assessment scores with statistical significance. Some predictors, however, are able to consistently improve the physicality of the models. Although we cannot identify any clear bottleneck in improving refinement, several points arise: (1) The refinement portion of CASP has too few targets to make many statistically meaningful conclusions. (2) Predictors are usually very conservative, limiting the possibility of large improvements in models. (3) No group is actually able to correctly rank their five submissions-indicating that potentially better models may be discarded. (4) Different sampling strategies work better for different refinement problems; there is no single strategy that works on all targets. In general, conservative strategies do better, while the greatest improvements come from more adventurous sampling-at the cost of consistency. Comparison with experimental data reveals aspects not captured by comparison to a single structure. In particular, we show that improvement in backbone geometry does not always mean better agreement with experimental data. Finally, we demonstrate that even given the current challenges facing refinement, the refined models are useful for solving the crystallographic phase problem through molecular replacement. Proteins 2011;. © 2011 Wiley-Liss, Inc.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Example of a difficult to refine target: TR592. Difficulties arise from the fact that the starting model is already within the thermal ensemble of the native conformation.
Figure 2
Figure 2
Target TR567. Crystal contacts stabilize an alpha helical secondary structure near the N-terminus. However, when those contacts are removed, this helix protrudes into space and is no longer stable..
Figure 3
Figure 3
Refinement target TR576 which might suffer from crystal packing effects. Contacts are stablished both within the biological dimeric unit and within the asymmetric unit (A.U).
Figure 4
Figure 4
Summary of aggregate (all models, all groups) results by score. The numeric values are the percentage of time the structure was made better or worse for each metric.
Figure 5
Figure 5
Summary of scores for the ten groups (all models, top ten groups as judged by cherry-picked overall score). The x-axis shows improvement with respect to the starting model. The numeric values are the percentage of the time that the refined model was better or worse than the starting model. The groups are ordered by the overall performance considering the best model for each target (see Figure 8).
Figure 6
Figure 6
Summary of scores by target (all models, all groups). The x-axis shows improvement with respect to the starting model. The numeric values are the percentage of the time that the refined model was better or worse than the starting model. The targets are ordered by the fraction of time the GDT-HA score was improved.
Figure 7
Figure 7
Summary of the results of the CASP9 Refinement Experiment. Only the models designated as “Model 1” are included. Each column shows one of the metrics we used to evaluate performance. The scales are marked at +/− 1 median absolute deviation (MAD) relative to the Null group. Black points are statistically distinguishable from the Null group; grey points are indistinguishable (Wilcoxon signed-rank test, p = 0.05). The area of each point is proportional to the number of targets that group attempted. A chevron indicates that the corresponding score was off the scale.
Figure 8
Figure 8
Summary of the results of the CASP9 Refinement Experiment. For each target and each group, the best overall performing model is selected. Each column shows one of the metrics we used to evaluate performance. The scales are marked at +/− 1 median absolute deviation (MAD) relative to the “Null” group. Black points are statistically distinguishable from the Null group; grey points are indistinguishable (Wilcoxon signed-rank test, p = 0.05). The area of each point is proportional to the number of targets that group attempted. A chevron indicates that the corresponding score was off the scale..
Figure 9
Figure 9
Groups are unable to correctly rank order their submissions. For each group we plot the average Spearman rank correlation coefficient versus the average change in GDT-HA considering the best model for each target. The shaded region is statistically indistinguishable from randomly ranking five predictions for twelve targets. For groups that did less than twelve targets or less than five predictions for each target, the shaded region would be wider than shown.
Figure 10
Figure 10
Relationship between breadth of sampling and improvement for the top groups (all models, top ten groups by cherry-picked overall score). The histograms indicate the marginal distribution along that axis. The solid line in the upper histogram shows the cumulative distribution
Figure 11
Figure 11
Example of an easily refinable target: TR569..
Figure 12
Figure 12
Example of a hard refinable target: TR557. Several difficulties arise, especially near one of the termini where changes in secondary structure are needed..
Figure 13
Figure 13
Refinement of TR624 was a particularly hard challenge. Local environments near the refinement areas are correct, but a topology change is needed: the β-sheets leading to the red and blue loops have to tilt in opposite ways. Only three groups captured this.
Figure 14
Figure 14
Changes in GDT-HA and agreement with NOE data are partially correlated (all models, all groups)
Figure 15
Figure 15
GDT-HA and MR Z-score are only partially correlated (all models, all groups). Structures with low GDT-HA perform poorly in molecular replacement. Models with higher GDT-HA perform better. A high GDT-HA, however, does not guarantee good performance in molecular replacement..
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. MacCallum Justin L, Hua L, Schnieders MJ, Pande VS, Jacobson MP, Dill KA. Assessment of the protein structure refinement category in CASP8. Proteins: Structure, Function, and Bioinformatics. 2009;77(S9):66–80. - PMC - PubMed
    1. Chopra Gaurav, Summa Christopher M, Levitt Michael. Solvent dramatically affects protein structure refinement. P Natl Acad Sci USA. 2008;105:20239–20244. - PMC - PubMed
    1. Fan Hao, Mark Alan E. Refinement of homology-based protein structures by molecular dynamics simulation techniques. Protein Sci. 2004;13:211–220. - PMC - PubMed
    1. Ishitani Ryuichiro, Terada Tohru, Shimizu Kentaro. Refinement of comparative models of protein structure by using multicanonical molecular dynamics simulations. Mol Simulation. 2008;34:327–336.
    1. Jagielska Anna, Wroblewska Liliana, Skolnick Jeffrey. Protein model refinement using an optimized physics-based all-atom force field. P Natl Acad Sci USA. 2008;105:8268–73. - PMC - PubMed

Publication types

LinkOut - more resources

-