Bioinformatics Vol. 16 no. 9 2000
Pages 776-785
© 2000 Oxford University Press
Original Paper |
MaxSub: an automated measure for the assessment of protein structure prediction quality
1 Department of Chemistry, Ben Gurion
University, Beer-Sheva 84015, Israel
2 Stockholm Bioinformatics Center, 106 91
Stockholm University, Sweden
3 International Institute of Molecular and
Cell Biology, Ks. Trojdena 4, 02-109 Warsaw, Poland
4 Department of Bioinformatics Computer
Science, Ben Gurion University, Beer-Sheva 84015, Israel
Received on February 17, 2000
; revised on May 5, 2000
; accepted on May 5, 2000
Motivation: Evaluating the accuracy of predicted models is critical for assessing structure prediction methods. Because this problem is not trivial, a large number of different assessment measures have been proposed by various authors, and it has already become an active subfield of research (Moult et al. (1999) Critical assessment of methods of protein structure prediction (CASP): round III. Proteins , Suppl32–6). The CASP (Moult et al. (1997) Critical assessment of methods of proteins structure prediction (CASP): round II. Proteins , Suppl3, Dedicated Issue, Moult et al. (1999) Critical assessment of methods of protein structure prediction (CASP): round III. Proteins , Suppl32–6) and CAFASP (Fischer et al. (1999) CAFASP-1: critical assessment of fully automated structure prediction methods. Proteins , Suppl3, 209–217) prediction experiments have demonstrated that it has been difficult to choose one single, ‘best’ method to be used in the evaluation. Consequently, the CASP3 evaluation was carried out using an extensive set of especially developed numerical measures, coupled with human-expert intervention. As part of our efforts towards a higher level of automation in the structure prediction field, here we investigate the suitability of a fully automated, simple, objective, quantitative and reproducible method that can be used in the automatic assessment of models in the upcoming CAFASP2 experiment. Such a method should (a) produce one single number that measures the quality of a predicted model and (b) perform similarly to human-expert evaluations.
Results: MaxSub is a new and independently developed method that
further builds and extends some of the evaluation methods introduced
at CASP3. MaxSub aims at identifying the largest subset of
atoms of a model that superimpose
‘well’ over the experimental structure, and produces a
single normalized score that represents the quality of the model.
Because there exists no evaluation method for assessment measures of
predicted models, it is not easy to evaluate how good our new
measure is. Even though an exact comparison of MaxSub and the CASP3
assessment is not straightforward, here we use a test-bed extracted
from the CASP3 fold-recognition models. A rough qualitative
comparison of the performance of MaxSub vis-a-vis the human-expert
assessment carried out at CASP3 shows that there is a good agreement
for the more accurate models and for the better predicting groups.
As expected, some differences were observed among the medium to poor
models and groups. Overall, the top six predicting groups ranked
using the fully automated MaxSub are also the top six groups ranked
at CASP3. We conclude that MaxSub is a suitable method for the
automatic evaluation of models.
Availability: MaxSub is available at: http://www.cs.bgu.ac.il/~dfischer/MaxSub/MaxSub.html
Contact: {nomsiew,dfischer}@cs.bgu.ac.il; arne{at}biokemi.su.se; leszek{at}iimcb.gov.pl
Supplementary
Information: Full tables are available at:
http://www.cs.bgu.ac.il/
dfischer/MaxSub/MaxSub.html CASP
web site: http://PredictionCenter.llnl.gov/casp3/CAFASP
web site: http://www.cs.bgu.ac.il/~dfischer/CAFASP2
To whom correspondence should be addressed.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  G. Nicosia and G. Stracquadanio Generalized Pattern Search Algorithm for Peptide Structure Prediction Biophys. J., November 15, 2008; 95(10): 4988 - 4999. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. Eramian, N. Eswar, M.-Y. Shen, and A. Sali How well can the accuracy of comparative protein structure models be predicted? Protein Sci., November 1, 2008; 17(11): 1881 - 1893. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E. Capriotti and M. A. Marti-Renom RNA structure alignment by a unit-vector approach Bioinformatics, August 15, 2008; 24(16): i112 - i118. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. Larsson, B. Wallner, E. Lindahl, and A. Elofsson Using multiple templates to improve quality of homology models in automated homology modeling Protein Sci., June 1, 2008; 17(6): 990 - 1002. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. J. Reid, C. Yeats, and C. A. Orengo Methods of remote homology detection can be combined to increase coverage by 10% in the midnight zone Bioinformatics, September 15, 2007; 23(18): 2353 - 2360. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. V. Glyakina, S. O. Garbuzynskiy, M. Yu. Lobanov, and O. V. Galzitskaya Different packing of external residues can explain differences in the thermostability of proteins from thermophilic and mesophilic organisms Bioinformatics, September 1, 2007; 23(17): 2231 - 2238. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Fasnacht, J. Zhu, and B. Honig Local quality assessment in homology models using statistical potentials and support vector machines Protein Sci., August 1, 2007; 16(8): 1557 - 1568. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J.-M. Chandonia StrBioLib: a Java library for development of custom computational structural biology applications Bioinformatics, August 1, 2007; 23(15): 2018 - 2020. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. Fariselli, I. Rossi, E. Capriotti, and R. Casadio The WWWH of remote homolog detection: The state of the art Brief Bioinform, March 1, 2007; 8(2): 78 - 87. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. B. Pandit, Y. Zhang, and J. Skolnick TASSER-Lite: An Automated Tool for Protein Comparative Modeling Biophys. J., December 1, 2006; 91(11): 4180 - 4190. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. Chivian and D. Baker Homology modeling using parametric alignment ensemble generation with consensus and energy-based model selection Nucleic Acids Res., October 18, 2006; 34(17): e112 - e112. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
F. Armougom, O. Poirot, S. Moretti, D. G. Higgins, P. Bucher, V. Keduas, and C. Notredame APDB: a web server to evaluate the accuracy of sequence alignments using structural information. Bioinformatics, October 1, 2006; 22(19): 2439 - 2440. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. N. Offman, P. W. Fitzjohn, and P. A. Bates Developing a move-set for protein model refinement Bioinformatics, August 1, 2006; 22(15): 1838 - 1845. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. Eramian, M.-y. Shen, D. Devos, F. Melo, A. Sali, and M. A. Marti-Renom A composite score for predicting errors in protein structure models Protein Sci., July 1, 2006; 15(7): 1653 - 1666. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. Wallner and A. Elofsson Identification of correct regions in protein models using structural, alignment, and consensus information Protein Sci., April 1, 2006; 15(4): 900 - 913. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. W. Ginzinger and J. Fischer SimShift: Identifying structural similarities from NMR chemical shifts Bioinformatics, February 15, 2006; 22(4): 460 - 465. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
F. Birzele, J. E. Gewehr, and R. Zimmer QUASAR--scoring and ranking of sequence-structure alignments Bioinformatics, December 15, 2005; 21(24): 4425 - 4426. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. S. Pettitt, L. J. McGuffin, and D. T. Jones Improving sequence-based fold recognition by using 3D model quality assessment Bioinformatics, September 1, 2005; 21(17): 3509 - 3515. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. Lupyan, A. Leo-Macias, and A. R. Ortiz A new progressive-iterative algorithm for multiple structure alignment Bioinformatics, August 1, 2005; 21(15): 3255 - 3263. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
H. Zhou, C. Zhang, S. Liu, and Y. Zhou Web-based toolkits for topology prediction of transmembrane helical proteins, fold recognition, structure and binding scoring, folding-kinetics analysis and comparative analysis of domain combinations Nucleic Acids Res., July 1, 2005; 33(suppl_2): W193 - W197. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. Wallner and A. Elofsson All are not equal: A benchmark of different homology modeling programs Protein Sci., May 1, 2005; 14(5): 1315 - 1327. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Ginalski, N. V. Grishin, A. Godzik, and L. Rychlewski Practical lessons from protein structure prediction Nucleic Acids Res., April 1, 2005; 33(6): 1874 - 1891. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Soding Protein homology detection by HMM-HMM comparison Bioinformatics, April 1, 2005; 21(7): 951 - 960. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
Z. Zhang, S. Kochhar, and M. G. Grigorov Descriptor-based protein remote homology identification Protein Sci., February 1, 2005; 14(2): 431 - 444. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. Rychlewski and D. Fischer LiveBench-8: The large-scale, continuous assessment of automated protein structure prediction Protein Sci., January 1, 2005; 14(1): 240 - 245. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
X. Ye, P. K. O'Neil, A. N. Foster, M. J. Gajda, J. Kosinski, M. A. Kurowski, J. M. Bujnicki, A. M. Friedman, and C. Bailey-Kellogg Probabilistic cross-link analysis and experiment planning for high-throughput elucidation of protein structure Protein Sci., December 1, 2004; 13(12): 3298 - 3313. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. E. Kim, D. Chivian, and D. Baker Protein structure prediction and analysis using the Robetta server Nucleic Acids Res., July 1, 2004; 32(suppl_2): W526 - W531. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. Fischer and L. Rychlewski The 2002 Olympic Games of Protein Structure Prediction Protein Eng. Des. Sel., March 1, 2003; 16(3): 157 - 160. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. R. Ortiz, C. E.M. Strauss, and O. Olmea MAMMOTH (Matching molecular models obtained from theory): An automated method for model comparison Protein Sci., November 1, 2002; 11(11): 2606 - 2621. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. Bonneau, I. Ruczinski, J. Tsai, and D. Baker Contact order and ab initio protein structure prediction Protein Sci., August 1, 2002; 11(8): 1937 - 1944. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Lundstrom, L. Rychlewski, J. Bujnicki, and A. Elofsson Pcons: A neural-network-based consensus predictor that improves fold recognition Protein Sci., November 1, 2001; 10(11): 2354 - 2362. [Abstract] [Full Text] [PDF]
|
|