Transcript ScoresCASP8_08

Three scores:
TS, TR and CS
ShuoYong Shi, Ruslan Sadreyev, Jing Tong, David Baker and Nick V. Grishin
http://prodata.swmed.edu/CASP8
Howard Hughes Medical Institute, Department of Biochemistry,
University of Texas Southwestern
Medical Center at Dallas
GDT-TS: the best single score
Why? Because it is 4 scores in one – from 4 different superpositions.
GDT-TS=[N(1)+N(2)+N(4)+N(8)]/(4N)
where N(r) is the number of superimposed residue
pairs with the CA–CA distance < r Å, and N is
the total number of residues in the target.
Since many approaches are trained to produce models scoring better according to
some evaluation method, flaws in the evaluation method will result in better-scoring
models that will not represent real protein structure in any better way.
One of such dangers is compression of coordinates, which decreases the gyration
radius and may increase some scores based on Cartesian superpositions.
http://prodata.swmed.edu/CASP8/evaluation/Scores.htm
Compression is bad, but GDT-type scores favor it
Attraction and repulsion in scores
GTD–TS score measures the fraction of residues in a model within a certain distance
from the same residues in the structure after a superposition. This approach is based
on a "reward".
Taking an analogy with physical forces, such a score is only the "attraction" part of a
potential, and there is no "repulsion" component in GDT–TS. It might have been
reasonable a few years ago, when predictions were quite poor. It was important to
detect any positive feature of a model, since there were more negatives about a model
than positives.
Today, many models reflect structures well. When the positives start to outweigh the
negatives, it becomes important to pay attention to the negatives. Thus we introduced
a "repulsion" component into the GDT–TS score.
When a residue is close to its "correct" residue, GDT–TS rewards it, and if a residue is
too close to "incorrect" residues (other than the residue that is modeled), we subtract
a penalty from the GDT–TS score.
This idea was suggested by David Baker as a part of our collaboration on CASP and
model improvement. We call the score Ruslan Sadreyev and ShuoYong Shi developed
in the Grishin Lab based on this idea TR, i.e. ‘The Repulsion'. TR score, in addition to
rewarding for close superposition of corresponding model and target residues,
penalizes for close placement of other residues.
TR score is calculated as follows:
1. Superimpose model with target using LGA in the sequence-dependent mode,
maximizing the number of aligned residue pairs within distance cutoff=4Å.
2. For each aligned residue pair, calculate a GDT–TS - like score:
S0(R1, R2) = 1/4 [N(1)+N(2)+N(4)+N(8)],
where N(r) is the number of superimposed residue pairs with the Ca–Ca distance <r Å.
3. Consider individual aligned residues in both structures. For each residue R, choose
residues in the other structure that are spatially close to R, excluding the residue
aligned with R and its immediate neighbors in the chain. Count numbers of such
residues with Ca-Ca distance to R within cutoffs of 1, 2, and 4Å. (As opposed to GDT–
TS, we do not use the cutoff of 8Å as too inclusive).
4. The average of these counts defines the penalty assigned to a given residue R:
P(R) = 1/3 * [N(1) + N(2) + N(4)].
5. For each aligned residue pair (R1, R2), the average of penalties for each residue
P(R1, R2) = 1/2 * (P(R1) + P(R2)) is weighted and subtracted from the GDT–TS
score for this pair. The final score is prohibited from being negative:
S(R1, R2) = Max [ S0(R1, R2)-w*P(R1, R2), 0 ].
Among tested values of weight w, we found that w=1.0 produced the scores that were
most consistent with the evaluation of model abnormalities by human experts.
Segments of superimposed structure (black) and model (red)
With 1A distance cutoff.
Superposition does not look very good, but assume that only segments of
larger structures are shown, and the rest of the structures looks better
3
1
2
1
2
3
Scale: 0.6A
GDT-TS calculation for 1A:
find the number of corresponding atoms within 1A.
Total GDT-TS contribution: 0+1+0=1
3
1.34A
2
1.34A
1
1
0.6A
2
3
Scale: 0.6A
Penalty calculation for 1A:
For those residue pairs that contribute to GDT-TS find “incorrect” atoms within 1A
Residue pairs (1,1) and (3,3) do not contribute to penalty,
as they do not contribute to GDT-TS.
3
1.34A
2
1
2
3
1.34A
1
Scale: 0.6A
Penalty calculation for 1A:
For those residue pairs that contribute to GDT-TS find “incorrect” atoms within 1A
Residue pair (2,2) may contribute to penalty.
3
2
1
0.6A
1
2
3
Scale: 0.6A
Penalty calculation for 1A:
Step 1: from the structure (black)
Which “incorrect” residues in the model are within 1A from residue 2 in the structure?
3
1
2
1
2
3
Scale: 0.6A
Penalty calculation for 1A:
Step 1: from the structure (black)
Which “incorrect” residues in the model are within 1A from residue 2 in the structure?
It is residue 1 (0.84A), as residue 2 is “correct”, and residue 3 is 1.2A away.
3
1.2A
2
0.6A
1
2
1
0.84A
3
Scale: 0.6A
Penalty calculation for 1A:
Step 1: from the structure (black)
In the structure, only residue #1 of the model contributes count 1 to the penalty
3
1.2A
2
0.6A
1
2
1
0.84A
3
Scale: 0.6A
Penalty calculation for 1A:
Step 2: from the model (red)
Which “incorrect” residues in the model are within 1A from residue 2 in the model?
3
1
2
1
2
3
Scale: 0.6A
Penalty calculation for 1A:
Step 2: from the model (red)
Which “incorrect” residues in the model are within 1A from residue 2 in the model?
It is residue 1 (0.84A) and residue 3 (0.84A) , as residue 2 is “correct”.
3
2
0.84A
1
1
0.84A
0.6A
2
3
Scale: 0.6A
Penalty calculation for 1A:
Step 2: from the model (red)
In the model, residues #1 and #2 contribute total count 2 to the penalty
3
2
0.84A
1
1
0.84A
0.6A
2
3
Scale: 0.6A
Penalty calculation for 1A:
Step 3: averaging penalty contributions
from the structure and the model
Total penalty= (penalty from the structure + penalty from the model)/2 =
(1+2)/2=1.5
The penalty for 1A is 1.5
TR contribution for 1A:
Compute it as TR = GDT – weight * penalty
Check if TR<0, set it to 0.
Weight
0.25
0.5
1
TR
1 - 0.25*1.5=0.625>0
1 - 0.5*1.5=0.25>0
1 - 1*1.5=-0.5<0, so set TR to 0
Next: compute these for 2A, 4A and 8A, average, and divide by the structure length
R=0.991
Correlation between TR score (vertical axis) and GDT-TS (horizontal axis)
Scores for top 10 first server models were averaged for each domain shown by its number positioned at a point with the coordinates equal to these
averaged scores. Domain numbers are colored according to the difficulty category suggested by our analysis: black - FM (free modeling); red - FR (fold
recognition); green - CM_H (comparative modeling: hard); cyan - CM_M (comparative modeling: medium); blue - CM_E (comparative modeling: easy).
Comparison of remote homologs:
compressing one homolog can increase GDT_TS
Sample of N=2050 of pairs of
SCOP domains sharing superfamily
2.0<DALI Z<5.0
1.00
1.00
0.98
0.98
Relative measure
Relative measure
Lower third by DALI Z
(2.0<DALI Z<5.8)
0.96
0.94
TS
TR
0.92
0.90
0.96
0.94
TS
TR
0.92
0.90
0.88
0.88
90
92
94
96
Shrinking ratio
N=680
98
100
90
92
94
96
Shrinking ratio
N=540
98
100
In 40% pairs of remote homologs
GDT_TS increases with compression
1.02
Relative measure
1.00
0.98
0.96
0.94
TS
TR
0.92
0.90
90
92
94
96
98
100
Shrinking ratio
Domain pairs where compression
causes GDT_TS growth (N=239 of 540)
Compression of FR models can cause GDT_TS growth
SAM-T08 server
1.02
0.98
0.96
1.000
0.94
Relative measure
Relative measure
1.00
0.995
TS
TR
All 108 models
of FR targets
TS
TR
0.990
0.92
0.985
90
92
94
96
98
100
Shrinking ratio
0.90
90
92
94
96
98
100
Shrinking ratio
Models of FR targets where compression
causes GDT_TS growth (N=43)
Contact score CS
Scores comparing intramolecular distances between a model and a structure (contact
scores) have different properties than intermolecular distance scores based on
optimal superposition.
One advantage of such scores is that superpositions, and thus arguments about their
optimality, are not involved.
The problems with developing a good a contact score are
1) contact definition;
2) mathematical expressions converting distance differences to scores.
CS score is calculated as follows:
1. contact between residues is defined by a distance ≤8.4Å between their Cα atoms.
2. The difference between such distances in a model and a structure is computed
and used as a fraction of the distance in the structure.
3. Fractional distances above 1 (distance difference above the distance itself) are
discarded and exponential is used to convert distances to scores (0→1). The factor
in the exponent is chosen to maximize the correlation between contact scores and
GDT–TS scores.
4. These residue pair scores are averaged over all pairs of contacting residues. We
call this score CS, i.e. 'contact score', for short.
R=0.962
Correlation between Contact score CS (vertical axis) and GDT-TS (horizontal axis)
Scores for top 10 first server models were averaged for each domain shown by its number positioned at a point with the coordinates equal to these
averaged scores. Domain numbers are colored according to the difficulty category suggested by our analysis: black - FM (free modeling); red - FR (fold
recognition); green - CM_H (comparative modeling: hard); cyan - CM_M (comparative modeling: medium); blue - CM_E (comparative modeling: easy).
Server rankings on all targets in domains for three scores
On 143 domains, ranking does not change much with score, illustrating that
1) scores correlate with each other and 2) the ranking is robust.
Server rankings on FR domains for three Z-scores
On 28 FR domains, ranking shows small variations illustrating the differences
between individual scores and between servers.
Summary:
1. Single score is not enough for model evaluation.
2. Do not train your method on a single score.
3. Introduction of “repulsion terms” in the score is
useful, as it penalizes compression and may help
improving alignments.
4. Superposition-independent contact scores are
fast and easy to compute, accurate and correlate
well with superposition-based scores.
Acknowledgement
Our group
Shuoyong Shi
Ruslan Sadreyev
Jimin Pei
Sasha Safronova
Hua Cheng
Indraneel Majumdar
Yong Wang
Bong-Hyun Kim
Collaborators
Jing Tong
Lisa Kinch
Ming Tang
Yuan Qi
Jamie Wrabl
Erik Nelson
S. Sri Krishna
Dorothee Staber
David Baker
Kimmen Sjölander
William Noble
U. Washington
UC Berkeley
U. Washington
HHMI, NIH, UTSW,
The Welch Foundation