Geometric deep learning KDD 2021
Turn the chain. Its graph does not move.
The paper writes a protein backbone as a graph, with a distance on every pair of residues and two dihedral angles on every residue, a description that stays the same however the protein is turned or moved, and trains one network, PG-GNN, to predict both parts together.The paper writes a protein backbone as a graph of distances and dihedral angles, which reads the same however the protein is turned, and trains one network to predict all of it.
Figure 1Illustration
This figure is interactive and needs JavaScript. The still shows its starting state: a toy chain of 28 residues and, next to it, the contact map read off its graph. In the live figure the chain turns, every coordinate changes, and the matrix and the angles stay exactly as they were.
Point at the matrix: every cell is one pair of residues.
01 The problem
Predicting a protein’s structure from its sequence is usually split into smaller problems. A common route predicts a contact map first, a grid that marks each pair of residues as closer than a threshold or not, and then rebuilds the structure from it. The paper’s first objection is that such a map leaves the backbone free to move.
Figure 2Illustrationa toy backbone built by this page, not the trained network: none of its numbers is a result from the paper
This figure is interactive and needs JavaScript. It starts from a toy coil of 28 residues and searches, one small turn of one angle at a time, for other shapes in which no contact (two Cα atoms under 8 Å) appears or disappears. The contact map stays identical while the coil bends into a different shape and distances between residues move by more than 30 Å. From the toy hairpin, whose arms are tied together by contacts, the same search still moves distances by more than 2 Å.
Press Search. Then point at the square to read one pair.
Each press tries 2400 turns of one angle by up to 14°. A turn is kept only if nothing clashes (no two Cα atoms three or more apart closer than 4 Å), if it does not bring the shape back towards the start and, under the first rule, if no contact appears or disappears. The search is seeded, so every visitor sees the same run. The two shapes are laid over each other at residue 14. The constants are this page’s. The contact rule is the paper’s: two Cα atoms under 8 Å (Section 5.1).
A matrix of exact Cα distances is finer than a contact map, but the paper calls it still insufficient, “because of the absence of structural information of other backbone atoms”. The bonds on either side of each Cα rotate, so the other atoms can sit in more than one place. The rotation on both sides of the Cα is the extra information the paper adds to the distances.
Section 1, first challenge
The paper notes that “residues sequentially far apart might be in spatially close contact in protein's 3D structure”. Earlier predictors slide small image filters over the pair map, and the paper argues that such filters focus on neighbours in the sequence. Section 03 below shows what that means cell by cell.
Section 4.2 (the quotation) and Section 1, second challenge (the image filters)
Fixed size networks crop or pad the pair map. The paper warns that this “could potentially cause the loss of relational details or undesired distortion”, and asks for a model that takes a whole protein of any length as one graph.
Section 1, third challenge
02 The idea
The backbone is a chain of flat units with two free angles per residue, φ and ψ. The paper’s target is a graph over the residues: every pair carries the distance between its two Cα atoms, and every residue carries its φ and ψ. Turn or move the protein and every coordinate changes, while none of these numbers do.
Figure 3Illustrationa toy backbone built by this page, not the trained network: none of its numbers is a result from the paper
This figure is interactive and needs JavaScript. The still below shows its starting state: a toy hairpin of 28 residues on the left and its contact map on the right. The stripe that runs away from the diagonal is the two arms of the hairpin lying side by side: residues far apart in the sequence and close in space.
The chain coordinates in space
Its graph distances on edges, φ and ψ on nodes
Under 6 apart: 0 (outside the paper’s ranges, so not counted below)
The stripe running away from the diagonal is the hairpin’s two arms lying side by side: far apart in the sequence, close in space. Point at the matrix, or focus it and use the arrow keys, to read one pair.
TryTurn it. Every coordinate changes, and “changed by turning it” stays at 0.000.
All of these are chosen by this page, not taken from the paper: 28 residues; bond lengths N to Cα 1.458 Å, Cα to C 1.525 Å, C to N 1.329 Å; bond angles 111.0°, 116.2° and 121.7°; the peptide torsion held flat at 180°. Only φ and ψ are free, which follows the paper’s description of the backbone as “coplanar units … with two primary degrees of freedom”. Every atom is placed from the three before it. The toy has no side chains and no energy, so it will happily take shapes a real protein cannot.
The preset angles (φ, ψ) are: Stretched (−139°, 135°), Coiled (−57°, −47°), Hairpin arms (−100°, 119°) with a four residue turn. Random draws each residue from a few ordinary angle pairs and rejects shapes in which two Cα atoms three or more apart come closer than 4 Å. The dials show φ and ψ measured back from the coordinates with the paper’s two plane definition. The first residue has no φ and the last has no ψ, because each needs a neighbour the chain does not have. The short, medium and long range bands outlined on the square are the paper’s (Section 5.3.1), and the accent cross marks every edge that touches the selected residue.
The authors write that, compared with Cartesian coordinates of nodes, the representation “gives a significant advantage because of its invariance to rotation and translation of the graph”. Two copies of one fold in different poses have different coordinates and the same graph, which is what Figure 1 on this page measures on the toy.
Sections 1 and 3
The distance matrix on its own “cannot fully characterize the overall backbone structure due to the rotation of chemical bonds around Cα”. And when distances and angles are generated on separate paths, the paper notes, “the predicted attributes may not be consistent as they are generated from different paths”. Figure 3 on this page shows how tightly the two are tied in the toy: a single angle moves a block of distances.
Appendix A.2 and Section 4.1
03 Reading the pair map
Earlier contact predictors treat the L × L map as a picture and slide small image filters over it. Around the cell for residues i and j, a 3 × 3 filter sees only pairs made from their next-door neighbours. The paper adds an edge-to-edge filter that reads the whole row and the whole column through the cell: every pair that shares residue i or residue j.
Figure 4Illustrationfootprints only, on the toy chain’s own contact map: no filter weights are involved and nothing here is a result
This figure is interactive and needs JavaScript. On the contact map of the toy hairpin it outlines two footprints round one target cell: the square that a stack of 3 × 3 image filters can reach, and the full row and column that one edge-to-edge filter reads. For the pair of residues 6 and 23, a stack with a stride of 1 and no dilation needs 22 layers before its square holds that whole row and column. The edge-to-edge filter reads all 55 cells of them in one layer.
One 3 × 3 layer reads 9 cells.
04 Follow one chain
Follow the toy hairpin from Figure 3 on this page through PG-GNN, the paper’s Protein Geometric Graph Neural Network. Shapes are written for its 28 residues. Sizes the paper does not give, such as the feature widths D and K, stay as letters.
Figure 5Illustration
The example at this stop targets from the toy, not network output
The input is a graph of the sequence: one node per residue and an edge between neighbours along the chain, L − 1 of them. Each node carries features of its residue and each pair carries pairwise features. The order of the residues is known. The shape is not.
For the edge path, the features of residue i and residue j pass through a 1D convolution and are joined onto the pair’s own features. The result is one stack of numbers for every cell of a 28 × 28 map.
Inside a block, 3 × 3 convolutions read each cell’s local neighbours, one edge-to-edge filter reads the cell’s whole row and column, and an identity branch carries the input through. The three are added and fed to the next block. The paper uses 10 blocks with 4 convolution layers in each.
The edge path ends in a softmax, trained with cross entropy against the true distance label of each pair. This is E′, the edge half of the target graph. The paper does not say how many distance classes it uses.
The node path takes the node features and, at each iteration, the distance matrix the edge path has just produced. That matrix becomes the edge representation for message passing, so the angles are predicted with the distances in view.
A message from residue w to residue v is w’s hidden state multiplied by a matrix that depends on the edge between them. A gated recurrent unit then folds the incoming messages into v’s state. The paper stacks 6 such blocks.
A readout maps each final state to sin φ, cos φ, sin ψ and cos ψ. The angles are recovered with an arctangent of the sine over the cosine, as the paper prints it. This is F′, the node half of the target graph.
The edge loss is cross entropy over all pairs. The node loss is the mean squared error on the sines and cosines. The total is the edge loss plus λ times the node loss. The paper describes the network as trained under the co-guidance of both paths, and does not report the value of λ.
Select any block to read what it does and where the paper says so. Arrow keys move between blocks.Tap any block to read what it does and where the paper says so.
Nodes are the L residues. The input edges are the L − 1 edges of the protein sequence, one between each pair of neighbours along the chain. Node features F include a position-specific scoring matrix, predicted secondary structure and solvent accessibility. Pair features E include co-evolution information, a distance potential and an inter-residue coupling score.
Section 3, Appendix B.1
In the paper’s words, “the node features Fi and Fj are transformed by 1D convolution then concatenated into Ei,j and Ej,i to be used as feature map for edge attributes generation.” The feature widths D and K are not given.
Appendix B.1
Each block has “one identity mapping branch, one edge-to-edge convolution branch, and one two-dimensional (2D) convolution branch”. The paper describes the edge path as a deep residual convolutional network, and this branch is its residual connection.
Sections 4.1 and 4.2
3 × 3 filters, each followed by batch normalisation and an ELU activation, 4 layers per block applied in sequence. The paper assigns this branch the local relations and notes that dilated filters can also be used here.
Section 4.2, Appendix B.2
A filter that “can consider the edges that connect to one residue”. The paper’s Fig. 4 labels its kernel “Kernel=W×1, 1×H” and draws it as a cross: the full row and the full column through the cell, which is what Figure 4 on this page lets you move around. One such layer is applied to the block’s input, in parallel with the convolutions (Appendix B.2. The paper’s Fig. 4 draws this row with an ellipsis, as it does the 2D row.) Shared weights keep the representation “independent of the size of the input”.
Section 4.2, Eq. 3, the paper’s Fig. 4, Appendix B.2
“All layer outputs and residual connections are added together and fed into the next block.” In the paper’s words, each edge is then generated from both its own two residues and the “edges that connected to the same nodes”.
Section 4.2
A softmax over distance labels for every pair, trained with cross entropy averaged over all L × L pairs. In the experiments these outputs are scored as contact predictions: Cα atoms under 8 Å. The number of distance classes and their boundaries are not given.
Section 4.1, the paper’s Fig. 3, Sections 5.1 and 5.3.1
“We take the output distance matrix from the edge translation path at each iteration to use as edge representation” for the node path. The message from w to v is A(evw) hw: w’s hidden state times a matrix chosen by the edge between them (Eq. 4). The paper names the quantity handed over in three ways, and its exact form is not given.
Sections 4.1 and 4.3, Eq. 4
In the paper’s words, “we update the hidden state by the Gated Recurrent Units (GRU)”, from the old state and the aggregated message (Eq. 5). The node path has 6 message passing blocks with ReLU activations.
Section 4.3, Appendix B.2
The readout R maps the final hidden states to four numbers per residue, the sine and cosine of each angle, “for training purposes”. The paper gives no further reason for this encoding.
Section 4.3, Eq. 6
φ and ψ are derived as the arctangent of the predicted sine over the predicted cosine. φ measures the rotation of the N to Cα bond and ψ the rotation of the Cα to CO bond. The first node feature is φ and the second is ψ.
Section 4.3, Section 3, the paper’s Fig. 11
The same residues as the input, now with pairwise distance information on the edges (E′) and the two dihedral angles on the nodes (F′). Both “are invariant under rotation of the coordinate system and graph translation”, which is what Figure 1 on this page measures on the toy.
Section 3
L = L(E, E′) + λ × L(F, F′) (Eq. 2). The dashed link in the drawing is this training signal, not data flowing forward. The reason given for joint training: on separate paths “the predicted attributes may not be consistent”. The value of λ is not reported.
Section 4.1, Eq. 1 and 2
05 The evidence
Models were trained on 4726 proteins from PDB25 (Section 5.1; Section 6.3 gives 4725) and tested on five sets: a held out PDB25 split of 500 proteins, the CASP11 and CASP13 domains, 76 hard CAMEO proteins and 400 membrane proteins. Proteins longer than 300 residues were excluded. The neural baselines were re-implemented so that, in the paper’s words, they train “with the same data and features”. CCMpred was run from its source code.
Figure 6Measured, Tables 1 and 2
| Datasets | GCN(k=5) | GCN(k=10) | GCN(k=20) | MPNN |
|---|---|---|---|---|
| CASP11 | 32.350 | 34.107 | 33.343 | 25.696 |
| CASP13 | 32.401 | 33.027 | 32.876 | 28.729 |
| CAMEO | 32.385 | 32.854 | 33.337 | 26.635 |
| membrane | 32.063 | 33.285 | 32.891 | 23.892 |
| PDB25 | 32.143 | 33.027 | 34.642 | 24.114 |
| Datasets | GCN(k=5) | GCN(k=10) | GCN(k=20) | MPNN |
|---|---|---|---|---|
| CASP11 | 81.451 | 80.500 | 80.592 | 42.003 |
| CASP13 | 81.864 | 83.178 | 84.556 | 46.944 |
| CAMEO | 81.863 | 84.185 | 80.602 | 44.110 |
| membrane | 82.229 | 81.423 | 84.559 | 35.713 |
| PDB25 | 82.224 | 83.177 | 85.316 | 37.688 |
GCN is a plain graph convolution network with dynamic k nearest neighbour selection. Column names are the paper’s, and the paper prints nothing in bold in these tables, so nothing is bold here. Both angles are drawn on one axis because every value in Table 2 is larger than the value in the same cell of Table 1. The paper does not comment on that difference, and this page offers no explanation for it.
Figure 7Measured, Section 6.2.1
| Test set | Without the edge-to-edge branch | With it, the proposed ETP | Difference |
|---|---|---|---|
| membrane | 43.3 | 45.5 | 2.2 |
| CASP13 | 44.3 | 48.4 | 4.1 |
| CAMEO | 42.1 | 44.8 | 2.7 |
| CASP11 | 56.4 | 57.6 | 1.2 |
| PDB25 | 59.3 | 60.4 | 1.1 |
ACC is the accuracy of the top L/k predicted contacts, k = 10, 5, 2, 1, over short, medium and long range pairs. The paper does not say how these are averaged into one number. The paper’s Figure 6 labels the network without the branch “ResNet”.
Against a network with only the edge path, the paper reports average ACC improvements for the joint framework of 5.6%, 3.6%, 4.7%, 0.9% and 1.2% on membrane, CASP13, CAMEO, CASP11 and PDB25, and says it “performs better in almost all metrics”. It does not say whether these are relative gains or percentage points, and the underlying values are not tabulated.
Section 6.2.3
The paper’s Figure 5 places eight combinations of edge path and node path on a plane for each test set, contact error (ERR) on the x axis and angle error (MAE) on the y axis, and reports that PG-GNN “shows the best performance across all five datasets”. The figure labels its axes but prints no value for any point, so this page describes it and does not plot it.
Section 6.1, the paper’s Fig. 5
The takeaway
A distance for every pair of residues and two dihedral angles for every residue. It reads the same from every viewpoint, and the paper trains both halves together because, on separate paths, the two “may not be consistent”.
An edge path whose filter reads a pair’s whole row and column, a message passing node path, and a hand-off of predicted distances between them, trained with one loss.
The network does not output 3D coordinates. The abstract’s own summary is that the results “demonstrate the potential of GNNs for 3D structure prediction”.
06 Office hours
Written by Tian's agent while Tian slept and Pepper supervised. Pepper accepts no liability either.
Each answer says where in the paper it comes from, so you can check the agent's work.
No. It outputs distances and angles. For the case study, full atom models were built afterwards by PyRosetta relaxation under the predicted distance and torsion restraints, and the lowest energy model was kept. The comparison with the native structures is visual, for the four chains shown in the paper’s Fig. 9 (2c4j chain A, 2x27 chain X, 3pb8 chain B, 4pzo chain A); the text of Section 6.4 says five domains. The paper reports no RMSD or other numeric structure score.
Source: Sections 5.3.2 and 6.4, the paper’s Fig. 9The paper does not claim or prove that. Its wording is that the Cα distance matrix “determines the overall shape of the backbone structure”, that on its own it “cannot fully characterize” the backbone, and that the angles “could provide further geometric information”. Figure 2 on this page shows the weaker statement the paper does make, about contact maps.
Source: Section 1, Appendix A.1 and A.2The paper’s argument is about what one filter looks at: an image kernel “only focuses on the source and its surrounding pixels”, while, in the paper’s words, all the pairwise features in a cell’s row and column “could potentially influence” it. The evidence it offers is the ablation in Figure 7 on this page (the paper’s Section 6.2.1 and the paper’s Fig. 6): average ACC rises from 43.3% to 45.5% on the membrane set and from 59.3% to 60.4% on PDB25 when the branch is added. A dilated residual network (DRN) is also among the edge path baselines in the paper’s Figure 5, which prints no value for any point, so the size of that gap could only be estimated by eye, which this page does not do.
Source: Section 1, Sections 4.2, 5.2, 6.1 and 6.2.1Proteins longer than 300 residues were excluded. The neural baselines are re-implementations trained on the paper’s data and features, and CCMpred was run from its source code. None of them is the original authors’ server. Training time, inference time, parameter counts, the loss weight λ and the number of distance classes are not reported, and the conclusion’s remark about “less training time” has no measurement behind it in the paper. There is no ablation showing how joint training changes the angle error: that ablation measures contacts only. No experiment measures whether the predicted distances and angles agree with each other. The paper and its poster do not mention a code or data release.
Source: Sections 5.1, 5.2, 6.2.3 and 7, Appendix B.2Not close, on purpose. Figures 1 to 4 on this page run on a 28 residue backbone that this page builds from bond lengths and angles chosen by the page (listed under Figure 3). They show properties of the representation: what turning does, what a contact map forgets, how one angle moves many distances, which cells a filter reads. No network runs on this page and no toy number is a result. Every result the paper reports that appears here is in section 05: this page’s Figures 6 and 7 (copied from Tables 1 and 2 and from the text of Section 6.2.1), the five improvements quoted from Section 6.2.3 beneath them, and the dataset sizes in that section’s opening paragraph. The page also borrows a few of the paper’s settings: the 8 Å contact rule wherever a contact is drawn or counted (Figures 1 to 5 on this page, Section 5.1), the short, medium and long range groups in the read-outs of Figures 1 and 3 and the bands outlined in Figure 3 (Section 5.3.1), and the block counts in Figure 5 (Appendix B.2).