Geometric deep learning IEEE BigData 2022
Unroll the chain and watch the pocket scatter.
APIP reads a protein as a plain sequence and a ligand as a graph of atoms, and predicts which residue interacts with which atom: where the two meet, not merely whether.
The paper reports that APIP, or its cross-entropy variant, has the highest average AUC among the methods it compares, on two benchmark datasets under three hold-out settings (Section VI-A).
Figure 1Illustration
An invented 64-position chain drawn flat, with an invented ligand. Not a real protein, and APIP is never shown a fold like this. Counts from this drawing, not results.
01 The problem
A protein folds into a pocket that holds the ligand. The paper starts from two observations: the fold is not always available while sequences are plentiful, and the useful answer is a map of residue and atom pairs, not a single yes or no.
Figure 2Illustration
When the structure of the complex is known, as it is in this toy, the interface can be read off it: every residue closer to an atom than a fixed distance is ringed. The dashed circle is that distance, drawn round atom a3: 3 residues fall inside it. The paper notes that “most of the protein-ligand interface prediction studies use both protein and ligand structure features”, and that “not all protein structures are available”. “Especially for some hard to crystallize proteins, such as membrane-spanning proteins”, it adds, “it is unfeasible to obtain the 3D structures”.
It also notes what makes the sequence-only version hard: “The amino acids that are spatially close to form the pocket might be sequentially far apart.”
Source: Section I, pages 1740 to 1741
The same complex as the model receives it. The protein is a sequence of residues, P = (p1, p2, ..., pS). The ligand is a graph: atoms are nodes and chemical bonds are edges, each with attributes. No fold, no distances, no ties.
The paper’s reason: “the number of protein sequences has increased exponentially in the past years”, so a model that needs only the sequence “could potentially take advantage of the large amounts of available sequence data”. It describes the task as one to be “handled in a structural-free way”.
Source: Sections I and III, pages 1740 to 1742
The output is a grid with one cell for every residue and atom pair, S by N likelihoods. “Instead of predicting if the drug and target are interacting or not, what we are more interested in is where it interacts at the atom/residue level.”
binds: yes or noa coarse answer
the answer APIP is asked for
toy 17 of 384 cells are contacts in this toy. In the paper’s words, “a few residues have interactions with the ligand although there are hundreds or even thousands of residues in the protein sequence”.
Source: Sections I and III, pages 1741 to 1742
IllustrationThe same invented complex as above. The paper does not print how its datasets decide that a residue and an atom interact, so this toy uses a fixed distance in the flat drawing. Its numbers are counts of cells, not results.
02 The idea
The protein side of APIP has two encoders. A motif sequence encoder, built from 1D convolutions, reads each residue’s neighbourhood. A global sequence encoder, one block of multi-head self-attention, then connects every position with every other. The paper trained all three settings of the switch below: each encoder alone, and both.
Figure 3Illustration
Drag the ligand. Pick a bead.
Position 29, letter Htoy
Hears from
33 of 64 positions
positions 13 to 45, a window along the chain; 4 of the 8 other residues that touch the ligand are inside
Starts out identical to
no other position
after the convolutions it carries a window of 33 letters, found once in this chain
One tick. Position 29 has something of its own to say, but only 33 of 64 positions can hear it.
The other two settings, same residue
Global encoder only (APIP_T). One tick. Every position can hear position 29, but in this toy it starts out no different from 2 other positions: if nothing marks its place in the chain (the paper does not say), it carries only its own letter H, which occurs 3 times in this chain.
Both (APIP). Two ticks. Position 29 has something of its own to say, and every position can hear it.
The same chain as the sequence APIP readshatched: positions 13 to 45 · equals marks: none
On the strip, Left and Right move one position, Page Up and Page Down move ten, Home and End jump to the ends. The Residue menu does the same.
Reach here is arithmetic for plain stacked convolutions using the paper’s layer count and kernel size (Section V-E). The paper does not state a receptive field, a stride or a dilation.
Connected does not mean attended to. The paper shows no attention weights. The thick band and the dotted line show who can exchange information, never what was learned.
Contacts are the toy’s: a residue and an atom closer than a fixed distance in this flat drawing.
Identical assumes a position enters as its BLOSUM62 row and nothing else. The paper does not say whether APIP or APIP_T adds a positional encoding; with one, positions that share a letter would differ.
IllustrationAn invented chain, an invented ligand, and no trained network. It shows which positions each kind of encoder can connect and which positions it cannot tell apart before any mixing. The three settings carry the names of the variants the paper ran; what removing each one cost is measured, in section 05. This figure does not show what APIP learned, and its numbers are counts, not results.
The paper: “The locality of convolution operation serves to learn the local patterns in protein sequences and contributes an inductive bias for adjacent amino acids to function together in motifs.” It adds that the design handles long sequences “by allowing convolutions to aggregate the local information before a computationally heavy self-attention mechanism” (Section IV-B).
The paper on convolutional predictors: “information flow in the network between distal components is restricted by the locality of convolutions” (Section I). And on its own block: “the long-range patterns cannot be characterized by a traditional convolution module”, so it uses multi-head self-attention “to learn the global information across the protein sequence” (Section IV-B-2).
Weights, training and accuracy. Reach and identical windows are properties of the wiring, true for any weights under those assumptions. Whether a trained APIP uses its reach to find a pocket is a question for the paper’s measurements, and the paper tests it only through prediction quality.
03 One cell of the grid, step by step
The paper has no architecture figure, so this drawing was made for this page from Section IV and Equations 1 to 12. Two lanes, one for the protein as a sequence and one for the ligand as a graph. As the paper describes them, the lanes meet only in the last module. Every box opens.
Figure 4Illustration
A protein is a sequence of S residues. A ligand is a graph: atoms are nodes, chemical bonds are edges. The model must output one likelihood for every residue and atom pair, and the paper handles the two inputs “separately in two different modules” before a pairwise module brings them together.
Each residue enters as its row of the BLOSUM62 substitution matrix. Four blocks of two 1D convolution layers with kernel size 5, leaky ReLU activations and skip connections turn each position into a summary of its surroundings.
The motif encoder’s output is multiplied by a learnable weight and passed on. Multi-head self-attention projects each position to a query, a key and a value; scaled dot products and a softmax give an attention matrix with one entry for every pair of positions; heads are concatenated and mixed. One block, hidden size 128, with 4 heads when compounds are new and 2 otherwise.
Connected does not mean attended to.
A two-layer network maps each residue’s features to a vector that can be compared with an atom’s vector (Eq. 2). In the paper’s configuration that vector has 128 numbers: Section V-E gives the global encoder a hidden size of 128, and Eq. 2 keeps that size.
Each atom gathers its bonded neighbours’ features, weighted by the bond’s features. A virtual supernode, connected to every atom, starts as the sum of all atom features. In every block a gate blends what the atoms say into the supernode and what the supernode says into each atom, and gated recurrent units apply the updates. Four blocks; the atom vectors that come out go to the pairwise module.
Reading the supernode as a shortcut across the molecule is ours. The paper says it is there “to assist the information flow and improve the expressive power”.
The likelihood for one residue and one atom is the dot product of their vectors, passed through a sigmoid that keeps it between 0 and 1 (Eq. 9). The same residue vector gives a different likelihood with each atom. Turn the made-up residue vector and watch six cells change together.
toy Whole numbers out of 100, from two invented vectors of length 2. They are not likelihoods from APIP.
Known interactions are 1. Everything else is unlabeled, which is not the same as confirmed absent. Computing the whole grid is costly, so each iteration samples some unlabeled cells as negatives, more negatives than positives. The loss that Section IV-E defines is a weighted squared error in which a positive counts much more than a sampled negative. A variant, APIP_BCE, uses binary cross-entropy instead. The overview of the paper uses another name for the loss: open the Weighted loss part.
The paper gives no value for the sampling ratio or for the two weights. These sliders are ours.The two counts above use this page’s own choices: 4 negatives per positive, and a weight of 5 on a positive.
Select any part of the drawing to read what it is, what goes in and out, and where the paper describes it. Arrow keys move between parts.
Lit in this step
Section III, page 1742
“We represent each input protein as a sequence P = (p1, p2, ..., pS)”. No structure enters anywhere in this lane.
toy 64 positions; residue 29 is the letter H
Section IV-B-1, page 1743
Every residue of the same kind enters as the same 20 numbers, wherever it sits in the chain.
toy H occurs 3 times in the toy chain, so 3 positions enter as the same 20 numbers
Section IV-B-1, page 1743; sizes in Section V-E, page 1746
Four blocks of two 1D convolution layers followed by a leaky ReLU, kernel size 5, with skip connections between blocks. The paper’s reason: “an inductive bias for adjacent amino acids to function together in motifs”.
within reach positions 13 to 45 for residue 29, by the arithmetic of Figure 3
The paper does not say the stride, dilation or padding of these convolutions, so it never states a receptive field.
Section IV-B, page 1743
“The output of each module is multiplied by this weight and passed to the next module. This allows the model to self-moderate its modules without the need for extra hyper-parameter.”
Section IV-B-2, Eq. 1, page 1743; sizes in Section V-E
Multi-head self-attention: queries, keys and values, scaled dot products, a softmax, heads concatenated and mixed. The attention matrix has an entry for every pair of positions, so one block connects them all. One block, hidden size 128; 4 heads for new compounds, 2 for the other two settings.
within reach all 64 positions, for every residue
The paper does not say whether the block has a positional encoding, layer normalisation or a feed-forward sublayer, and it shows no attention weights.
Section IV-B-3, Eq. 2, page 1743
“A two-layer neural network is used to transform the output of the sequence encoders into compatible feature vectors for the pairwise prediction module.”
Section III, page 1742
Atoms are nodes and chemical bonds are edges. The input is defined by attributes of atoms and bonds; no coordinate, distance or angle appears in any equation, so this page calls it a graph of atoms and bonds.
toy 6 atoms, 6 bonds, no element names; atom a3 is followed
Section IV-C-1, Eq. 3, page 1743
“We denote graph-level representation as a supernode with a virtual connection to all nodes in the graph.” It starts as the sum of all initial atom features and is embedded with a tanh.
within reach every atom, in one step
Section IV-C-2, Eq. 6, page 1743
Each atom joins its own features with the sum of its bonded neighbours’ features, each weighted by the bond’s features, and a learned matrix mixes the result.
within reach atom a3 has 2 bonded neighbours
Section IV-C, Eq. 4 and Eq. 7, pages 1743 to 1744
Each gate is a weighted sum with a coefficient α: one blends a transformed view of the atoms into the supernode, the other blends a transformed view of the supernode into each atom’s message. So every atom hears a summary of the whole molecule in every block.
Section IV-C, Eq. 5 and Eq. 8, pages 1743 to 1744
One recurrent unit updates the supernode from its blend, another updates each atom from its blend. After four blocks it is the “node latent representations” that feed the pairwise module: per-atom vectors, not the supernode.
Section IV-D, Eq. 9, page 1744
The two lanes have not seen each other until here: as the paper describes them, residue vectors are computed without the ligand and atom vectors without the protein. That reading is ours; the paper does not discuss it.
toy with the made-up vectors of step 6, atom a3 gets 20 in 100
Section IV-D, Eq. 9, page 1744
The paper’s reasons: it “can capture nonlinear interactions between two factors”, and it is “a bounded function (constrains the output in the range of [0, 1]) which can improve the convergence speed of the neural network”.
Section III, page 1742
The model “outputs” a matrix that “indicates the likelihood of interaction between residue Pi and atom Lj”. A grid that says where, pair by pair, is how this page reads the paper’s “more interpretable prediction”.
toy 384 cells, 17 of them contacts with the ligand where you left it
Section IV-E, Eq. 10, page 1744
A zero means unlabeled, not a confirmed non-contact. The paper notes that “a small number of unlabeled samples are positive but may be treated as negative”.
The paper does not say what makes a residue and an atom a positive pair. The datasets come from an earlier work it cites.
Section IV-E, page 1744
“Since the interaction matrix is large, the computational cost of the whole matrix” would be “a bottleneck”, so the paper samples. It sets “the number of negative samples more than that of positive ones”, citing earlier work that one negative per positive is not enough.
The paper does not say the ratio it used.
Section IV-E, Eq. 11 and Eq. 12, page 1744; Section V-D and V-E, page 1746
A squared error per cell, times a weight: the paper sets the positive weight “to be much larger” than the negative one “so that a positive sample is as important as many unlabeled samples”. Trained with the Adam optimizer at an initial learning rate of 0.0005. The variant APIP_BCE swaps in binary cross-entropy.
Two names for the loss: the overview (Section IV-A) calls it a weighted binary cross-entropy; Section IV-E defines the squared loss of Eq. 11, and Section V-D makes cross-entropy the variant APIP_BCE. This page follows Sections IV-E and V-D. The paper does not give the two weights.
IllustrationA diagram drawn for this page from Section IV and Equations 1 to 12. Live valuesThe values shown come from the invented toy of Figure 3 and follow the residue you selected there; they are counts and whole numbers out of 100, not results from the trained network.
04 The evidence
The paper has no results table. Every comparison in it is a bar chart without printed values, so the numbers below are the ones printed in the running text of Section VI-A. The metric is average AUC: the area under the ROC curve, computed for each protein-ligand complex and averaged over the test set. Higher is better.
Figure 5Measured, Section VI-A
computed from the printed values On both datasets each printed APIP value is lower than the one before it, in the order new compounds, new proteins, both new: KIKD 0.946, 0.888, 0.856; IC50 0.968, 0.884, 0.858. The paper calls the third setting “a more challenging task” and does not itself compare the three, and Section V-B-2 speaks of “protein AUC results and compound AUC results”, so read these as three sets of printed numbers, not as a measured cost of novelty (question 4).
| Setting and method | KIKD | IC50 |
|---|---|---|
| PISC: Predict Interaction Sites of Compounds, compounds are new APIP | 0.946 | 0.968 |
| PISP: Predict Interaction Sites of Proteins, proteins are new APIP | 0.888 | 0.884 |
| PISP: Predict Interaction Sites of Proteins, proteins are new APIP_BCE | not printed | 0.887 |
| PPI: Predict Pairwise Interactions, both are new APIP | 0.856 | 0.858 |
Underlined: the values the paper’s text calls the best for that setting and dataset. For Predict Pairwise Interactions the text says only that APIP “outperforms the baseline models”. APIP_BCE is the same network trained with binary cross-entropy. PISC and PISP use five-fold cross-validation, Predict Pairwise Interactions nine-fold. No spread over folds is printed.
Figure 6Illustration
How to read it. Rows are proteins, columns are compounds, the hatched part is one test fold. The paper names each setting for what is predicted and defines it by what is held out (Section V-C). PISC splits compounds into five disjoint groups and holds one out in turn; PISP does the same with proteins; Predict Pairwise Interactions splits both into three groups and holds out one of the nine blocks.
Groups come from clustering-based cross-validation at a threshold of 0.3. What the paper leaves open about the groups is in question 4.
IllustrationIllustration of the protocol in Section V-C. The numbers of proteins and compounds drawn are arbitrary. It shows a protocol, not results.
TableFrom the paper, Section V-D
| Name in the paper | What it is | How it was run |
|---|---|---|
| CNN | the method of the paper’s reference [1], convolutional and graph networks | “The source code was directly used” |
| ResNet | ResNet blocks from an earlier work | adapted “in our framework” |
| DCNN | dilated convolution blocks | adapted “in our framework” |
| LSTM | an LSTM sequence encoder; the paper groups it under attention-based methods | re-implemented “in our framework” |
| Transformer | a transformer module with an earlier work’s parameters | adapted “in the proposed framework” |
| APIP_BCE | APIP with binary cross-entropy loss | “the same framework” |
| APIP_C | APIP “without the global sequence encoder” | other parameters unchanged |
| APIP_T | APIP “without motif sequence encoder” | other parameters unchanged |
In words the paper says that LSTM and Transformer “show relatively poor performance” when compounds are new, and that “the attention-based and CNN-based models perform poorly” when proteins are new. All baselines were “either re-implemented based on the published work or re-run from the authors’ provided codes”, so that they “train with the same dataset and features”.
05 Take one half away
The paper describes four variants: the full model, one without each encoder, and one with a different loss. For each removed encoder the paper prints one pair of percentages, for the one setting it singles out with “Especially”.
Figure 7Measured, Section VI-B
Both encoders, the weighted squared loss of Eq. 11. The paper prints six AUC values for it in Section VI-A, from 0.856 to 0.968 (the APIP rows of Figure 5), and a seventh, 0.93, for a reduced training set (Section VI-D).
Source: Section VI-A, Section VI-D
The paper reports that adding the global encoder back “improves the performance by 4.1% for the KIKD dataset and 2.4% for the IC50 dataset” in the Predict Pairwise Interactions setting. Its Figure 2 draws the other two settings as bars.
Source: Section V-D, Section VI-B-1
The paper reports that with the motif encoder, “in the PISC setting, APIP improves the average AUC score by 7.9% on KIKD datasets and 10.9% on the IC50 dataset”. The pair it prints for the other encoder is for a different setting, so the two pairs do not rank the encoders. The other two settings are bars in its Figure 2.
Source: Section V-D, Section VI-B-2
The one printed place a variant is ahead: PISP on IC50, 0.887 against APIP’s 0.884. The paper says APIP “further improves the model in almost all the experimental settings”, and that this variant is ahead of “other comparison methods by up to 12%”, without naming the method, dataset or setting that figure belongs to.
Source: Section V-D, Section VI-A, Section VI-B-3
In Figure 3 above, APIP_C is the setting whose residues see a window, and APIP_T is the setting that starts each position from a single letter, if nothing marks its place in the chain (the toy assumes so; the paper does not say). The toy shows what each can connect; these percentages are what the paper measured, and the toy does not explain them.
All four bars are gains the paper reports for APIP, the full model, over a variant that lacks one encoder. Two groups, two different settings: do not rank one group against the other.
| Encoder put back, setting, dataset | Reported gain |
|---|---|
| Put the global encoder back (APIP against APIP_C), on Predict Pairwise Interactions | |
| Both new, KIKD | 4.1 |
| Both new, IC50 | 2.4 |
| Put the motif encoder back (APIP against APIP_T), on PISC: compounds are new | |
| PISC, KIKD | 7.9 |
| PISC, IC50 | 10.9 |
StripMeasured, Section VI-C
said in the paper’s text a bar in the paper’s Figure 3, no printed value
The paper reports that performance “degrades when the clustering threshold cutoff is too high” and that the “maximum performance degradation is 0.7%”, with the same trend on both datasets and all three settings. The paper also chose its hyperparameters at 0.3 (Section V-E), which is worth knowing when reading that 0.3 comes out best.
StripMeasured, Section VI-D
a value printed in the paper’s text a bar in the paper’s Figure 6, no printed value
KIKD, compounds new. The paper reports that performance “progressively increased with the increase of training dataset size”, that the set of 2500 “has already reached 0.93”, and that this is “only 1.7%” below the set of 5000. It counts the sizes in “samples” (Section VI-D ends on a set size of 2500 “samples”), without defining the word.
The takeaway
The paper proposes: a protein encoder that needs only the sequence and stacks a convolutional motif encoder under one self-attention block; a ligand graph encoder with a virtual supernode and gates; and a pairwise module that scores every residue against every atom.
With no protein structure as input, average AUCs between 0.856 and 0.968 across its three hold-out settings. Its printed ablation gains: 7.9% and 10.9% for putting the motif encoder back on PISC, 4.1% and 2.4% for putting the global encoder back on Predict Pairwise Interactions. They are for different settings, so they do not rank the two encoders.
In its own words: the three modules “jointly tackle the different types of input in different learning paths”, and the study “demonstrates the effectiveness and potential of this hybrid design in representation learning” (Section VII). What it leaves open is in the questions below.
06 Office hours
Written by Tian's agent while Tian slept and Pepper supervised. Pepper accepts no liability either.
Each answer says where in the paper it comes from, so you can check the agent's work.
It is not told, and the paper does not claim that it works the fold out. It learns from examples in which interacting residue and atom pairs are labelled, with an encoder in which any residue can inform any other. The evidence offered is predictive: average AUC on held-out compounds, proteins or both, plus six plotted examples about which the paper writes, “Even though some of the active sites are hundreds of residues away, our model APIP successfully predicted most of the active sites”. Those six are one random pick from each of three AUC bands, [0.7, 0.8), [0.8, 0.9) and [0.9, 1.0], for ligands and for proteins; nothing below 0.7 is shown. The paper shows no attention weights and does not test whether attention follows spatial proximity. The reach test on this page shows what the architecture makes possible, not what the trained model does.
Source: Section IV-E, Section V-B-2, Section VI-A, Section VI-E, Figure 5The paper draws them as bars without values, on axes that do not start at zero. It prints numbers only for APIP and, once, for its cross-entropy variant. The only printed margin over the baselines is that this variant is ahead of “other comparison methods by up to 12%”, without naming the method, dataset or setting. There is one metric, average AUC, with no spread over folds, no seeds and no significance test, and no threshold-based score although the paper itself says that only a few residues interact. So Figure 5 charts the seven values of Section VI-A, and this page gives the rest in the paper’s words.
Source: Figures 1 and 4, Section V-B-1, Section VI-A, Section VI-B-3Not by itself, on the paper’s own account. It reports that the LSTM and Transformer baselines “show relatively poor performance” when compounds are new, and concludes that “simply adapting the attention-based methods with the global receptive field cannot successfully model the interaction sites”. The ablation figures it prints are gains: 7.9% and 10.9% on PISC from putting the convolutional motif encoder back, 4.1% and 2.4% on Predict Pairwise Interactions from putting the attention block back. Caveats: each pair is for the one setting the paper singles out with “Especially”, so the two pairs do not rank the encoders; the paper does not say whether the percentages are relative or absolute, and it prints no AUC for the variants. No other part was removed.
Source: Section VI-A, Section VI-BThree settings hold out compounds, proteins, or both, with five-fold, five-fold and nine-fold cross-validation. Groups are formed by clustering-based cross-validation at a threshold of 0.3, following two earlier works, “to reduce the data redundancy problem”; the paper does not say what is clustered, what the threshold measures, or what happens to pairs that share only a protein or only a compound with a held-out block. Across thresholds of 0.3, 0.4 and 0.5 the paper reports a maximum performance degradation of 0.7%, with 0.3 the best. Hyperparameters were chosen per cross-validation setting at threshold 0.3, and the paper does not describe a separate validation split. The printed values are 0.946 and 0.968 when compounds are new, 0.888 and 0.884 when proteins are new, and 0.856 and 0.858 when both are. The paper does not say whether the three settings score the same quantity (Section V-B-2 speaks of “protein AUC results and compound AUC results”), so these are three sets of printed numbers, not a measured cost of novelty.
Source: Section V-B-2, Section V-C, Section V-E, Section VI-A, Section VI-CDataset sizes and the rule that makes a pair a positive: both are left to the work the datasets come from. Any runtime, memory or parameter count. A limitations section, failure cases, or examples below an AUC of 0.7. A code or data link. A comparison with docking or any structure-based predictor, although the motivation is the absence of structures. Binding affinity. Several details a second implementation would need: the atom and bond features, the hidden size of the graph module, how the coefficient of the gates is computed, the number of epochs and the batch size. Future work is one sentence: the authors believe the framework “can also be applied to other drug discovery-related tasks”.
Source: Section V-A, Section IV-E, Section V-B-2, Section VII