Geometric deep learning IEEE BigData 2024
Same dots, different wires. Two readers out of three cannot tell.
A spatial network has wiring and it has shape. The paper builds a neural network, SVNNet, that reads both at once, walking the wires one, two and three steps out from every node, and is built so that turning the whole thing does not change its answer.
The paper reports the best value in every column of its three results tables. Section 04 shows how large the margins are.
Figure 1Illustration
Network A: one ring
Network B: two triangles
Flip to network B and watch the three readers. No dot moves: only the wires do.
Six invented points. The paper says that approaches using only spatial or only network data cannot distinguish certain graphs, and gives no example. This pair is ours. The two wirings are different graphs; “Wires only” is the colour test the paper cites, which cannot separate them. Toy numbers, not results.
01 The problem
A spatial network is a graph whose nodes sit at positions in space: it has wiring and it has shape. The paper’s starting point is that most proposed approaches either use only one of the two or are tied to one kind of input. Here are three invented pairs, and five readers that each see a little more. Pick a pair, then look through each reader.
Figure 2Illustration
Same: every position, and how the wiring looks from every dot (its Weisfeiler-Lehman colour). Differs: the wire lengths.
Same: the wiring and every wire length. Differs: the angles between steps.
Same: the wiring, every wire length and every angle. Differs: the twist.
| Reader | Pair A | Pair B | Pair C |
|---|---|---|---|
| 1. Dots onlypositions, no wires | identical | different | different |
| 2. Wires onlywiring, no positions | identical | identical | identical |
| 3. Wires and lengthsone step, Eq. (1) | different | identical | identical |
| 4. Plus anglestwo steps, Eq. (2) | already apart | different | identical |
| 5. Plus twiststhree steps, Eq. (3) | already apart | already apart | different |
Read across. Dots only misses pair A. Wires only misses all three. The paper’s three rungs of geometry, read along the wires, separate all three: lengths catch A, angles catch B, twists catch C. In the paper’s words, “the first-order neighbors are not sufficient to model the complete geometric graph”, and third-order neighbours “provide a unique 3D property by the two planes formed from the four nodes”.
IllustrationThree invented pairs and five fingerprints computed in your browser, in toy units. The readers are ours, not the paper’s baselines: in the paper’s experiments every model received both coordinates and connectivity (Section V). “Wires only” is the Weisfeiler-Lehman colour test the paper cites: two nodes get the same colour when the wiring around them looks the same. The two wirings of pair A are different graphs, and identical only as far as that test can see. The angle is the one Equation (2) prints, between the two steps of a path (0° for a straight line). The sign of a twist is the toy’s convention. These numbers are not results.
Dots without wires
“Existing works on point clouds don't consider the network connections in spatial networks.”Section I, page 1883
Wires without dots
“the regular GNNs are at most as powerful as the Weisfeiler-Lehman Isomorphism Test”Section I, page 1884
Every dot wired to every dot
“such networks cannot handle the spatial networks, where both spatial data and network connectivity matter”Section II-C, page 1884, on point-cloud and equivariant models that treat the graph as complete
The paper’s motivation names molecules, brain networks, and mobility and transportation networks. Every real-world experiment in the paper is a molecular dataset.
02 The idea
“we need a model that preserves the relative rotational information for both geometric and network features but is invariant to global rotations.”Section I, page 1884
SVNNet listens along the wires. Every node hears paths of one, two and three wires, each written down twice: as numbers that ignore rotation and as arrows that turn. A small unit the paper calls the Spatial Vector Neuron updates the two together. Below is a seven-node toy with node 0 listening. Change its wiring first, then its shape, and only then turn it.
Figure 3Illustration
IllustrationA hand-built toy with seven invented nodes, one layer, and fixed numbers in place of learned weights. It follows the form of the paper’s Equations (1) to (6) and (10). It is not the trained network and its numbers are not results.
What just happened
Click a wire in the drawing, or press it here. Solid wires can be cut, dashed ones added.
This toy has one layer, so you can see what each step adds by itself. The paper searches over 4, 8 or 16 layers (Table V) and does not report which it used. The paper’s formula for the twist contains a function it does not define; the sign convention here is the toy’s.
Rule intact: every weight acts on channels, never on x, y and z separately, and every input is a difference between two nodes.
Both broken variants are ours. The paper states the two rules and does not test breaking them.
With scripts on, this figure becomes a bench: cut and add wires, twist the tail, slide a node, turn the network, and read what node 0 reports.
03 Follow one three-step path
Follow the one three-step path of the toy, from node 3 through nodes 2 and 1 into node 0, through a layer. The live values are the ones you left on the bench. Every box opens.
Figure 4Illustration
single line: numbers, stays put double line: arrows, turns
From the listening node, the network follows wires out to first, second and third-order neighbours. Every such path becomes a message. The paper calls them edges, triplets and quadruplets.
One wire gives a length and a unit direction. Two wires add the angle between the two steps. Three wires add a dihedral angle, from the two planes the four nodes form. Lengths and angles go to the numbers lane; directions go to the arrows lane.
The scalar path of the Spatial Vector Neuron: a linear map of the numbers, a linear map of the arrows’ lengths, the two joined, mapped back and passed through ReLU. A length does not change when the network turns. This is the first crossing.
The vector path: a linear mix across channels (the same weight acts on x, y and z); a sigmoid gate computed from the numbers multiplies a second mix of the arrows, the paper’s “scaled gating”; the two sets are joined and mixed again. A gate rescales an arrow; it cannot bend it. This is the second crossing.
The paper does not say which function the final non-linearity on arrows is; the toy leaves it out.
Messages of the same order are combined without regard to order, the three results are concatenated, and another neuron fuses them into the node’s new numbers and arrows. The paper adds a skip connection, layer normalisation on the numbers, and a stack of two SVN layers that widens by 4 and narrows again.
The paper writes this layer twice, in Equations (11) to (12) and (14) to (15), with small differences; the diagram shows what both share. The toy takes a mean and leaves out the skip, the normalisation and the stack.
The invariant block replaces each arrow by its three inner products with three reference arrows, which are learned mixtures of the same arrows. Inner products do not change when everything turns together. The paper’s reason: predictions “are invariant to the pose of the input”. This is the third crossing, and the arrows lane ends here.
Each layer’s node features pass through their own output block: invariant block, pooling over all nodes, a small MLP. The outputs of all layers, including the initial one, are added, and a final MLP makes the prediction.
The toy has one layer, takes a mean over its seven nodes and has no MLP.
The three crossings, in one line. Our reading of Equations (4), (5) and (10): arrows influence numbers only through lengths and inner products, and numbers influence arrows only by rescaling them. Nothing else crosses.
Select any box in the drawing to read what it does, where the paper describes it, and its live toy value.
First-order neighbours, an edge: λij. Section III, Eq. (1), page 1885
stays put
The difference of two positions, d = ci − cj, and its length r. Turning the whole network turns the difference and leaves its length alone.
A slip in the paper. The prose writes rji and the equation rij; this page follows the equation.
Second-order neighbours, a triplet: λijk. Section III, Eq. (2), page 1885
stays put
The arctangent of the cross product’s length over the dot product of the two steps: the angle between the two steps, 0° for a straight line. The three nodes must be distinct.
Third-order neighbours, a quadruplet: λijkp. Section III, Eq. (3), page 1885
stays put
The paper: “the angle between the normal directions of the planes” that the four nodes form. It is the only one of the three that needs a third wire, and the only one the twist slider changes.
Not in the paper. The formula contains a function s( ) that the paper does not define. The toy reads it as a sign; that convention is ours.
The embeddings of the nodes on the path. Section IV-A, Eq. (6); Section IV-C, page 1887
numbers stay put arrows turn
In the paper a message is built from the embeddings of the nodes on the path together with the path’s geometry, and node features and their types are embedded as the network’s first step. The toy has no node features: its messages are built from geometry alone.
Not in the paper. What the node and edge types are, and how they are embedded, is not described.
Direction vectors Dij. Section III, Eq. (1), page 1885
turns
Each wire’s difference divided by its length. In the paper’s words, “The direction vector provides the relative direction of the neighbor.” These are the arrows that enter the lower lane.
Scalar transformation Ws. Section IV-A.1, Eq. (4), page 1885
stays put
A linear map of the numbers. They were built from relative geometry, so the paper calls them “roto- and translation invariant”: a turn of the network does nothing to them.
Row-wise L2 norm, then Wrdc. Section IV-A.1, Eq. (4); proof of Theorem 1, page 1886
crosses between the lanes
Every arrow is reduced to its length, and the lengths pass through a linear map into the numbers lane. The proof rests on one line: the length of a turned arrow equals the length of the arrow.
Wsf and the non-linearity σs. Section IV-A.1, Eq. (4), page 1885
stays put
The mapped numbers and the mapped lengths are concatenated, mapped back to the width of the numbers and passed through ReLU. Everything that entered was already immune to a turn, so the result is too.
The “vector perceptron” Wv. Section IV-A.2, Eq. (5), pages 1885 to 1886
turns
A linear mix across channels. The proof: “the same parameters are applied to all columns”, so x, y and z are treated alike and a turn passes straight through the map.
Wvsc. Section IV-A.2, Eq. (5), pages 1885 to 1886
turns
A second channel mix of the same arrows. Its output is what the gate rescales. Like the first mix, it never treats x, y and z differently.
“Scaled gating”: a sigmoid of Wssc s. Section IV-A.2, Eq. (5); proof of Theorem 1, page 1886
crosses between the lanes
The numbers pass through a linear map and a sigmoid, and each result multiplies one arrow. A number times an arrow is still an arrow that turns with the network: the gate rescales it and cannot bend it.
Row-wise concatenation, Wvf, then σv. Section IV-A.2, Eq. (5), second line, page 1886
turns
The two sets of arrows are concatenated, mixed once more across channels and passed through a final non-linearity.
Not in the paper. The paper never says which function σv is; its proof asserts that it is equivariant. The toy leaves it out.
Aggr, a permutation invariant aggregation. Section IV-A, Eq. (6), page 1886
numbers stay put arrows turn
Messages of the same order that end at the same node are combined “using permutation invariant aggregation functions”, so the order in which nodes are listed does not matter.
Not in the paper. Table V searches MEAN and SUM; which one the reported models use is not stated. The toy takes a mean.
Edges E, triplets T and quadruplets Q. Section IV-A, Eq. (6); Eq. (14), page 1887
numbers stay put arrows turn
The paper: “we row-wise concatenate the output of n-order message passing functions”. On the bench, setting Hops heard to 2 steps empties the third of these.
SVN applied to the concatenated messages. Section IV-C, Eq. (11); proof of Theorem 2, Eq. (14), page 1887
numbers stay put arrows turn
Another Spatial Vector Neuron, with the same two lanes and the same two crossings inside it, maps the concatenated messages to the node’s new numbers and arrows.
Add the input; LayerNorm on scalars. Section IV-C, Eq. (11); proof of Theorem 2, Eq. (14) and (15), page 1887
numbers stay put arrows turn
The paper: “We use a skip connection to add the input to the output.” Its proof lists “layer normalization on scalars” among the operations that keep the symmetry.
Two accounts in the paper. The two printed accounts of the layer place the normalisation differently. The toy has neither.
q, a stack of two SVN layers. Section IV-C, Eq. (12), page 1887
numbers stay put arrows turn
The paper: “the first layer scales up the representation with a scale of 4, and the second layer scales it back to the original size.” The result goes both to the next layer and to this layer’s own output block.
Not in the toy. The toy has no such stack.
Vinv = V (Winv V)T. Section IV-B, Eq. (10), pages 1886 to 1887; proof of Theorem 2, page 1888
crosses between the lanes
Each arrow is replaced by its inner products with three reference arrows, which are learned mixtures of the same arrows. Inner products do not change when everything turns together. The paper says a full product of two C-channel features “takes quadratic time complexity”, and that reducing the reference to 3 by 3 “effectively reduces the complexity to O(n)”.
Not in the paper. The paper does not say what n is in that remark, and reports no measurement of cost.
Wfu. Proof of Theorem 2, “Graph Output Block”, page 1888
stays put
The network combines the numbers with the invariant block’s output through Wfu into a single representation. From here on there are no arrows left to turn.
Graph pooling to a “virtual node”; a multi-layer perceptron. Section IV-B, page 1887; proof of Theorem 2, page 1888
stays put
Node features are aggregated to one virtual node and a multi-layer perceptron produces the block’s output. The proof: “At this point, the representation is invariant under all R.”
Not in the paper. The proof lists MEAN, SUM, MAX and MIN; Table V searches MEAN and SUM; the choice is not reported. The toy takes a mean and has no MLP.
Gout, the sum of every layer’s graph output. Section IV-C, Eq. (13), page 1887; proof of Theorem 2, page 1888
stays put
Every version of the node representation, “including the initial node representation”, passes through its own graph output block. The outputs are added, and a final multi-layer perceptron makes the prediction.
Not in the paper. How many layers the reported models have is not stated: Table V searches 4, 8 and 16. The toy has one.
IllustrationA diagram drawn for this page from Equations (1) to (15). The paper has no architecture figure. Live values come from the toy layer of Figure 3, not the trained network, and are not results.
04 The evidence
Four synthetic targets and nine molecular datasets, ten models each. Every value below is a printed cell of Tables I to III. Which cell is best, every gap and every count is computed by this page from those cells. So is the reading of the ± ranges (a printed value, minus and plus its ±): it is this page’s, not the paper’s, and the paper reports no significance test.
Figure 5Measured, Tables I to III
SVNNet prints the best value in 13 of 13 columns. Against the best baseline in each column, the gap is larger than the two printed spreads together in 8 (τ, υ, ρ, ToxCast, Tox21, HIV, MUV, BBBP) and smaller in 5 (Ω, LIPO, ESOL, ClinTox, BACE). This is a plain reading of the printed ± values, not a statistical test; the paper reports none.
The runner-up changes from column to column: GatedGNN twice, SGMP three times, DimeNet four times, PointNet twice, SGCN once, PPFNet once.
Point at the chart, tap it, or focus it and use the arrow keys to read exact values.
± as printed. Table II’s caption says three runs with standard deviation. Tables I and III do not say what ± is. All ten models received the same 3D coordinates and connectivity; for molecules the 3D positions were generated with RDKit (Section V).
Measured, Tables I to IIIEvery cell as printed, in the paper’s row order. Bold and underlined: what the paper prints in bold, which is the whole SVNNet row. Computed by this page: that row is also the best cell in each of the 13 task columns.
| Methods | τ | υ | ρ | Ω | LIPO | ESOL |
|---|---|---|---|---|---|---|
| GIN | 0.162±0.003 | 0.881±0.080 | 1.010±0.080 | 1.863±0.221 | 1.132±0.024 | 0.661±0.051 |
| GAT | 0.165±0.009 | 1.058±0.006 | 1.153±0.136 | 1.992±0.109 | 1.131±0.015 | 0.725±0.107 |
| GatedGNN | 0.063±0.005 | 0.427±0.005 | 0.565±0.005 | 1.699±0.102 | 0.822±0.027 | 0.621±0.025 |
| PointNet | 0.152±0.001 | 0.665±0.044 | 0.810±0.037 | 1.642±0.089 | 0.863±0.013 | 0.588±0.034 |
| PPFNet | 0.133±0.001 | 0.652±0.025 | 0.783±0.024 | 1.771±0.025 | 0.750±0.025 | 0.696±0.044 |
| SGCN | 0.134±0.000 | 1.173±0.021 | 0.905±0.017 | 1.691±0.000 | 0.885±0.047 | 0.817±0.062 |
| SchNet | 0.158±0.000 | 0.903±0.021 | 1.045±0.059 | 1.894±0.052 | 0.901±0.108 | 0.753±0.020 |
| DimeNet | 0.103±0.027 | 1.266±0.147 | 0.556±0.094 | 1.412±0.059 | 0.613±0.013 | 0.730±0.047 |
| SGMP | 0.068±0.005 | 0.748±0.168 | 0.450±0.046 | 1.332±0.031 | 0.941±0.180 | 0.851±0.033 |
| SVNNet | 0.046±0.002 | 0.343±0.021 | 0.204±0.019 | 1.265±0.058 | 0.539±0.081 | 0.552±0.011 |
| Dataset | ToxCast | Tox21 | ClinTox | HIV | MUV | Avg. |
|---|---|---|---|---|---|---|
| # Graphs | 8,575 | 7,831 | 1,478 | 41,127 | 93,087 | |
| # Nodes | 18.78±11.58 | 18.57±11.35 | 26.16±15.61 | 25.51±12.11 | 24.23±5.05 | |
| # Edges | 38.52±25.27 | 38.59±25.05 | 55.77±33.7 | 54.94±26.43 | 52.56±11.53 | |
| # Labels | 617 | 12 | 2 | 1 | 17 | |
| GIN | 36.34±0.12 | 50.98±3.91 | 92.92±0.50 | 68.78±4.25 | 53.85±0.49 | 60.58 |
| GAT | 39.26±0.03 | 43.69±0.32 | 93.17±0.09 | 69.97±5.79 | 49.92±0.54 | 59.20 |
| GatedGNN | 46.67±0.00 | 45.07±0.02 | 93.52±0.08 | 71.40±1.20 | 47.79±0.34 | 60.89 |
| PointNet | 37.64±0.10 | 44.69±0.93 | 93.75±0.15 | 71.74±1.32 | 53.40±0.33 | 60.25 |
| PPFNet | 36.94±0.21 | 56.51±0.25 | 95.90±0.85 | 68.39±2.37 | 43.67±0.18 | 60.28 |
| SGCN | 65.03±0.14 | 68.55±0.00 | 92.26±0.91 | 63.16±1.63 | 52.67±0.00 | 59.13 |
| SchNet | 52.61±0.02 | 38.50±0.10 | 94.70±0.54 | 71.72±1.93 | 42.43±0.00 | 57.39 |
| DimeNet | 61.49±0.06 | 70.65±0.24 | 95.43±0.33 | 61.51±0.48 | 53.87±0.00 | 60.99 |
| SGMP | 45.23±0.08 | 43.39±1.62 | 94.01±0.57 | 55.65±7.98 | 40.65±0.00 | 55.78 |
| SVNNet | 78.08±0.12 | 79.03±1.02 | 96.13±0.15 | 77.28±0.11 | 68.06±1.21 | 79.72 |
Printed as in the paper. For SGCN, SchNet and DimeNet the printed average differs from the mean of the row’s five cells (68.33, 59.99 and 68.59, computed by this page). This page does not chart the “Avg.” column.
| Dataset | BBBP | BACE | Avg. |
|---|---|---|---|
| # Graphs | 2039 | 1513 | |
| # Avg. Nodes | 24.06±10.59 | 34.09±8.52 | |
| # Avg. Edges | 51.91±23.43 | 73.72±17.53 | |
| GIN | 82.11±1.13 | 68.20±3.74 | 75.16 |
| GAT | 82.28±1.49 | 64.69±3.80 | 73.49 |
| GatedGNN | 85.85±1.47 | 66.23±8.96 | 76.04 |
| PointNet | 80.49±2.13 | 72.59±3.74 | 76.54 |
| PPFNet | 86.67±1.49 | 78.95±3.48 | 82.81 |
| SGCN | 83.58±2.94 | 65.35±3.63 | 74.46 |
| SchNet | 82.76±1.13 | 80.92±0.66 | 81.84 |
| DimeNet | 85.20±1.23 | 82.02±0.38 | 83.61 |
| SGMP | 86.83±1.29 | 79.61±1.74 | 83.22 |
| SVNNet | 89.52±0.91 | 83.52±2.21 | 86.52 |
05 Which part matters
The paper removes one part at a time and reports the error on LIPO. Table IV prints one number per row and no spreads, so the shaded band is borrowed from Table I, where the full model on the same task is printed as 0.539±0.081.
Figure 6Measured, Table IV
Pick a row above, or focus the chart and use the arrow keys.
The largest change is removing the invariant block: 0.615 against 0.539 for the full model, a difference of 0.076. Table I prints the full model’s spread on this task as ±0.081.
How to read it. One task, one number per row, no spreads. Root mean square error, lower is better. Every bar ends inside the band from 0.458 to 0.620. The paper ablates no other dataset, and has no row that removes the two-step or three-step messages, so it does not measure what the angles and twists contribute. It does not define the rows “- Vector Update” and “- Vector Scale”; the drawing points at the nearest operation the paper names.
| Removed | Group | LIPO |
|---|---|---|
| - sFeature | Feature | 0.551 |
| - VFeature | Feature | 0.548 |
| - InvariantArchitecture | Architecture | 0.615 |
| - Vector UpdateArchitecture | Architecture | 0.562 |
| - Vector ScaleArchitecture | Architecture | 0.565 |
| SVNNetnothing removed | nothing removed | 0.539 |
The takeaway
Features read along paths of one, two and three wires, written as numbers and as arrows; a neuron, the Spatial Vector Neuron, that updates the two in separate lanes; and an invariant block that turns arrows into numbers before anything is predicted.
The best printed value in all thirteen columns of Tables I to III, and an ablation on one task. Read by this page: in five of those columns the two printed ± ranges overlap.
No timing, memory or path counts, no test of the neuron inside another network, no variation of depth, and no real-world spatial network that is not a molecule.
06 Office hours
Written by Tian's agent while Tian slept and Pepper supervised. Pepper accepts no liability either.
Each answer says where in the paper it comes from, so you can check the agent's work.
Yes on the first: the paper says every benchmark used “the 3D coordinates and the graph connectivity representations as the node and edge features”, following earlier work, so the graph neural network baselines were not blind to positions. The paper groups the nine baselines as three graph neural network models and six spatial network models, and says it leaves out domain-specific models, although its own descriptions call SchNet and DimeNet domain-specific. On the second: native 3D structures were not available, so conformations were generated with RDKit and the same ones were given to every model. Baseline tuning is not described. The readers in section 01 of this page are our own fingerprints, not these baselines.
Source: Section V, page 1888Best in all 13, computed from the cells. In 8 the gap to the next-best model is larger than the two printed spreads together, so the two printed ± ranges stay apart; in 5 it is smaller and the two ranges overlap (Ω, LIPO, ESOL, ClinTox, BACE). In 3 of those 5 (LIPO, ClinTox, BACE) the gap is also smaller than one of the two spreads alone: on LIPO the gap to the next lowest row, DimeNet, is 0.074 and the full model’s own printed spread is 0.081. Only Table II says what ± means (three runs); no significance test is reported. This page shows cells and gaps and does not restate the percentages in the paper’s running text, some of which depend on which baseline is taken as the reference. Sixteen of the 45 baseline AUC-ROC cells in Table II are below 50 (on ToxCast, Tox21 and MUV); the paper prints them and does not discuss them.
Source: Tables I, II and III, pages 1887 and 1889; counts and gaps computed by this pageThe paper ablates one task, LIPO. Removing the invariant block moves RMSE from 0.539 to 0.615, the largest change; removing the vector update or the vector scale gives 0.562 and 0.565; removing the scalar or the vector features gives 0.551 and 0.548. Table IV prints no spreads, and every one of these values lies inside the ±0.081 that Table I prints for the full model on the same task. No row removes the two-step or three-step messages, so the paper does not measure what the angles and twists contribute, and it does not define exactly what “Vector Update” and “Vector Scale” remove.
Source: Table IV and Section V-D, page 1889; Table I, page 1887Proved by argument, not measured. Theorem 1 covers the neuron and Theorem 2 the network, for rotations in SO(3); translation is handled by building every input from differences between nodes, and node order by order-free aggregation. The proofs rest on weights acting on channels and never on x, y and z separately. The paper does not name the final non-linearity applied to vectors and asserts that it is equivariant. It reports no numerical equivariance test, and it says nothing about reflections. The bench on this page observes the proved property on a toy; it has no mirror control for that reason.
Source: Theorems 1 and 2 and their proofs, pages 1886 to 1888; Section IV-A.2The paper reports no training time, memory, parameter count or path count for any model; its one complexity remark concerns the invariant block. Experiments ran on one NVIDIA A100 with 40 GB. Not tested: the neuron inside any other network (the “drop-in” claim is a design claim), any variation of depth, any vector-valued target, and any real-world spatial network that is not a molecule; brain, social and transport networks are motivation only. The size of the synthetic dataset and the chosen hyper-parameters are not printed. The paper has no limitations or future-work section; these limits are read from what the experiments cover.
Source: Section IV-B, pages 1886 to 1887; Section V and Table V, pages 1888 to 1890