Skip to the content
At Tian’s.The Studio
Back to the roomThe roomBack to the indexThe index

Geometric deep learning NeurIPS 2023

A distance says how far. Give atom 1 the direction too.

SaVeNet: A Scalable Vector Network for Enhanced Molecular Representation Learning

  1. Sarp Aykent
  2. Tian Xia

Affiliation at the time of publication Both at Auburn University

SaVeNet reads only one hop neighbours, describes each pair with a distance and direction vectors, and keeps every operation on those vectors safe under rotation.

On QM9 the paper reports that its base model edges past Equiformer, the baseline with the lowest std. MAE in its table (0.69 against 0.70), while one training step takes 23.1 ms instead of 337.3 ms (Table 3).

Read the paper (PDF) (opens in a new tab) Try the test bench At NeurIPS (opens in a new tab)

Figure 1Illustration

1r2

Distance only. Every dot is at distance r from atom 1: a sphere of places fits.

1d2

Distance and direction. One place is left. With scripts on, this figure turns and switches under your hand.

Distance r1.351toy unitsstays the same
Direction dwithheldturns with the molecule
Positions that fita sphereevery dot is at distance r
Atom 1 is told
28°
Lemma 1 of the paper, on invented coordinates: a distance alone leaves the neighbour anywhere on a sphere, and the direction d leaves exactly one place. d is drawn as Eq. 1 prints it, from atom 2 toward atom 1, so atom 1 finds its neighbour by stepping back along d by r. An edge here joins two neighbouring atoms: the paper links atoms within a radius (Table 1 counts edges at 5.0 Å, 280.72 per QM9 molecule on average), not only chemically bonded ones.

01 The problem

Accuracy on molecules has been bought with extra hops and heavy algebra.

The paper starts from a trade-off between expressiveness and efficiency. To gain expressiveness, earlier networks look past one hop neighbours (the paper says they do so to model angles) or use complex embeddings, and the paper says both compromise scalability because of how much they compute. Count what looking further has to visit.

Figure 2Illustration

Add atoms and count what each extra hop has to visit

One hop pairs of neighbours306what SaVeNet reads
Two hops chains of three atoms4,89616 for every pair
Three hops chains of four atoms73,440240 for every pair
  1. Go to 29 atoms, the largest molecule in QM9. The third bar passes half a million.
  2. Switch to a cutoff. Every count drops, and at 18 or 29 atoms each extra hop still multiplies the work.
  3. Show another example. The drawing shows one pair, one chain of three and one chain of four; the bars count every one of them.

Counts shown for 18 atoms that are all neighbours of each other. With scripts on, a slider changes the number of atoms.

18
Who is a neighbour

IllustrationExact counts of pairs and chains in a toy point cloud, enumerated by this page. They are not a runtime measurement and they do not describe any named model.

How to read it. The ringed atom starts one worked example of each kind: a pair (solid), a chain of three (dashed) and a chain of four (dotted). The bars count all of them, on one log axis. 18 atoms is about an average QM9 molecule and 29 is the largest (Table 1 lists 18.02 atoms on average). A cutoff lowers every count; from about a dozen atoms up the order between them stays. Above 24 atoms only the ringed atom’s own edges are drawn, inside an outline of the cloud.

Table 8From the paper

Read the paper’s own complexity table

Table 8 (Appendix E, page 16). Complexity analysis, as printed (this table spells the first model Dimenet++).
ComplexityDimenet++SphereNetComENetSaVeNet
Initializationn3n4n2n2
Encoderle n3 d2le n3 d2le n2 d2le n2 d2
Decoderld n d2ld n d2ld n d2ld n d2
Overalln3 d2n3 d2n2 d2n2 d2

The paper does not define the symbols. Here n reads as the number of atoms and d as the hidden width, which is our reading. SaVeNet has the same order as ComENet in every row, so the gap in theory is to DimeNet++ and SphereNet. Equiformer is not in this table.

Figure 3Measured, Table 3

Read the measured time of one training step on QM9

With scripts off, read the Latency column of Table 3 in the evidence section: from 23.1 ms for SaVeNet-B to 337.3 ms for Equiformer.

Latency is one forward and backward training pass on one RTX 3090, each baseline with its authors’ hyperparameters. About networks such as SEGNN and Equiformer, which use Clebsch-Gordan tensor products, the paper writes that these products “are empirically found to be significantly slower than the previously discussed methods” (Section 2).

02 The licence to stay at one hop

One hop is enough to rebuild the whole shape.

SaVeNet reads only one hop neighbour pairs. For a connected graph (Theorem 1 assumes a strongly connected one), the paper’s argument that nothing geometric is lost is a construction: walk the edges, and at each one use its distance r and its direction d to drop the next atom into place (Lemma 1 and Theorem 1). Do the walk yourself.

Figure 4Illustration

Place the atoms one edge at a time

The originalRebuilt from one hop facts3154231542
EdgeDistance rDirection d
2 from 11.351(−0.90, −0.35, −0.25)
3 from 11.345(0.70, −0.60, 0.39)
4 from 11.407(0.30, 0.83, −0.47)
5 from 21.570(−0.57, 0.57, 0.59)
Atoms placed5of 5
Worst misplacement0toy unitsafter one translation
Each edge walked gives

IllustrationThe construction from the proof of Theorem 1, run on invented coordinates. It is a statement about the input encoding, not about what a trained network learns. The second switch withholds one thing only: the direction of each edge that is walked.

How to read it. Atom 1 starts at the origin, as in the proof. Each press reads one row and sets the new atom at the known atom’s position minus r times d, which is Eq. 1 solved for the neighbour. The theorem’s last clause is why the copy sits somewhere else: “the translation transformation is required for exact alignment with the original graph”.

03 The idea

Treat an arrow like a number and the symmetry breaks.

Next to its numbers, every atom in SaVeNet carries a small bundle of arrows. The rule is that arrows may be stretched, mixed and added, never bent, so that turning the molecule first or turning the answer afterwards gives the same result. An ordinary activation function bends them. Turn the molecule and watch.

Figure 5Illustration

Turn the molecule, then change how the layer activates its arrows

The layer activates its arrows

ReLU(x), ReLU(y), ReLU(z) what works for numbers

  • turn, then compute
  • compute, then turn
  • prediction (dial)
  • ring: full 3D length
40°
20°
Arrow gap0toy unitslargest distance between an arrow and its ghost
Scalar prediction0toy unitssame molecule, only turned
Vector prediction0off its ghostit should turn rigidlyrim 5

Scalar prediction at every yaw angleshould stay the same

  1. Drag the molecule. One coordinate at a time, the solid arrows leave their dashed ghosts and the line above wanders.
  2. Switch to by length. Every ghost disappears under its arrow and the line goes flat.
  3. Raise direction noise γ, further down. In this toy a gap opens again. The paper schedules γ to reach zero before training ends, so the noise has no effect at inference (Appendix C).

Look inside the activation

What the activation keeps of each arrow

  • channel 1
  • channel 2
  • channel 3
  • channel 4

0.0
0.00
51423vector predictionrim 4.5 toy units

One coordinate at a time. After a turn of 40 degrees the solid arrows (turn, then compute) have left their dashed ghosts (compute, then turn).

51423vector predictionrim 1.25 toy units

By length (Eq. 3). After the same turn every solid arrow lies exactly on its ghost, so only one of each pair can be seen. With scripts on, this figure turns under your hand.

IllustrationTwo hand-built layers with fixed random weights that were never trained, on an invented five-atom molecule. They follow the form of Equations 1 to 6 but are not the trained network. Every number here is computed live from this toy, in toy units. The gate function, the distance basis and one starting direction per channel are our choices; the paper does not specify them.

How to read it. For every pose the page runs the toy twice: on the turned molecule (solid arrows) and on the original with the output turned afterwards (dashed ghosts). Equation 3 of the paper, VA(V) = V ⊙ σ(‖V‖ + b), looks only at an arrow’s length, so its direction survives. The paper’s warning is that activations that work “on each element of the vector independently, can break the model's equivariance” (Section 4.1). It also reports that vector activations “significantly enhance the numerical stability during training”; this toy cannot show that, or accuracy, or speed. Arrows are drawn at eight tenths of their length. The dial prints the value of its rim, and the ring around a prediction is its full length in 3D, because an arrow that points toward you looks short.

04 Follow one message

In every layer, numbers set the length of arrows, and arrows report their length back.

Follow one message through a SaVeNet layer: atom 2 of the toy molecule sends to atom 1. Two lanes run side by side, one for numbers and one for arrows, and they only ever meet in ways that a rotation cannot disturb. Every box opens.

Figure 6Illustration

Step through one message from atom 2 to atom 1

One SaVeNet layer drawn as two lanes. The numbers lane on top carries scalars, the arrows lane below carries vectors. An edge feeds both, messages are summed at atom 1, then the two lanes exchange lengths and gates. Numbers lane / scalars s / stay the same when the molecule turns Arrows lane / vectors V / turn with the molecule r d, t, o rescale only atom 1’s own numbersatom 1’s own arrows gate new numbersnew arrows 1 2 One edge one length r, three arrows Atom 2’s numbersfrom its atom type Distance filterbumps and a cutoff Number messagenumbers × filter Fresh arrowsmixed from d, t, o Carried arrowsatom 2’s own, rescaled Atom 2’s arrowsstart: Eq. 2 sum overneighbours MixV′ = W V Measure lengtha length never turns Numbers updatelengths join numbers ScaleV ⊙ gate Activateby length, Eq. 3

  1. Each edge gives one length and three arrows

    Atom 2 is a neighbour of atom 1. From the two positions the layer takes the distance r and three arrows: d, the unit direction along the edge; t, the cross product of the two positions measured from the centre of the molecule; and o, the cross product of d and t (Eq. 1). Only one hop neighbours are read. Theorem 1 is stated for (r, d) alone; Eq. 1 adds two more arrows, t and o, built from the same positions.

    r
    d
    t
    o

  2. Atoms do not start with empty arrows

    Numbers start from the atom’s type: the paper’s invariant inputs are atomic numbers and interatomic distances (in this toy, a lookup of four numbers). For arrows, the paper notes that earlier models often start at null, which “can unnecessarily consume computational resources, particularly in the network's initial layers”. SaVeNet gives each atom type a starting arrow on a sphere: two learnable angles set where it points and γ sets its length (Eq. 2). γ is scheduled to reach zero before training ends.

    0.00
    numbers
    arrows

  3. Distance becomes a filter

    The distance passes through radial basis functions and a cutoff, then a small perceptron: η(r) = φr(χ(r) ∗ ω(r)). Three filters leave, one for each kind of message. The paper does not say which basis or cutoff it uses; this toy uses six Gaussian bumps and a cosine cutoff.

    bumps
    ηs

  4. The number message: what atom 2 knows, filtered by distance

    es = φs(sj) ⊙ ηs(r). Everything in it is built from an atom type and a distance, so it does not change when the molecule turns (Eq. 5).

    φs(sj)
    message

  5. The arrow message has two sources, and both are only ever rescaled

    Fresh arrows are mixed from d, t and o with the same weights for x, y and z. Carried arrows are atom 2’s existing ones. Each is multiplied by numbers that come from atom 2’s scalars and from the distance: a number can lengthen, shorten or flip an arrow, but it cannot bend it (Eq. 5). In the drawing each channel has its own scale.

    fresh
    carried

  6. Atom 1 adds up its neighbours and keeps what it had

    Each atom keeps its own numbers and arrows and adds the sum of its neighbours’ messages, once per lane (Eq. 4). The proof leans on the fact that “the addition of equivariant vectors preserves equivariance”.

    numbers
    arrow lengths

  7. Arrows report their length to the numbers; numbers set the length of arrows

    In the atom-wise block the arrow channels are mixed, their lengths join the numbers, and a perceptron of lengths and numbers hands back one gate per arrow, followed by the vector activation (Eq. 6). The paper calls this “bi-directional communication between different representations”.

    lengths
    gates
    after VA

Select any box in the drawing to read what it does. Arrow keys move between boxes.

One edge

Eq. 1 gives every edge a 3 by 3 block of arrows: d = (ci − cj) / r, t = (ci − c̄) × (cj − c̄) and o = d × t, where c̄ is the average position. The distance r goes to the numbers lane. Nothing reaches past one hop neighbours, which is the design choice the rest of the page depends on.

Atom 2’s numbers

The scalar channel. The paper’s invariant inputs are atomic numbers and interatomic distances, which no rotation or translation changes. In this toy it is four numbers looked up from the invented atom type.

Atom 2’s arrows

The vector channel. Before the first layer it holds the starting arrows of Eq. 2, whose length γ is faded to zero during training: what the paper calls direction noise. It is an initialisation, not a test with noisy structures. The paper gives no shape for the decay of γ, so none is drawn here. From the second layer on it holds whatever the previous layer produced.

Distance filter

η(r) = φr(χ(r) ∗ ω(r)), with χ a basis function and ω a cutoff function. The paper encodes distances “using radial basis functions” and counts edges with a radius of 5.0 Å in Table 1. The basis and cutoff drawn here are this page’s choice.

Number message

es = φs(sj) ⊙ ηs(r), where φ is a stack of perceptron layers and ⊙ multiplies element by element. stays the same

Fresh arrows

φb(β) ⊙ φd(sj) ⊙ ηd(r). φb is a vector update: one weight matrix across channels, applied identically to x, y and z. The paper: “By conducting identical operations across spatial dimensions, vector updates ensure that the equivariance property is maintained.” turns with the molecule

Carried arrows

Vj ⊙ φv(sj) ⊙ ηr(r). For this term the proof notes that “the resulting vector is equivariant since the same values are multiplied across spatial dimensions”. In this toy, with γ at zero, the first layer has nothing to carry yet. turns with the molecule

Sum over neighbours

s′i = si + Σ es and V′i = Vi + Σ ev over the neighbours of atom i (Eq. 4). A sum treats every neighbour alike and adds arrows tip to tail, so both symmetries pass through it.

Mix

The vector update V′ = W V with a learnable matrix W across channels. The same weights act on x, y and z, so mixing two arrows gives an arrow that turns with them. The paper mentions no bias here, and an added constant arrow would not turn.

Measure length

“The norm operation ‖ · ‖ applied to V is invariant, as the magnitude of the vector does not change with rotation.” This is how information held in arrows reaches predictions that must not change when the molecule turns.

Numbers update

φm(‖φvu(V)‖ ∪ s): a perceptron of the arrow lengths joined with the numbers. A second perceptron of the same input produces the gates that cross back down.

Scale

φvu(V) ⊙ φv(‖φvu(V)‖ ∪ s). In the paper’s words, “the scalar representations scale the vector representations”. One number per arrow: numbers inform arrows without bending them.

Activate by length

VA(V) = V ⊙ σ(‖V‖ + b), Eq. 3, with one bias per channel. This is the switch on the test bench, and the values in this figure follow the bench’s bias and direction noise.

IllustrationDiagram drawn for this page from Equations 1 to 6 and Section 4.2. The paper has no architecture figure. Values are from the first layer of the toy, not from the trained network.

Then repeat

On QM9 the paper stacks eight of these layers in SaVeNet-B and twelve in SaVeNet-L, which is also twice as wide: 256 hidden dimensions against 128 (Appendix F.1).

Two exits

Numbers exit through perceptron layers for targets that must not change when the molecule turns. Arrows exit through more atom-wise blocks, scaled by numbers, for targets that must turn. Whole molecule answers are summed over atoms (Section 4.2).

The guarantee

Theorem 2: “The equivariant representation of SaVeNet is equivariant to any given rotation matrix in R ∈ SO(3). The invariant representation of SaVeNet is invariant to any given transformation matrix in SE(3).” The proof goes operation by operation and never uses the value of a weight.

05 The evidence

The paper reports lower error than Equiformer at a fraction of its training cost, and says where it loses.

Three datasets: QM9 for numbers that must not turn, a simulated N-Body system for arrows that must, and Molecule3D for both. Every value below is the printed cell of the paper’s table, error is always lower is better, and the paper states that all its experiments were conducted on one NVIDIA 3090 GPU with 24 GB.

Cost against error: Tables 3 and 6

Figure 7Measured, Table 3

Pick a cost and see who pays it for which error

Dataset
Error, vertical
Cost, horizontal
Horizontal scale

Point at a mark, tap it, or use the arrow keys. A click or Enter pins one to compare.

Table 3 (page 8). Efficiency comparison of the proposed SaVeNet with the baselines on QM9.
Modelstd. MAElog MAEBatch sizeMemoryLatencySamples/sFLOPsParam.MACs
DimeNet++0.98-5.673574.8238.9164630.151.8915.08
ComENet0.93-5.6911740.9626.4242214.363.817.18
SphereNet0.91-5.732386.6551.6124031.801.8915.90
EQGAT0.86-5.289801.7453.3120017.000.938.49
Equiformer0.70-5.839414.04337.3190-3.53-
SaVeNet-B0.69-6.0416600.8523.1276711.601.375.76
SaVeNet-L0.63-6.145802.4850.6123755.367.7227.51
Table 6 (page 10). Efficiency comparison of the proposed SaVeNet with the baselines on Molecule3D. The table prints DimeNetPP for the model that Tables 2, 3 and 5 print as DimeNet++.
Modelstd. MAElog MAEBatch sizeMemoryLatencySamples/sFLOPsParam.MACs
DimeNetPP0.0188-4.01396212.1484.270470.531.8935.27
SphereNet0.0184-4.03274317.32134.644892.561.9046.28
ComENet0.0220-3.84032773.3740.7153647.257.3623.63
SaVeNet-B0.0156-4.22573292.8427.8230415.331.057.63
SaVeNet-L0.0128-4.44081595.3751.9121638.933.0119.37

↓ lower is better, ↑ higher is better, marked beside each column name. Bold with a solid underline is printed in bold in the paper and dotted underline is printed underlined. The captions do not define either mark; in every column they fall on the best and second best value. A dash means the paper prints none: Equiformer has no FLOPs or MACs and the paper does not say why. Equiformer’s log MAE is printed as -5.83 here and as -5.82 in Table 2.

How it was measured. Latency is “a single forward/backward pass”, a training step and not inference, averaged over the training set after 5,000 warm-up steps. Memory is measured on a worst case batch of the largest samples. FLOPs and MACs use a batch of 64. Each baseline runs with its authors’ hyperparameters. The five baselines of Table 3 are the ones the paper includes in its efficiency comparison. SaVeNet-L has the highest FLOPs, MACs and parameter count of the QM9 rows that report them, and on Molecule3D it is slower than ComENet (51.9 against 40.7 ms) and uses more memory (5.37 against 3.37 GB), although it has fewer FLOPs, MACs and parameters, and its error is lower. The two tables use different model configurations, so this page never joins a QM9 point to a Molecule3D point.

Twelve QM9 targets: Table 2

Figure 8Measured, Table 2

Choose a target and rank every model that reports it

The paper reports that SaVeNet-B has lower error than every baseline on seven of twelve targets, that SaVeNet-L has the lowest error of all models on nine of twelve, and that SaVeNet-L has the lowest std. and log MAE.

Target
Rows

Where SaVeNet is not lowest. For both sizes: εHOMO and εLUMO (Equiformer) and R² (ET). For the base model also εLUMO (EQGAT is lower too), Cν (SphereNet and ComENet) and ZPVE (SphereNet).

Targets have different units, so they are never drawn on one axis. The column you pick is marked in Table 2 below.

Table 2 (page 7). Performance comparisons on QM9: mean absolute error on twelve targets, plus two aggregate measures. † denotes using different data partitions.
TaskαΔεεHOMOεLUMOμCνGHR²UU0ZPVEstd.log
NMP†.092694338.030.0401917.18020201.501.78-5.08
SchNet.235634134.0330.03301414.07319141.701.76-5.17
Cormorant†.085613438.038.0262021.96121222.032.14-4.75
LieConv†.084493025.032.0382224.80019192.281.35-4.99
PhysNet.06142.532.924.7.0529.02809.48.42.7658.348.151.391.37-5.35
DimeNet.04734.827.819.7.029.02498.988.11.3317.898.021.291.05-5.57
DimeNet++.04432.624.619.5.0297.02307.566.53.3316.286.321.210.98-5.67
TFN†.223584038.064.101--------
SE(3)-Tr.†.142533533.051.054--------
EGNN†.071482925.029.0311212.10612111.551.23-5.43
PaiNN.04545.727.620.4.0120.0247.355.98.0665.835.851.281.01-5.85
ET.05936.120.317.5.011.0267.626.16.0336.386.151.840.84-5.90
SphereNet.04631.122.818.9.0245.02157.786.33.2686.366.261.120.91-5.73
ComENet.04532.423.119.8.0245.02207.986.86.2596.826.691.200.93-5.69
SEGNN†.060422421.023.0311516.66013151.621.08-5.27
EQGAT.053322016.011.0242324.38225252.000.86-5.28
Equiformer.0463015.414.7.0117.02307.636.63.2516.746.591.260.70-5.82
SaVeNet-B.03924.818.416.3.0093.02276.645.43.0585.485.431.180.69-6.04
SaVeNet-L.03522.716.615.1.0085.02106.104.83.0494.744.831.100.63-6.14

↓ lower is better in every column. † rows use different data partitions, so they are not strictly like for like. Bold with a solid underline is printed in bold in the paper and dotted underline is printed underlined; the caption does not define them, and in every column they fall on the lowest and second lowest value. A dash means no value is printed. std. and log are the paper’s standardized and logarithmic aggregate measures (it also writes “standard mean absolute error”); their formulas are not given.

Arrows as the answer: Tables 4 and 5

Figure 9Measured, Table 4

N-Body: predict where particles will be

Task

Figure 10Measured, Table 5

Molecule3D: four numbers and one arrow

Target
Table 4 (page 8). Performance comparisons on N-Body dataset: mean squared error of predicted future positions. The table states no units.
MethodES(5)ES(20)G+ES(20)L+ES(20)std.MSElogMSE
GNN0.01310.07200.07210.09080.0151-4.3876
TFN0.02360.07940.08450.12430.0192-4.0978
SE(3)-Tr.0.03290.13490.10000.14380.0252-3.8037
Radial Field0.02070.03770.03990.07790.0110-4.6212
PaiNN0.01580.0997*0.1029*0.1356*0.0835-2.5329
ET0.16530.17880.21220.29890.0535-2.9587
EGNN0.00790.01280.01180.03680.0044-5.6241
ClofNet0.00650.00730.00720.02510.0029-6.0324
GCPNet0.00700.00710.00730.01730.0024-6.1104
SaVeNet0.00620.00630.00820.01230.0020-6.2268

↓ lower is better in every column. ES(5) and ES(20) are 5 and 20 body systems, G+ES(20) adds a gravity field and L+ES(20) a Lorentz-like force field. On G+ES(20), ClofNet (0.0072) and GCPNet (0.0073) are lower than SaVeNet (0.0082). * The paper’s footnote 1: PaiNN’s results “are reported after adding vector activation functions. These results were initially unavailable due to numerical instability.” Most baseline results are adopted from earlier papers.

Table 5 (page 9). Performance comparisons on Molecule3D dataset: mean absolute error. The first six columns sit under the Invariant heading of the paper (four targets and two aggregate measures); the last is the equivariant dipole vector. The table states no units.
TaskμεHOMOεLUMOΔεstd. MAElogMAEμ vector
GIN-Virtual0.08820.06920.06320.10360.0592-2.86771.5233
SchNet0.05320.02750.02650.04280.0263-3.66331.2082
DimeNet++0.02930.02400.01900.03060.0188-4.01391.2014
SphereNet0.02880.02390.01830.03010.0184-4.03271.1836
ComENet0.03450.02880.02520.03260.0220-3.84031.3521
SaVeNet-B0.01830.01900.01730.02900.0156-4.22570.0108
SaVeNet-L0.01360.01590.01430.02390.0128-4.44080.0090

↓ lower is better in every column. The μ vector is the dipole moment as an arrow, which has to turn with the molecule. The baselines sit between 1.1836 and 1.5233 and the two SaVeNet rows at 0.0108 and 0.0090. The paper does not describe how the baselines were set up to predict a vector.

Figure 11Measured, Table 7

Walk down the ablation, one change at a time

How to read it. Depth accounts for the two largest steps; the two new pieces account for 2.5% and 2.8%. The paper also reports that vector activations improve numerical stability during training, particularly with larger models. The twelve layer SaVeNet-L is also wider than the eight layer SaVeNet-B on QM9, and the paper does not say whether the last row shares that width.

Δstd. is measured from the first row; the percentages in the list are the paper’s, each against the row before. Only these five configurations are reported, so there is no free switch for each piece.

  1. Six layers, neither new piece. The starting point: 0.79.
  2. Add direction noise. 0.79 to 0.77. The paper: “the std. MAE is reduced by 2.5%”.
  3. Go from six to eight layers. 0.77 to 0.71, “a 7.8% reduction in error”.
  4. Add vector activations. 0.71 to 0.69, which “further improves the performance by 2.8%”.
  5. Go from eight to twelve layers. 0.69 to 0.63, “a substantial performance improvement of 8.7%”.
Table 7 (page 10). Ablation study of SaVeNet on QM9. DN is direction noise, VA is vector activations, L is the number of layers, and “- DN” means without it.
ModelLstd.Δstd.log
SaVeNet - DN - VA60.79--5.85
SaVeNet - VA60.77-2.5%-5.95
SaVeNet - VA80.71-10.1%-5.98
SaVeNet80.69-12.7%-6.04
SaVeNet120.63-20.3%-6.14

The three datasets

Table 1 (page 6). Dataset details. The paper counts edges with a radius graph of 5.0 Å.
DatasetQM9N-BodyMolecule3D
Average Number of Nodes18.022029.11
Max Number of Nodes2920137
Average Number of Edges280.72380553.00
Number of Graphs130,8317,0003,899,647
Number of Tasks1245
Splits84:8:842:29:296:2:2
Invariantyesnoyes
Equivariantnoyesyes

Invariant targets must not change when the molecule turns. Equivariant targets must turn with it. Table 1 prints 130,831 graphs for QM9; the appendix prints a different figure.

The takeaway

Keep the arrows safe, stay at one hop, and spend the saving on depth.

What is new

A one hop message passing framework with invariant and equivariant decoders, plus two pieces for the arrow channel: starting arrows that fade out during training, and an activation that reads only an arrow’s length.

What it buys

A one hop layer cheap enough to stack. In the paper’s ablation the two largest gains come from going deeper.

Not claimed

No test with noisy structures, no cost measured against molecule size, no inference speed, and no error bars or repeated seeds.

06 Office hours

The night shift answers.

Written by Tian's agent while Tian slept and Pepper supervised. Pepper accepts no liability either.

Each answer says where in the paper it comes from, so you can check the agent's work.

Pepper, the tabby cat, supervising: fast asleep on his blanket on the sofa, beside the laptop.
“Direction noise”: is this a result about robustness to noisy structures?

No. It is how the arrows are initialised. Each atom type gets a learnable direction on a sphere, scaled by γ, and γ follows a decay schedule “that ensures it reaches zero before training concludes”, so nothing of it is left at inference. The paper does not test noisy or perturbed inputs. In the ablation, adding it lowers std. MAE on QM9 from 0.79 to 0.77 at six layers.

Source: Section 4.1, Eq. 2, Appendix C, Table 7
If the network only ever looks one hop away, how can it know about angles?

The paper’s argument is about the input. A distance and a unit direction per edge are enough to rebuild the whole geometry of a connected molecule up to a translation (Lemma 1, Theorem 1), so nothing geometric is thrown away by skipping triplets. The authors call this a “lossless geometric encoding”. It is a statement about the encoding, not a proof of what the trained network can represent; the evidence for that is the experiments.

Source: Section 4.1, Appendices A and B
Is it cheaper in theory, or only on the authors’ GPU?

Both are reported, with limits. The complexity table gives n2d2 for SaVeNet, the same order as ComENet and one power of n below DimeNet++ and SphereNet; the symbols are not defined and Equiformer is not in that table. The measurements are training steps on one RTX 3090 with each baseline’s published hyperparameters: 23.1 ms for SaVeNet-B against 26.4 ms for ComENet and up to 337.3 ms for Equiformer. Inference latency and cost against molecule size are not measured. The large model is not cheap in every column: it has the most parameters and FLOPs in Table 3, and on Molecule3D it is slower than ComENet. One gap: ET has the second lowest std. MAE among the QM9 baselines (0.84) but no row in the efficiency table, and the paper does not say why.

Source: Table 8, Tables 3 and 6, Section 5.1, Appendix G
Where does it lose?

On QM9, Equiformer has lower error on εHOMO and εLUMO and ET on R², for both SaVeNet sizes; the base model is also behind SphereNet on ZPVE and behind SphereNet and ComENet on Cν. On N-Body with a gravity field, ClofNet and GCPNet are lower. Seven QM9 baselines use different data partitions. The headline gap to Equiformer is small on accuracy (0.69 against 0.70 std. MAE); the large gap is cost. No result in the paper has error bars or repeated seeds, and the method needs a known 3D structure at equilibrium for every molecule, which the paper names as its primary limitation and shares with its baselines.

Source: Tables 2, 3 and 4, Section 5.1, Section 6
How much do the two new pieces matter, compared with just going deeper?

In the QM9 ablation, direction noise gives 2.5% and vector activations 2.8%, while going from six to eight layers gives 7.8% and from eight to twelve layers 8.7%. Beyond the 2.8%, the paper introduces vector activations to keep the arrows equivariant, and reports that they “significantly enhance the numerical stability during training, particularly with larger-scale models”; it also reports that PaiNN only produced N-Body results once they were added. The twelve layer model on QM9 is also wider (256 against 128 hidden dimensions), and the paper does not separate the two.

Source: Table 7, Section 5.2, Section 4.1, Table 4 footnote, Appendix F.1