How does a graph attention layer compute per-neighbour coefficients, and what is the softmax taken over?
answer
- score every edge, then normalize
- the normalizer runs over one node only
- shared transform, then a learned scoring vector
- masked softmax, coefficients sum to one
basics
~20 sA graph attention layer scores each edge from its two endpoints' transformed feature vectors using a small learned function, then softmax-normalizes those scores over only that node's own neighbourhood, so each node's incoming weights are positive and sum to one.
solid answer
~50 sEvery node feature vector is first pushed through one shared learned matrix, giving `z_i = W h_i`. For each edge into node `i` from neighbour `j`, a single learned vector `a` reads the concatenation of the two transformed vectors and a LeakyReLU turns it into one raw score: `e_ij = LeakyReLU(a . [z_i || z_j])`. Those raw scores are then softmax-normalized over exactly the neighbours of `i`, with `i` itself included through a self-loop: `alpha_ij = exp(e_ij) / sum_k in N(i) exp(e_ik)`. The update is the weighted sum `h_i' = sigma(sum_j alpha_ij z_j)`. The normalizer is per-node, not global — no score is ever computed for a non-edge, so the cost scales with the number of edges. That local softmax is what lets a knowledge-graph entity with 200 attached relations concentrate almost all of its weight on the three that determine its type.
go deeper
Recall the shape of it: each edge gets a learned score, the scores are softmaxed over one node's neighbours, and the node's new vector is that weighted sum. Knowing the coefficients are positive and sum to one per node is most of the credit here.
Be ready to write the three steps on a whiteboard in order — shared transform, scored concatenation through a LeakyReLU, masked softmax — and to say explicitly which set the denominator sums over and that the self-loop is inside it.
Expect to be pushed on consequences: asymmetry of the coefficients, cost scaling with edges rather than node pairs, and why the layer stays usable on a graph it never saw in training. Say what the coefficients do and do not mean before someone asks.
Own the framing that attention over neighbours is a modelling choice with a cost curve, not a free upgrade. Be able to argue when the extra scoring parameters and per-edge memory earn their place in a system you will operate for years.
## What the layer is trying to do A message-passing layer updates each node by combining the vectors of the nodes it is connected to. The simplest combiners fix every neighbour's weight in advance: each neighbour counts the same, or its weight is decided by node degrees alone. A graph attention layer instead **learns** a weight per edge from the features sitting at that edge's two endpoints, so that inside one neighbourhood some neighbours dominate and others are pushed toward zero. ## Step 1 — one shared linear transform Every node's feature vector `h_j` (length `F`) goes through the same learned matrix `W`, producing `z_j = W h_j` of length `F'`. One matrix for the whole layer — not one per node, not one per edge. This is what keeps the layer applicable to graphs of any size and to nodes never seen during training. ## Step 2 — an unnormalized score per edge For a target node `i` and a neighbour `j`, the layer emits one scalar: ``` e_ij = LeakyReLU(a . [z_i || z_j]) ``` `||` is concatenation and `a` is a single learned vector of length `2F'`. The score is therefore a tiny one-layer network reading **both** endpoints, which is why the same neighbour `j` can score high for one target and low for another. LeakyReLU rather than a plain rectifier keeps a gradient alive on negatively scoring edges instead of collapsing them all onto the same flat zero. ## Step 3 — softmax over exactly that node's neighbourhood ``` alpha_ij = exp(e_ij) / sum over k in N(i) of exp(e_ik) ``` The denominator runs over `N(i)` — node `i`'s neighbours, with `i` itself included via a self-loop — and over nothing else. This is *masked* attention: scores are never computed for pairs that are not edges. Three consequences follow. 1. **The coefficients are a local allocation.** They are non-negative and sum to one *per target node*. They say how node `i` divides its budget among its own neighbours; they are not comparable across nodes as absolute importances. 2. **They are asymmetric.** `alpha_ij` and `alpha_ji` are normalized over different neighbourhoods, and the score itself is order-dependent because `z_i` and `z_j` occupy different halves of the concatenation. A node may lean heavily on a neighbour that barely notices it back. 3. **The cost is proportional to the number of edges**, not to the square of the node count. Nothing dense is ever materialised, which is precisely why this style of attention works on a graph with millions of nodes. ## Step 4 — aggregate ``` h_i' = sigma(sum over j in N(i) of alpha_ij z_j) ``` A convex combination of the transformed neighbour vectors, followed by a nonlinearity. Note what is being weighted: the *transformed* vectors `z_j`, not the raw features. ## Why the local softmax matters Take a knowledge-graph entity carrying 200 attached relations, only three of which actually determine its type. A fixed-weight combiner gives each of the 200 the same say, or a say decided by degree, and the three informative ones are diluted roughly two-hundred-fold. Because the softmax runs over exactly that node's 200 edges, the layer can drive nearly all the mass onto those three — a large score gap turns into a near-one-hot allocation. A softmax over the whole graph could not do this: the scores of edges belonging to entirely unrelated nodes would enter the same denominator and the coefficients would stop meaning "share of this node's neighbourhood". ## What is learned, and how much of it Per head the learnable parameters are the matrix `W` (`F x F'`) and the scoring vector `a` (`2F'`). The scorer is tiny relative to the transform; most of the layer's capacity still lives in `W`. Nothing about the graph's identity is learned — no per-node or per-edge embedding table — so the same weights apply unchanged to a graph the model has never seen, which is what makes the layer inductive. ## The static-attention subtlety Because the score is linear in the concatenation, `a . [z_i || z_j]` splits into `a1 . z_i + a2 . z_j`. The first term is constant across `j`, so before the LeakyReLU the *ranking* of neighbours is the same for every target node — the original formulation gives what later work called **static attention**. GATv2 fixes it by moving the nonlinearity inside, scoring `a . LeakyReLU(W [h_i || h_j])`, which makes the ranking genuinely depend on the query node. If an interviewer pushes on "can this attention really be query-dependent?", this is the answer they are fishing for. ## What this layer does not do It has no notion of position or ordering — the neighbourhood is a set, and the coefficients are permutation-equivariant. It sees no path structure beyond one hop per layer. And a coefficient of zero is not a deleted edge: the edge still constrains what the softmax can do for the others, and the message still exists at the next layer if the model changes its mind.
- Why restrict the softmax to a node's neighbours instead of taking it over every node in the graph?Three reasons. Cost: a global softmax needs a score for every node pair, quadratic in node count, while masked attention costs one score per edge. Meaning: normalizing per node makes the coefficients a share of that node's neighbourhood rather than a share of the whole graph. Generalization: the layer never depends on which nodes exist, so the same weights transfer to an unseen graph.
- If node i weights neighbour j at 0.6, what does that tell you about the weight j puts on i?Nothing. The two coefficients are normalized over different neighbourhoods — j may have twenty other neighbours competing for its budget — and the raw score itself is asymmetric because the two endpoint vectors sit in different halves of the concatenation the scorer reads. Attention over neighbours is directed even when the underlying edge is not.
- What is static attention in the original formulation, and how does GATv2 change it?The original score is linear in the concatenated endpoints, so it decomposes into a term from the target plus a term from the neighbour. The target's term is constant across its neighbours, which means the induced ranking of neighbours is global rather than query-dependent. GATv2 applies the nonlinearity before the scoring vector, so the ranking can differ from node to node.
saying these in an interview costs you the question
- Says the softmax is taken over all nodes in the graph
- Claims the coefficients are symmetric between the two endpoints
- Describes the score as raw feature similarity with no learned parameters
- Forgets the node's own self-loop competes inside the softmax
- Assumes a dense adjacency matrix must be held in memory