> ## Documentation Index
> Fetch the complete documentation index at: https://bench.flashinfer.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# GDN

Gated Delta Net (GDN) linear attention mechanism. GDN implements the delta rule for linear attention with gated updates, enabling efficient recurrent computation. It supports both GQA (grouped query attention, num\_q\_heads > num\_v\_heads) and GVA (grouped value attention, num\_v\_heads > num\_q\_heads) configurations.

The delta rule update is:

```
g = -exp(A_log) * softplus(a + dt_bias)   # Log-space decay gate
beta = sigmoid(b)                          # Update gate
state = state * exp(g)                     # Apply decay to state
v_new = v - k @ state                      # Prediction error (delta rule)
v_new = v_new * beta                       # Apply update gate
state = state + k^T @ v_new                # Update state (outer product)
output = scale * q @ state                 # Compute output
```

Variants:

* prefill (chunked computation for variable-length sequences)
* decode (single-token generation with recurrent state update)

## prefill

Axes (6 dimensions):

* `total_seq_len`, `num_seqs`: variable
* `num_q_heads`, `num_k_heads`, `num_v_heads`, `head_size`: constant

Inputs (7 tensors + 1 scalar):

* `q`: query tensor \[total\_seq\_len, num\_q\_heads, head\_size]
* `k`: key tensor \[total\_seq\_len, num\_k\_heads, head\_size]
* `v`: value tensor \[total\_seq\_len, num\_v\_heads, head\_size]
* `g`: forget gate (alpha) \[total\_seq\_len, num\_sab\_heads], float32, optional (defaults to ones)
* `beta`: update gate \[total\_seq\_len, num\_sab\_heads], float32, optional (defaults to ones)
* `cu_seqlens`: cumulative sequence lengths \[num\_seqs + 1], int64
* `initial_state`: initial KV state \[num\_seqs, num\_sab\_heads, head\_size, head\_size], float32, optional
* `scale`: softmax scale (scalar), optional (defaults to 1/sqrt(head\_size))

Outputs (2 tensors):

* `output`: attention output \[total\_seq\_len, num\_o\_heads, head\_size]
* `final_state`: final KV state \[num\_seqs, num\_sab\_heads, head\_size, head\_size], float32

Where:

* `num_sab_heads = max(num_q_heads, num_v_heads)` (state and beta heads)
* `num_o_heads = max(num_q_heads, num_v_heads)` (output heads)

Constraints:

* `total_seq_len == cu_seqlens[-1]`
* `num_seqs == len(cu_seqlens) - 1`
* For GQA: `num_q_heads >= num_k_heads` and `num_q_heads % num_k_heads == 0`
* For GVA: `num_v_heads >= num_q_heads` and `num_v_heads % num_q_heads == 0`
* `num_k_heads == num_v_heads` (keys and values must have same number of heads)

Notes:

* The final state is in k-last layout \[N, H, V, K]
* Gate tensors (g, beta) are in float32 for numerical stability

## decode

Single-token decoding with recurrent state update. Uses gating parameters (A\_log, a, dt\_bias, b) to compute decay and update gates. Optionally applies L2 normalization to q and k for numerical stability.

Axes (6 dimensions):

* `batch_size`: variable (number of sequences being decoded)
* `num_q_heads`, `num_k_heads`, `num_v_heads`, `head_size`: constant

Inputs (10 tensors + 1 scalar):

* `q`: query tensor \[batch\_size, 1, num\_q\_heads, head\_size], bfloat16
* `k`: key tensor \[batch\_size, 1, num\_k\_heads, head\_size], bfloat16
* `v`: value tensor \[batch\_size, 1, num\_v\_heads, head\_size], bfloat16
* `state`: recurrent state \[batch\_size, num\_sab\_heads, head\_size, head\_size], float32
* `A_log`: log decay parameter \[num\_sab\_heads], float32
* `a`: input-dependent decay \[batch\_size, 1, num\_sab\_heads], bfloat16
* `dt_bias`: decay bias \[num\_sab\_heads], bfloat16
* `b`: update gate input \[batch\_size, 1, num\_sab\_heads], bfloat16
* `scale`: scale factor (scalar), float32 (default: 1.0 or 1/sqrt(head\_size))
* `use_qk_l2norm`: whether to apply L2 normalization to q and k, bool (default: true)

Outputs (2 tensors):

* `output`: attention output \[batch\_size, 1, num\_o\_heads, head\_size], bfloat16
* `new_state`: updated recurrent state \[batch\_size, num\_sab\_heads, head\_size, head\_size], float32

Where:

* `num_sab_heads = max(num_q_heads, num_v_heads)` (state and beta heads)
* `num_o_heads = num_v_heads` (output heads follow value heads)

Gating computation:

```
g = -exp(A_log) * softplus(a + dt_bias)   # softplus(x) = log(1 + exp(x))
beta = sigmoid(b)
```

Constraints:

* For GVA: `num_v_heads >= num_q_heads` and `num_v_heads % num_q_heads == 0`
* `num_k_heads == num_q_heads` (keys and queries must have same number of heads)

State Layout:

* **k-last**: \[B, H, V, K] - V dimension before K dimension, faster for decode
* **k-first**: \[B, H, K, V] - K dimension before V dimension

Notes:

* L2 normalization helps with numerical stability when head\_size is large