> ## Documentation Index
> Fetch the complete documentation index at: https://bench.flashinfer.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# RMSNorm

Root Mean Square Layer Normalization (RMSNorm) is a normalization technique that normalizes the input by the root mean square of its elements.

Variants:

* Standard RMSNorm: basic RMS normalization that scales input by RMS and applies learned weight parameters
* Fused Add RMSNorm: adds residual connection before normalization in a single fused operation

Axes (2 dimensions):

* `batch_size`: variable
* `hidden_size`: constant

Inputs (2 or 3 tensors):

* `hidden_states`: \[batch\_size, hidden\_size]
* `weight`: \[hidden\_size]
* For Fused Add RMSNorm only:
  * `residual`: \[batch\_size, hidden\_size]

Outputs (1 tensor):

* `output`: \[batch\_size, hidden\_size]