> ## Documentation Index
> Fetch the complete documentation index at: https://bench.flashinfer.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace

This document describes the JSON schema for a **Trace**.

A `Trace` describes a benchmark result of a [Solution](/docs/docs/flashinfer-trace/solution) on a [Definition](/docs/docs/flashinfer-trace/definition) with a specific [Workload](/docs/docs/flashinfer-trace/workload). The collection of all Trace files forms the database of benchmark results.

In a `Trace` object, `solution` and `evaluation` are optional. If they are not provided, it describes
a workload entry in the dataset.

In a `Trace` object, the `definition` and `solution` are externally linked through their names;
`workload` and `evaluation` are embedded in the `Trace` object. This is because `definition` and
`solution` are relatively large objects and will be used repeatedly in the dataset.

## JSON Schema Description

### **Top-Level Object Structure**

| **Field**    | **Type** | **Required** | **Description**                                                                                                      |
| ------------ | -------- | ------------ | -------------------------------------------------------------------------------------------------------------------- |
| `definition` | string   | Yes          | The `name` of the `Definition` used in this run.                                                                     |
| `workload`   | object   | Yes          | An object describing the specific input configuration for this run. See [Workload](/docs/docs/flashinfer-trace/workload). |
| `solution`   | string   | No           | The `name` of the `Solution` tested in this run.                                                                     |
| `evaluation` | object   | No           | An object containing the detailed results of this run.                                                               |

### `evaluation` : Benchmark Statistics Summary

This object represents a single, complete benchmark result.

| **Field**                                                                                                            | **Type** | **Required** | **Description**                                                         |
| -------------------------------------------------------------------------------------------------------------------- | -------- | ------------ | ----------------------------------------------------------------------- |
| `status`                                                                                                             | string   | Yes          | The final status of the evaluation run. Has to be one of the following: |
| `"PASSED"`, `"INCORRECT_SHAPE"`, `"INCORRECT_NUMERICAL"`, `"INCORRECT_DTYPE"`, `"RUNTIME_ERROR"`, `"COMPILE_ERROR"`. |          |              |                                                                         |
| `log`                                                                                                                | string   | Yes          | The embedded record of the stdout and stderr of the evaluation run.     |
| `correctness`                                                                                                        | object   | Yes          | The summarized correctness results across all entries in the dataset.   |
| `performance`                                                                                                        | object   | Yes          | The summarized performance metrics across all entries in the dataset.   |
| `environment`                                                                                                        | object   | Yes          | A snapshot of the hardware and software execution environment.          |
| `timestamp`                                                                                                          | string   | Yes          | The ISO 8601 timestamp of when this summary was generated.              |

### `correctness` : Correctness Summary

| **Field**            | **Type** | **Required** | **Description**                        |
| -------------------- | -------- | ------------ | -------------------------------------- |
| `max_relative_error` | float    | Yes          | The maximum relative difference found. |
| `max_absolute_error` | float    | Yes          | The maximum absolute difference found. |

### `performance` : Performance Summary

| **Field**              | **Type** | **Required** | **Description**                                                                  |
| ---------------------- | -------- | ------------ | -------------------------------------------------------------------------------- |
| `latency_ms`           | float    | Yes          | The mean latency in milliseconds per execution for this implementation.          |
| `reference_latency_ms` | float    | Yes          | The mean latency of the `Definition`'s reference code on the same data/hardware. |
| `speedup_factor`       | float    | Yes          | The calculated speedup (`reference_latency_ms / latency_ms`).                    |

> Note that it's normal for the speedup factor to be very large since the references are torch only, unoptimized implementations.

### **`environment`: Environment Definition Object**

The `environment` object specifies the exact execution environment for this benchmark run.

| **Field**  | **Type** | **Required** | **Description**                                                                                                           |
| ---------- | -------- | ------------ | ------------------------------------------------------------------------------------------------------------------------- |
| `hardware` | string   | Yes          | The name of the hardware, e.g., `"NVIDIA_H100"`.                                                                          |
| `libs`     | object   | Yes          | A snapshot of the relevant software libraries and their versions. Keys are library names, and values are version strings. |

### The `correctness` and `performance` Nullable Table

The `correctness` and `performance` fields are set to be nullable depending on the `status`.

| status                 | correctness | performance |
| ---------------------- | ----------- | ----------- |
| PASSED                 | Required    | Required    |
| INCORRECT\_NUMERICAL   | Required    | **None**    |
| INCORRECT\_SHAPE/DTYPE | **None**    | **None**    |
| RUNTIME\_ERROR         | **None**    | **None**    |
| COMPILE\_ERROR         | **None**    | **None**    |

### Example: RMSNorm Trace

```python theme={null}
{
  "definition": "rmsnorm",
  "solution": "rmsnorm_triton_v1",
  "workload": {
    "uuid": "6120f144-b973-4bd9-b884-77ecb132914e",
    "axes": {
      "batch_size": 32
    },
    "inputs": {
      "input": {
        "type": "safetensors",
        "path": "/data/rmsnorm_evals/b32_input.safetensors",
        "tensor_key": "input"
      },
      "weight": {
        "type": "safetensors",
        "path": "/data/rmsnorm_evals/rmsnorm_weight.safetensors",
        "tensor_key": "weight"
      }
    }
  },
  "evaluation": {
    "status": "PASSED",
    "log": "...",
    "correctness": {
      "max_relative_error": 1.15e-05,
      "max_absolute_error": 0.89e-05
    },
    "performance": {
      "latency_ms": 0.008,
      "reference_latency_ms": 0.019,
      "speedup_factor": 2.375
    },
    "environment": {
      "hardware": "NVIDIA_H100",
      "libs": {
        "cuda": "12.6",
        "torch": "2.6.0",
        "triton": "2.4.0"
      }
    },
    "timestamp": "2025-06-27T12:45:00Z"
  }
}
```