Skip to main content
This document describes the JSON schema for a Trace. A Trace is an atomic, immutable record of a single benchmark run. It links a specific Solution to a specific Definition, details the exact workload configuration used for the run (i.e., shapes and input data), and records the complete evaluation result. The collection of all Trace files forms the database of benchmark results.

​
JSON Schema Description

​
Top-Level Object Structure

FieldTypeRequiredDescription
definitionstringYesThe name of the Definition used in this run.
solutionstringYesThe name of the Solution tested in this run.
workloadobjectYesAn object describing the specific input configuration for this run.
evaluationobjectYesAn object containing the detailed results of this run.

​
workload : Input Shapes and Data

This object provides the concrete data required to instantiate a Definition. This data includes the variable dimensions of inputs and outputs and, for cases where latency is correlated with the input distribution, the specific input values themselves.
FieldTypeRequiredDescription
uuidstringYesA randomly generate UUID for this workload entry.
axesobjectYesAn object mapping var axis names from the Definition to their concrete integer values.
inputsobjectYesAn object describing the location and format of the required input tensor data files.

​
inputs : Input Descriptor Objects

This object maps input names (e.g., "A", "weight", "mask") to input descriptors that explain where the data comes from and (when necessary) how it should be generated or loaded. Each descriptor must contain at least the type field. Additional fields become required or optional depending on the chosen type.
FieldTypeRequiredDescription
typestringYesData source type. Could be random, scalar, or safetensors.
Additional fields for type scalar:
FieldTypeRequiredDescription
valueint, float, boolYesThe concrete value of the input.
Additional fields for type safetensors:
FieldTypeRequiredDescription
pathstringYesRelative path or URI of the .safetensors file.
tensor_keystringYesThe key inside the safetensors container that holds this tensor.

​
evaluation : Benchmark Statistics Summary

This object represents a single, complete benchmark result.
FieldTypeRequiredDescription
statusstringYesThe final status of the evaluation run. Has to be one of the following:
"PASSED", "INCORRECT_SHAPE", "INCORRECT_NUMERICAL", "INCORRECT_DTYPE", "RUNTIME_ERROR", "COMPILE_ERROR".
logstringYesThe embedded record of the stdout and stderr of the evaluation run.
correctnessobjectYesThe summarized correctness results across all entries in the dataset.
performanceobjectYesThe summarized performance metrics across all entries in the dataset.
environmentobjectYesA snapshot of the hardware and software execution environment.
timestampstringYesThe ISO 8601 timestamp of when this summary was generated.

​
correctness : Correctness Summary

FieldTypeRequiredDescription
max_relative_errorfloatYesThe maximum relative difference found.
max_absolute_errorfloatYesThe maximum absolute difference found.

​
performance : Performance Summary

FieldTypeRequiredDescription
latency_msfloatYesThe mean latency in milliseconds per execution for this implementation.
reference_latency_msfloatYesThe mean latency of the Definition’s reference code on the same data/hardware.
speedup_factorfloatYesThe calculated speedup (reference_latency_ms / latency_ms).
Note that it’s normal for the speedup factor to be very large since the references are torch only, unoptimized implementations.

​
environment: Environment Definition Object

The environment object specifies the exact execution environment for this benchmark run.
FieldTypeRequiredDescription
hardwarestringYesThe name of the hardware, e.g., "NVIDIA_H100".
libsobjectYesA snapshot of the relevant software libraries and their versions. Keys are library names, and values are version strings.

​
The correctness and performance Nullable Table

The correctness and performance fields are set to be nullable depending on the status.
statuscorrectnessperformance
PASSEDRequiredRequired
INCORRECT_NUMERICALRequiredNone
INCORRECT_SHAPE/DTYPENoneNone
RUNTIME_ERRORNoneNone
COMPILE_ERRORNoneNone

​
Example: RMSNorm Trace

{
  "definition": "rmsnorm",
  "solution": "rmsnorm_triton_v1",
  "workload": {
    "uuid": "6120f144-b973-4bd9-b884-77ecb132914e",
    "axes": {
      "batch_size": 32
    },
    "inputs": {
      "input": {
        "type": "safetensors",
        "path": "/data/rmsnorm_evals/b32_input.safetensors",
        "tensor_key": "input"
      },
      "weight": {
        "type": "safetensors",
        "path": "/data/rmsnorm_evals/rmsnorm_weight.safetensors",
        "tensor_key": "weight"
      }
    }
  },
  "evaluation": {
    "status": "PASSED",
    "log": "...",
    "correctness": {
      "max_relative_error": 1.15e-05,
      "max_absolute_error": 0.89e-05
    },
    "performance": {
      "latency_ms": 0.008,
      "reference_latency_ms": 0.019,
      "speedup_factor": 2.375
    },
    "environment": {
      "hardware": "NVIDIA_H100",
      "libs": {
        "cuda": "12.6",
        "torch": "2.6.0",
        "triton": "2.4.0"
      }
    },
    "timestamp": "2025-06-27T12:45:00Z"
  }
}
⌘I