jax-ml/jax · error · RuntimeError

Profiler collected an odd number of trace events. This likel

Error message

Profiler collected an odd number of trace events. This likely indicates memory corruption due to insufficient profiling space. Try again with a larger profiling space.

What it means

Trace events are stored as (name_id, timestamp) pairs, so a valid trace always has an even event count. An odd count means the tail of the buffer was partially clobbered by newer writes — the classic symptom of insufficient profiling space — and dump() aborts instead of returning corrupt data.

Source

Thrown at jax/experimental/mosaic/gpu/profiler.py:282

      raise RuntimeError("Insufficient space to capture a full trace")
    traces = entries[..., 3:]

    # Estimate the overhead of profiling.
    time_events = traces[:, :, 1::2]
    valid_times_mask = np.arange(traces.shape[-1])[1::2] < traces_used[..., None]
    # 12 cycles is a ballpark estimate for H100
    profiling_overhead = (time_events[:, :, 1:] - time_events[:, :, :-1]).min(
        where=valid_times_mask[:, :, 1:], initial=12
    )
    profiling_overhead = max(0, profiling_overhead - 1)

    unintern = {v: k for k, v in self.interned_names.items()}
    events = []
    for block_idx, trace_idx in np.ndindex(num_blocks, traces_per_block):
      valid_entries = traces_used[block_idx, trace_idx]
      local_clock_offset = None
      if valid_entries % 2:
        raise RuntimeError(
            "Profiler collected an odd number of trace events. This likely "
            "indicates memory corruption due to insufficient profiling space. "
            "Try again with a larger profiling space."
        )
      start_time = start_times[block_idx, trace_idx]
      block_events = []
      last_time = float("-inf")
      for i in range(0, valid_entries, 2):
        tag = traces[block_idx, trace_idx, i]
        time = traces[block_idx, trace_idx, i + 1]
        if local_clock_offset is None:
          local_clock_offset = time
        time -= local_clock_offset
        time -= (i // 2) * profiling_overhead  # Account for the overhead of profiling.
        if time < 0:
          break  # Detect a timer wraparound
        name_id = tag
        begin = True

View on GitHub (pinned to 1e1c6a8fc0)

Solutions

1. Enlarge the profiling buffer (entries_per_warpgroup) as the message suggests
2. Decrease the number of record() events per warp

Example fix

# before
spec = ProfilerSpec(trace_scope=ThreadSubset.WARP)

# after
spec = ProfilerSpec(trace_scope=ThreadSubset.WARP, entries_per_warpgroup=2048)

try:
    spec.dump(buffer, f, grid, block)
except RuntimeError as e:
    if 'odd number of trace events' in str(e):
        # grow buffer and retry once
        ...

When it happens

Trigger: Same root cause as the 'insufficient space' error but caught differently: buffer happened to be overwritten in a way that leaves an odd number of used entries (e.g. overflow occurring mid-event).

Common situations: Kernels with variable per-warp event counts that occasionally exceed capacity; increasing concurrency so some warps overflow.

Related errors

• {iterations=} must be positive
• Unsupported trace scope: {trace_scope}
• Scope {self.trace_scope} not supported
• Block size is not a multiple of {scope_size}
• Allocated too many names

AI-assisted analysis of jax-ml/jax@1e1c6a8fc0 (2026-08-27). Data as JSON: /api/errors/229573a0fabd27f4. Report an issue: GitHub.