jax-ml/jax · error · RuntimeError

Insufficient space to capture a full trace

Error message

Insufficient space to capture a full trace

What it means

When dumping the trace buffer, dump() checks that traces_used + 3 (header entries) fits within entries_per_warpgroup. If a kernel recorded more events than the buffer was sized for, the extra events were silently overwritten and the dump is unreliable, so it raises.

Source

Thrown at jax/experimental/mosaic/gpu/profiler.py:264

      return name_id
    name_id = self.interned_names[name] = len(self.interned_names)
    if name_id & self.EXIT:
      raise RuntimeError("Allocated too many names")
    return name_id

  def dump(self, buffer, f, grid: tuple[int, ...], block: tuple[int, ...]):
    buffer = np.asarray(buffer)
    num_blocks = math.prod(grid)
    traces_per_block = self._num_traces((), block)
    entries = buffer.reshape(
        num_blocks, traces_per_block, self.entries_per_warpgroup
    )
    start_times = entries[..., 0]
    sm_ids = entries[..., 1]
    traces_used = entries[..., 2]
    entries_used = traces_used + 3
    if np.any(entries_used > self.entries_per_warpgroup):
      raise RuntimeError("Insufficient space to capture a full trace")
    traces = entries[..., 3:]

    # Estimate the overhead of profiling.
    time_events = traces[:, :, 1::2]
    valid_times_mask = np.arange(traces.shape[-1])[1::2] < traces_used[..., None]
    # 12 cycles is a ballpark estimate for H100
    profiling_overhead = (time_events[:, :, 1:] - time_events[:, :, :-1]).min(
        where=valid_times_mask[:, :, 1:], initial=12
    )
    profiling_overhead = max(0, profiling_overhead - 1)

    unintern = {v: k for k, v in self.interned_names.items()}
    events = []
    for block_idx, trace_idx in np.ndindex(num_blocks, traces_per_block):
      valid_entries = traces_used[block_idx, trace_idx]
      local_clock_offset = None
      if valid_entries % 2:
        raise RuntimeError(

View on GitHub (pinned to 1e1c6a8fc0)

Solutions

1. Increase the profiling buffer size (entries per warpgroup) in ProfilerSpec
2. Reduce the number of profiled regions/events per warp

Example fix

# before
spec = ProfilerSpec(trace_scope=ThreadSubset.WARP)  # default capacity

# after
spec = ProfilerSpec(trace_scope=ThreadSubset.WARP, entries_per_warpgroup=1024)

estimated = max_events_per_warp()  # your kernel's count
if estimated + 3 > spec.entries_per_warpgroup:
    spec = ProfilerSpec(trace_scope=spec.trace_scope,
                        entries_per_warpgroup=estimated * 2 + 8)

try:
    spec.dump(buffer, f, grid, block)
except RuntimeError as e:
    if 'Insufficient space' in str(e):
        spec = ProfilerSpec(trace_scope=spec.trace_scope,
                            entries_per_warpgroup=spec.entries_per_warpgroup * 2)
        # relaunch and redump

When it happens

Trigger: Profiling a kernel that emits more trace events than the ProfilerSpec buffer capacity (entries_per_warpgroup set too small for the number of record() calls per warp).

Common situations: Deeply nested instrumented regions or looped record() calls inside the kernel while keeping the default buffer size.

Related errors

• {iterations=} must be positive
• Unsupported trace scope: {trace_scope}
• Scope {self.trace_scope} not supported
• Block size is not a multiple of {scope_size}
• Allocated too many names

AI-assisted analysis of jax-ml/jax@1e1c6a8fc0 (2026-08-27). Data as JSON: /api/errors/827645e3011758b5. Report an issue: GitHub.