jax-ml/jax · error · RuntimeError

Insufficient space to capture a full trace

Error message

Insufficient space to capture a full trace

What it means

When dumping the trace buffer, dump() checks that traces_used + 3 (header entries) fits within entries_per_warpgroup. If a kernel recorded more events than the buffer was sized for, the extra events were silently overwritten and the dump is unreliable, so it raises.

Solutions

1. Increase the profiling buffer size (entries per warpgroup) in ProfilerSpec
2. Reduce the number of profiled regions/events per warp

Example fix

# before
spec = ProfilerSpec(trace_scope=ThreadSubset.WARP)  # default capacity

# after
spec = ProfilerSpec(trace_scope=ThreadSubset.WARP, entries_per_warpgroup=1024)

estimated = max_events_per_warp()  # your kernel's count
if estimated + 3 > spec.entries_per_warpgroup:
    spec = ProfilerSpec(trace_scope=spec.trace_scope,
                        entries_per_warpgroup=estimated * 2 + 8)

try:
    spec.dump(buffer, f, grid, block)
except RuntimeError as e:
    if 'Insufficient space' in str(e):
        spec = ProfilerSpec(trace_scope=spec.trace_scope,
                            entries_per_warpgroup=spec.entries_per_warpgroup * 2)
        # relaunch and redump

When it happens

Trigger: Profiling a kernel that emits more trace events than the ProfilerSpec buffer capacity (entries_per_warpgroup set too small for the number of record() calls per warp).

Common situations: Deeply nested instrumented regions or looped record() calls inside the kernel while keeping the default buffer size.

Related errors

• Allocated too many names
• Block size is not a multiple of
• must be positive
• Profiler collected an odd number of trace events. This…
• Scope not supported

AI-assisted analysis of jax-ml/jax@1e1c6a8fc0 (2026-08-27). Data as JSON: /api/errors/827645e3011758b5. Report an issue: GitHub.

Appendix: source

Thrown at jax/experimental/mosaic/gpu/profiler.py:264

      return name_id
    name_id = self.interned_names[name] = len(self.interned_names)
    if name_id & self.EXIT:
      raise RuntimeError("Allocated too many names")
    return name_id

  def dump(self, buffer, f, grid: tuple[int, ...], block: tuple[int, ...]):
    buffer = np.asarray(buffer)
    num_blocks = math.prod(grid)
    traces_per_block = self._num_traces((), block)
    entries = buffer.reshape(
        num_blocks, traces_per_block, self.entries_per_warpgroup
    )
    start_times = entries[..., 0]
    sm_ids = entries[..., 1]
    traces_used = entries[..., 2]
    entries_used = traces_used + 3
    if np.any(entries_used > self.entries_per_warpgroup):
      raise RuntimeError("Insufficient space to capture a full trace")
    traces = entries[..., 3:]

    # Estimate the overhead of profiling.
    time_events = traces[:, :, 1::2]
    valid_times_mask = np.arange(traces.shape[-1])[1::2] < traces_used[..., None]
    # 12 cycles is a ballpark estimate for H100
    profiling_overhead = (time_events[:, :, 1:] - time_events[:, :, :-1]).min(
        where=valid_times_mask[:, :, 1:], initial=12
    )
    profiling_overhead = max(0, profiling_overhead - 1)

    unintern = {v: k for k, v in self.interned_names.items()}
    events = []
    for block_idx, trace_idx in np.ndindex(num_blocks, traces_per_block):
      valid_entries = traces_used[block_idx, trace_idx]
      local_clock_offset = None
      if valid_entries % 2:
        raise RuntimeError(

View on GitHub (pinned to 1e1c6a8fc0)