{"record":{"id":"827645e3011758b5","repo":"jax-ml/jax","slug":"insufficient-space-to-capture-a-full-trace","errorCode":null,"errorMessage":"Insufficient space to capture a full trace","messagePattern":"Insufficient space to capture a full trace","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"jax/experimental/mosaic/gpu/profiler.py","lineNumber":264,"sourceCode":"      return name_id\n    name_id = self.interned_names[name] = len(self.interned_names)\n    if name_id & self.EXIT:\n      raise RuntimeError(\"Allocated too many names\")\n    return name_id\n\n  def dump(self, buffer, f, grid: tuple[int, ...], block: tuple[int, ...]):\n    buffer = np.asarray(buffer)\n    num_blocks = math.prod(grid)\n    traces_per_block = self._num_traces((), block)\n    entries = buffer.reshape(\n        num_blocks, traces_per_block, self.entries_per_warpgroup\n    )\n    start_times = entries[..., 0]\n    sm_ids = entries[..., 1]\n    traces_used = entries[..., 2]\n    entries_used = traces_used + 3\n    if np.any(entries_used > self.entries_per_warpgroup):\n      raise RuntimeError(\"Insufficient space to capture a full trace\")\n    traces = entries[..., 3:]\n\n    # Estimate the overhead of profiling.\n    time_events = traces[:, :, 1::2]\n    valid_times_mask = np.arange(traces.shape[-1])[1::2] < traces_used[..., None]\n    # 12 cycles is a ballpark estimate for H100\n    profiling_overhead = (time_events[:, :, 1:] - time_events[:, :, :-1]).min(\n        where=valid_times_mask[:, :, 1:], initial=12\n    )\n    profiling_overhead = max(0, profiling_overhead - 1)\n\n    unintern = {v: k for k, v in self.interned_names.items()}\n    events = []\n    for block_idx, trace_idx in np.ndindex(num_blocks, traces_per_block):\n      valid_entries = traces_used[block_idx, trace_idx]\n      local_clock_offset = None\n      if valid_entries % 2:\n        raise RuntimeError(","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/jax-ml/jax/blob/1e1c6a8fc06dfcd1247076ec5cae4640cea5d7bb/jax/experimental/mosaic/gpu/profiler.py#L246-L282","documentation":"When dumping the trace buffer, dump() checks that traces_used + 3 (header entries) fits within entries_per_warpgroup. If a kernel recorded more events than the buffer was sized for, the extra events were silently overwritten and the dump is unreliable, so it raises.","triggerScenarios":"Profiling a kernel that emits more trace events than the ProfilerSpec buffer capacity (entries_per_warpgroup set too small for the number of record() calls per warp).","commonSituations":"Deeply nested instrumented regions or looped record() calls inside the kernel while keeping the default buffer size.","solutions":["Increase the profiling buffer size (entries per warpgroup) in ProfilerSpec","Reduce the number of profiled regions/events per warp"],"exampleFix":"# before\nspec = ProfilerSpec(trace_scope=ThreadSubset.WARP)  # default capacity\n\n# after\nspec = ProfilerSpec(trace_scope=ThreadSubset.WARP, entries_per_warpgroup=1024)","handlingStrategy":"validation","validationCode":"estimated = max_events_per_warp()  # your kernel's count\nif estimated + 3 > spec.entries_per_warpgroup:\n    spec = ProfilerSpec(trace_scope=spec.trace_scope,\n                        entries_per_warpgroup=estimated * 2 + 8)","typeGuard":null,"tryCatchPattern":"try:\n    spec.dump(buffer, f, grid, block)\nexcept RuntimeError as e:\n    if 'Insufficient space' in str(e):\n        spec = ProfilerSpec(trace_scope=spec.trace_scope,\n                            entries_per_warpgroup=spec.entries_per_warpgroup * 2)\n        # relaunch and redump","preventionTips":["Over-provision the trace buffer (2x worst-case events)","Count record() calls per warp when designing instrumentation"],"tags":["profiling","mosaic","gpu","buffer-overflow"],"backgroundTag":"profiler-buffer-overflow","analyzedSha":"1e1c6a8fc06dfcd1247076ec5cae4640cea5d7bb","analyzedAt":"2026-08-27T09:53:25.647Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}