{"record":{"id":"229573a0fabd27f4","repo":"jax-ml/jax","slug":"profiler-collected-an-odd-number-of-trace-events","errorCode":null,"errorMessage":"Profiler collected an odd number of trace events. This likely indicates memory corruption due to insufficient profiling space. Try again with a larger profiling space.","messagePattern":"Profiler collected an odd number of trace events\\. This likely indicates memory corruption due to insufficient profiling space\\. Try again with a larger profiling space\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"jax/experimental/mosaic/gpu/profiler.py","lineNumber":282,"sourceCode":"      raise RuntimeError(\"Insufficient space to capture a full trace\")\n    traces = entries[..., 3:]\n\n    # Estimate the overhead of profiling.\n    time_events = traces[:, :, 1::2]\n    valid_times_mask = np.arange(traces.shape[-1])[1::2] < traces_used[..., None]\n    # 12 cycles is a ballpark estimate for H100\n    profiling_overhead = (time_events[:, :, 1:] - time_events[:, :, :-1]).min(\n        where=valid_times_mask[:, :, 1:], initial=12\n    )\n    profiling_overhead = max(0, profiling_overhead - 1)\n\n    unintern = {v: k for k, v in self.interned_names.items()}\n    events = []\n    for block_idx, trace_idx in np.ndindex(num_blocks, traces_per_block):\n      valid_entries = traces_used[block_idx, trace_idx]\n      local_clock_offset = None\n      if valid_entries % 2:\n        raise RuntimeError(\n            \"Profiler collected an odd number of trace events. This likely \"\n            \"indicates memory corruption due to insufficient profiling space. \"\n            \"Try again with a larger profiling space.\"\n        )\n      start_time = start_times[block_idx, trace_idx]\n      block_events = []\n      last_time = float(\"-inf\")\n      for i in range(0, valid_entries, 2):\n        tag = traces[block_idx, trace_idx, i]\n        time = traces[block_idx, trace_idx, i + 1]\n        if local_clock_offset is None:\n          local_clock_offset = time\n        time -= local_clock_offset\n        time -= (i // 2) * profiling_overhead  # Account for the overhead of profiling.\n        if time < 0:\n          break  # Detect a timer wraparound\n        name_id = tag\n        begin = True","sourceCodeStart":264,"sourceCodeEnd":300,"githubUrl":"https://github.com/jax-ml/jax/blob/1e1c6a8fc06dfcd1247076ec5cae4640cea5d7bb/jax/experimental/mosaic/gpu/profiler.py#L264-L300","documentation":"Trace events are stored as (name_id, timestamp) pairs, so a valid trace always has an even event count. An odd count means the tail of the buffer was partially clobbered by newer writes — the classic symptom of insufficient profiling space — and dump() aborts instead of returning corrupt data.","triggerScenarios":"Same root cause as the 'insufficient space' error but caught differently: buffer happened to be overwritten in a way that leaves an odd number of used entries (e.g. overflow occurring mid-event).","commonSituations":"Kernels with variable per-warp event counts that occasionally exceed capacity; increasing concurrency so some warps overflow.","solutions":["Enlarge the profiling buffer (entries_per_warpgroup) as the message suggests","Decrease the number of record() events per warp"],"exampleFix":"# before\nspec = ProfilerSpec(trace_scope=ThreadSubset.WARP)\n\n# after\nspec = ProfilerSpec(trace_scope=ThreadSubset.WARP, entries_per_warpgroup=2048)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    spec.dump(buffer, f, grid, block)\nexcept RuntimeError as e:\n    if 'odd number of trace events' in str(e):\n        # grow buffer and retry once\n        ...","preventionTips":["Treat odd-event-count errors as overflow; enlarge entries_per_warpgroup and rerun","Keep event counts deterministic per warp"],"tags":["profiling","mosaic","gpu","memory-corruption"],"backgroundTag":"profiler-buffer-overflow","analyzedSha":"1e1c6a8fc06dfcd1247076ec5cae4640cea5d7bb","analyzedAt":"2026-08-27T09:53:25.647Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}