jax-ml/jax · error · ValueError

Vector clock size ({self.vector_clock_size}) must be greater

Error message

Vector clock size ({self.vector_clock_size}) must be greater than the total number of cores/threads ({num_cores_or_threads}).

What it means

Raised by Mosaic TPU interpret-mode parameter handling when an explicitly configured vector_clock_size is not strictly greater than the total number of simulated cores/threads (num_devices * num_cores_or_threads). The interpret mode simulates parallel execution with vector clocks for race detection; there must be more clock slots than simulated threads.

Source

Thrown at jax/_src/pallas/mosaic/interpret/params.py:110

  def __post_init__(self):
    if self.num_cores_or_threads < 1:
      raise ValueError(
          "Number of cores or threads must be at least 1, but got"
          f" {self.num_cores_or_threads}."
      )
    if self.vector_clock_size is not None and self.vector_clock_size < 1:
      # Further validation is done in `get_vector_clock_size` below.
      raise ValueError(
          "Vector clock size must be at least 1, but got"
          f" {self.vector_clock_size}."
      )

  def get_vector_clock_size(self, num_devices) -> int:
    """Returns the number of vector clocks to use for TPU interpret mode.`"""
    num_cores_or_threads = num_devices * self.num_cores_or_threads
    if self.vector_clock_size is not None:
      if num_cores_or_threads >= self.vector_clock_size:
        raise ValueError(
            f"Vector clock size ({self.vector_clock_size}) must be greater than"
            f" the total number of cores/threads ({num_cores_or_threads})."
        )
      return self.vector_clock_size
    else:
      # Default to twice the total number of cores/threads.
      return 2 * num_cores_or_threads


@dataclasses.dataclass(frozen=True, kw_only=True)
class InterpretParams(SharedInterpretParams):
  """Parameters for TPU interpret mode.

  TPU interpret mode is a way run Pallas TPU kernels on CPU, while simulating
  a TPU's shared memory (HBM, VMEM, etc.), communication (remote and local
  DMAs), and synchronization operations (semaphores, barriers, etc.).  This mode
  is intended for debugging and testing.

View on GitHub (pinned to 1e1c6a8fc0)

Solutions

1. Omit vector_clock_size (set it to None) so interpret mode defaults to 2x the total cores/threads
2. Set vector_clock_size to a value strictly greater than num_devices * num_cores_or_threads (e.g. 2x + 1)
3. Reduce num_cores_or_threads or the simulated num_devices

Example fix

// before
params = mosaic_interpret.InterpretParams(num_cores_or_threads=4, vector_clock_size=4)
// after
params = mosaic_interpret.InterpretParams(num_cores_or_threads=4, vector_clock_size=None)  # auto: 2x total

total = num_devices * params.num_cores_or_threads
assert params.vector_clock_size is None or params.vector_clock_size > total, 'vector_clock_size must exceed total cores/threads'

When it happens

Trigger: Calling interpret-mode Pallas/Mosaic kernels with InterpretParams(vector_clock_size=N) where N <= num_devices * num_cores_or_threads, e.g. running multi-device simulation with a small explicit clock size.

Common situations: Copying interpret params configured for 1 device/1 core and then simulating multiple devices; explicitly setting vector_clock_size while also raising num_cores_or_threads; version changes that altered the multiplication semantics.

Related errors

• position {position} is out of range for clock {self.clock}
• Acc ref must be at least 2D, got shape {shape}
• Out-of-bounds read of ({device_id} {local_core_id} {memory_s
• masked load_p
• run_scoped_p with collective axes is not supported

AI-assisted analysis of jax-ml/jax@1e1c6a8fc0 (2026-08-27). Data as JSON: /api/errors/18025949e1447046. Report an issue: GitHub.