jax-ml/jax · error · ValueError

process_id and num_processes must be nonnegative, with…

Error message

process_id and num_processes must be nonnegative, with process_id < num_processes. Got process_id={process_id}, num_processes={num_processes}.

What it means

Cross-validation in jax.distributed.initialize: the rank (process_id) must lie strictly within the pool, 0 <= process_id < num_processes. A rank equal to or exceeding the pool size means the process would have no valid position in the collective.

Solutions

  1. Ensure rank and world size come from the same launcher source: int(os.environ['RANK']) and int(os.environ['WORLD_SIZE'])
  2. Add a launch-time assertion: assert 0 <= rank < world_size
  3. Unset stale job env vars between runs (env -i or fresh shells)

Example fix

# before
rank, world = int(os.environ['RANK']), 8  # hardcoded, RANK may be 8
# after
rank = int(os.environ['RANK']); world = int(os.environ['WORLD_SIZE'])
assert 0 <= rank < world
jax.distributed.initialize(coordinator_address=addr,
    num_processes=world, process_id=rank)
Defensive patterns

Strategy: validation

Validate before calling

rank, world = int(os.environ['RANK']), int(os.environ['WORLD_SIZE'])
assert 0 <= rank < world

Prevention

When it happens

Trigger: process_id=8 with num_processes=8; negative ranks; inconsistent rank/world-size env vars, e.g. RANK from a different launcher than WORLD_SIZE.

Common situations: Mixing launcher env vars (torchrun RANK with a hardcoded world size); off-by-one in manual rank assignment; env leakage between sequential jobs in CI.

Related errors


AI-assisted analysis of jax-ml/jax@1e1c6a8fc0 (2026-08-27). Data as JSON: /api/errors/6d72197133444ff1. Report an issue: GitHub.

Appendix: source

Thrown at jax/_src/distributed.py:149

              cluster_detection_method,
              initialization_timeout,
          )
      )

    if coordinator_address is None:
      raise ValueError('coordinator_address should be defined.')
    if num_processes is None:
      raise ValueError('Number of processes must be defined.')
    if process_id is None:
      raise ValueError('The process id of the current process must be defined.')
    if not isinstance(process_id, int):
      raise TypeError("process_id must be a nonnegative int. "
                      f"Got process_id={process_id} of type {type(process_id)}.")
    if not isinstance(num_processes, int):
      raise TypeError("num_processes must be a positive int. "
                      f"Got num_processes={num_processes} of type {type(num_processes)}.")
    if not (0 <= process_id < num_processes):
      raise ValueError("process_id and num_processes must be nonnegative, with process_id < num_processes. "
                       f"Got process_id={process_id}, num_processes={num_processes}.")

    self.coordinator_address = coordinator_address

    # The default value of [::]:port tells the coordinator to bind to all
    # available addresses on the same port as coordinator_address.
    default_coordinator_bind_address = '[::]:' + coordinator_address.rsplit(':', 1)[1]
    coordinator_bind_address = (coordinator_bind_address or
                                os.environ.get('JAX_COORDINATOR_BIND_ADDRESS',
                                               default_coordinator_bind_address))
    if coordinator_bind_address is None:
      raise ValueError('coordinator_bind_address should be defined.')

    if local_device_ids:
      visible_devices = ','.join(str(x) for x in local_device_ids)
      logger.info('JAX distributed initialized with visible devices: %s', visible_devices)
      config.update("jax_cuda_visible_devices", visible_devices)
      config.update("jax_rocm_visible_devices", visible_devices)

View on GitHub (pinned to 1e1c6a8fc0)