jax-ml/jax · error · RuntimeError

mTLS for the JAX distributed service requires jaxlib 0.11.2…

Error message

mTLS for the JAX distributed service requires jaxlib 0.11.2 or newer.

What it means

The distributed service's mTLS options (cert/key/CA files, verify_secure_credentials) only exist in jaxlib >= 0.11.2 (extension version >= 483). Requesting any of them on an older jaxlib raises this RuntimeError from _get_mtls_kwargs during jax.distributed.initialize.

Solutions

1. Upgrade jaxlib (and jax) to >= 0.11.2: pip install -U jax jaxlib
2. If you cannot upgrade, omit all mTLS kwargs and verify_secure_credentials and use plain TCP for the coordinator
3. Verify versions: python -c "import jaxlib; print(jaxlib.__version__)" and check extension version >= 483

Example fix

# before (old jaxlib)
jax.distributed.initialize(..., mtls_cert_file='c.pem', mtls_key_file='k.pem')
# after
pip install -U "jax[cpu]>=0.11.2" "jaxlib>=0.11.2"
jax.distributed.initialize(..., mtls_cert_file='c.pem', mtls_key_file='k.pem')

from jaxlib import extension_version if False else None
import jax._src.lib as jlib
MTLS_OK = jlib.jaxlib_extension_version >= 483
assert MTLS_OK or not any([cert, key, ca, verify]), 'jaxlib too old for mTLS'

When it happens

Trigger: Calling jax.distributed.initialize with mtls_cert_file/mtls_key_file/mtls_ca_file or verify_secure_credentials=True while the installed jaxlib extension version is below 483.

Common situations: Pinning old jaxlib in requirements while using new jax; installing jax via a stale pip environment or older CUDA wheel; security-hardened clusters requiring mTLS with an outdated toolchain.

Related errors

• is not available; available versions are
• Cannot deserialize DisabledSafetyCheck with unknown kind
• coordinator_address should be defined.
• coordinator_bind_address should be defined.
• Denominator should be reduced along the same axes numerator…

AI-assisted analysis of jax-ml/jax@1e1c6a8fc0 (2026-08-27). Data as JSON: /api/errors/4e128737e0037c05. Report an issue: GitHub.

Appendix: source

Thrown at jax/_src/distributed.py:84

  kwargs: dict[str, Any] = dict(
      mtls_cert_file=value_or_config(
          mtls_cert_file, config.mtls_cert_file),
      mtls_key_file=value_or_config(
          mtls_key_file, config.mtls_key_file),
      mtls_ca_file=value_or_config(
          mtls_ca_file, config.mtls_ca_file),
      mtls_peer_uri_prefix=value_or_config(
          mtls_peer_uri_prefix, config.mtls_peer_uri_prefix),
      verify_secure_credentials=value_or_config(
          verify_secure_credentials,
          config.distributed_verify_secure_credentials),
  )
  if jaxlib_extension_version < 483:
    verify_secure_credentials = kwargs.pop('verify_secure_credentials')
    if (verify_secure_credentials
        or any(v is not None for v in kwargs.values())):
      raise RuntimeError('mTLS for the JAX distributed service requires '
                         'jaxlib 0.11.2 or newer.')
    return {}
  return kwargs


class State:
  process_id: int = 0
  num_processes: int = 1
  service: _jax.DistributedRuntimeService | Any | None = None
  client: _jax.DistributedRuntimeClient | Any | None = None
  preemption_sync_manager: Any | None = None
  coordinator_address: str | None = None
  partition_index: int | None = None

  def initialize(self,
                 coordinator_address: str | None = None,
                 num_processes: int | None = None,
                 process_id: int | None = None,

View on GitHub (pinned to 1e1c6a8fc0)