sgl-project/sglang · warning · RuntimeError

NUMA node {node} has no CPU cores allowed by the current aff

Error message

NUMA node {node} has no CPU cores allowed by the current affinity {sorted(allowed_cpus)}, skipping NUMA binding{gpu_str}.

What it means

Emitted when NUMA binding is requested for a node whose CPU set intersects emptiness with the process's allowed affinity mask — the node has no usable cores, so binding is skipped (logged as a warning). Only if SGLANG_CRASH_ON_NUMA_BIND_FAILURE is set does it raise RuntimeError instead.

Source

Thrown at python/sglang/srt/utils/numa_utils.py:337

    """Emit the NUMA-bind failure warning, or raise it when
    ``SGLANG_CRASH_ON_NUMA_BIND_FAILURE`` is set.

    Two call modes:
      * ``reason is None`` (default): the failure is an empty CPU intersection,
        so the message reports ``allowed_cpus`` (which must be provided).
      * ``reason`` provided: the failure is something else (e.g. numactl rejected
        the binding at runtime); the caller supplies the exact message and
        ``allowed_cpus`` / ``gpu_id`` are not needed.
    """
    if reason is None:
        gpu_str = f" for GPU {gpu_id}" if gpu_id is not None else ""
        reason = (
            f"NUMA node {node} has no CPU cores allowed by the current affinity "
            f"{sorted(allowed_cpus)}, skipping NUMA binding{gpu_str}."
        )
    logger.warning(reason)
    if envs.SGLANG_CRASH_ON_NUMA_BIND_FAILURE.get():
        raise RuntimeError(reason)


def _can_set_mempolicy() -> bool:
    """Check if the process has permission to use NUMA memory policy syscalls."""
    try:
        libnuma = get_libnuma()
        if libnuma is None or libnuma.numa_available() < 0:
            return False
        mode = ctypes.c_int()
        ret = libnuma.get_mempolicy(
            ctypes.byref(mode), None, ctypes.c_ulong(0), None, ctypes.c_ulong(0)
        )
        return ret == 0
    except Exception:
        return False


def _is_numa_available() -> bool:

View on GitHub (pinned to 0132848349)

Solutions

  1. Request a NUMA node whose cores overlap the allowed affinity (check /proc/self/status Cpus_allowed vs node cpulist in /sys/devices/system/node/nodeN/cpulist)
  2. Widen the cpuset (taskset -pc, container cpuset config) to include the target node
  3. Unset SGLANG_CRASH_ON_NUMA_BIND_FAILURE if a warning-and-continue behavior is acceptable
  4. Drop the --numa-node flag to let the scheduler place freely

Example fix

# before
SGLANG_CRASH_ON_NUMA_BIND_FAILURE=1 taskset -c 0-7 python -m sglang.launch_server --numa-node 1 ...
# after
taskset -c 0-63 python -m sglang.launch_server --numa-node 0 ...
Defensive patterns

Strategy: fallback

Validate before calling

def numa_node_has_allowed_cores(node: int) -> bool:
    try:
        allowed = {int(c) for c in open('/proc/self/status').read().split('Cpus_allowed_list:')[1].split()[0].replace('-',':').split(',') for c in [c]} if False else parse_cpus_allowed()
        node_cpus = set(range(*[int(x) for x in open(f'/sys/devices/system/node/node{node}/cpulist').read().strip().split('-')]))
        return bool(allowed & node_cpus)
    except OSError:
        return False

Type guard

null

Try / catch

try:
    numa_bind_to_node(node)
except RuntimeError as e:
    if 'no CPU cores allowed' in str(e):
        logger.warning('skipping NUMA bind: %s', e)  # run unbound
    else:
        raise

Prevention

When it happens

Trigger: Calling numa_bind_to_node / configure_subprocess with --numa-node N while taskset/cgroup/affinity restricts the process to cores outside node N; SGLANG_CRASH_ON_NUMA_BIND_FAILURE=1 turns the warning into a crash.

Common situations: Containers or SLURM jobs with restricted cpusets, taskset -c pinning to one NUMA node while requesting another, or hybrid CPU partitions on multi-socket hosts.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6e81eb6fad8da023. Report an issue: GitHub.