{"record":{"id":"6e81eb6fad8da023","repo":"sgl-project/sglang","slug":"numa-node-node-has-no-cpu-cores-allowed-by-the-c","errorCode":null,"errorMessage":"NUMA node {node} has no CPU cores allowed by the current affinity {sorted(allowed_cpus)}, skipping NUMA binding{gpu_str}.","messagePattern":"NUMA node (.+?) has no CPU cores allowed by the current affinity (.+?), skipping NUMA binding(.+?)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"warning","filePath":"python/sglang/srt/utils/numa_utils.py","lineNumber":337,"sourceCode":"    \"\"\"Emit the NUMA-bind failure warning, or raise it when\n    ``SGLANG_CRASH_ON_NUMA_BIND_FAILURE`` is set.\n\n    Two call modes:\n      * ``reason is None`` (default): the failure is an empty CPU intersection,\n        so the message reports ``allowed_cpus`` (which must be provided).\n      * ``reason`` provided: the failure is something else (e.g. numactl rejected\n        the binding at runtime); the caller supplies the exact message and\n        ``allowed_cpus`` / ``gpu_id`` are not needed.\n    \"\"\"\n    if reason is None:\n        gpu_str = f\" for GPU {gpu_id}\" if gpu_id is not None else \"\"\n        reason = (\n            f\"NUMA node {node} has no CPU cores allowed by the current affinity \"\n            f\"{sorted(allowed_cpus)}, skipping NUMA binding{gpu_str}.\"\n        )\n    logger.warning(reason)\n    if envs.SGLANG_CRASH_ON_NUMA_BIND_FAILURE.get():\n        raise RuntimeError(reason)\n\n\ndef _can_set_mempolicy() -> bool:\n    \"\"\"Check if the process has permission to use NUMA memory policy syscalls.\"\"\"\n    try:\n        libnuma = get_libnuma()\n        if libnuma is None or libnuma.numa_available() < 0:\n            return False\n        mode = ctypes.c_int()\n        ret = libnuma.get_mempolicy(\n            ctypes.byref(mode), None, ctypes.c_ulong(0), None, ctypes.c_ulong(0)\n        )\n        return ret == 0\n    except Exception:\n        return False\n\n\ndef _is_numa_available() -> bool:","sourceCodeStart":319,"sourceCodeEnd":355,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/numa_utils.py#L319-L355","documentation":"Emitted when NUMA binding is requested for a node whose CPU set intersects emptiness with the process's allowed affinity mask — the node has no usable cores, so binding is skipped (logged as a warning). Only if SGLANG_CRASH_ON_NUMA_BIND_FAILURE is set does it raise RuntimeError instead.","triggerScenarios":"Calling numa_bind_to_node / configure_subprocess with --numa-node N while taskset/cgroup/affinity restricts the process to cores outside node N; SGLANG_CRASH_ON_NUMA_BIND_FAILURE=1 turns the warning into a crash.","commonSituations":"Containers or SLURM jobs with restricted cpusets, taskset -c pinning to one NUMA node while requesting another, or hybrid CPU partitions on multi-socket hosts.","solutions":["Request a NUMA node whose cores overlap the allowed affinity (check /proc/self/status Cpus_allowed vs node cpulist in /sys/devices/system/node/nodeN/cpulist)","Widen the cpuset (taskset -pc, container cpuset config) to include the target node","Unset SGLANG_CRASH_ON_NUMA_BIND_FAILURE if a warning-and-continue behavior is acceptable","Drop the --numa-node flag to let the scheduler place freely"],"exampleFix":"# before\nSGLANG_CRASH_ON_NUMA_BIND_FAILURE=1 taskset -c 0-7 python -m sglang.launch_server --numa-node 1 ...\n# after\ntaskset -c 0-63 python -m sglang.launch_server --numa-node 0 ...","handlingStrategy":"fallback","validationCode":"def numa_node_has_allowed_cores(node: int) -> bool:\n    try:\n        allowed = {int(c) for c in open('/proc/self/status').read().split('Cpus_allowed_list:')[1].split()[0].replace('-',':').split(',') for c in [c]} if False else parse_cpus_allowed()\n        node_cpus = set(range(*[int(x) for x in open(f'/sys/devices/system/node/node{node}/cpulist').read().strip().split('-')]))\n        return bool(allowed & node_cpus)\n    except OSError:\n        return False","typeGuard":"null","tryCatchPattern":"try:\n    numa_bind_to_node(node)\nexcept RuntimeError as e:\n    if 'no CPU cores allowed' in str(e):\n        logger.warning('skipping NUMA bind: %s', e)  # run unbound\n    else:\n        raise","preventionTips":["Check Cpus_allowed_list against node cpulists before requesting --numa-node","Keep SGLANG_CRASH_ON_NUMA_BIND_FAILURE unset unless strictness is required","In containers, align cpuset with the requested NUMA node"],"tags":["numa","cpu-affinity","cgroup","sglang","resource-binding"],"backgroundTag":"numa-affinity-binding-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}