sgl-project/sglang · error · RuntimeError

kill_process_tree: {len(alive)} process(es) not reaped withi

Error message

kill_process_tree: {len(alive)} process(es) not reaped within {wait_timeout}s after SIGKILL; pids={[p.pid for p in alive]}

What it means

kill_process_tree sent SIGKILL to a process group but _still_holding_resources still reports processes holding resources after wait_timeout seconds; the API raises instead of returning silently so callers know cleanup is incomplete (typical of unkillable D-state processes stuck in CUDA/driver IO).

Source

Thrown at python/sglang/srt/utils/common.py:2197

    SIGKILL is asynchronous -- children hold GPU context, pinned memory and
    fds until the kernel reaps them. Raise on timeout so a stuck process
    surfaces instead of leaving a latent race.

    Polls /proc via is_running()/status() rather than psutil.wait_procs, whose
    os.pidfd_open path (used for non-child procs) raises OSError(EINVAL) against
    a just-killed process on some kernels and aborts the whole wait.
    """
    warn_at = min(10.0, wait_timeout / 2)
    deadline = time.monotonic() + wait_timeout
    warn_deadline = time.monotonic() + warn_at
    warned = False
    while True:
        alive = _still_holding_resources(procs)
        if not alive:
            return
        now = time.monotonic()
        if now >= deadline:
            raise RuntimeError(
                f"kill_process_tree: {len(alive)} process(es) not reaped within "
                f"{wait_timeout}s after SIGKILL; pids={[p.pid for p in alive]}"
            )
        if not warned and now >= warn_deadline:
            logger.warning(
                "kill_process_tree: %d process(es) still alive after %.1fs SIGKILL; "
                "continuing to wait up to %.1fs total. pids=%s",
                len(alive),
                warn_at,
                wait_timeout,
                [p.pid for p in alive],
            )
            warned = True
        time.sleep(0.1)


def kill_process_tree(
    parent_pid,

View on GitHub (pinned to 0132848349)

Solutions

1. Check nvidia-smi / ps for D-state processes and their wchan; if stuck in kernel IO, only host-level action clears them
2. Increase wait_timeout passed to kill_process_tree to tolerate slow driver teardown
3. Upgrade GPU driver/NCCL if wedges recur; use CUDA_LAUNCH_BLOCKING or coredumps to find the stuck call
4. In containers, ensure PID 1 reaps children (init: true / --pid=host as appropriate)

Example fix

// before
kill_process_tree(pid)
// after
kill_process_tree(pid, wait_timeout=30.0)

try:
    kill_process_tree(proc.pid, wait_timeout=30.0)
except RuntimeError as e:
    log_and_alert(e); escalate_to_host_cleanup()

When it happens

Trigger: Scheduler/tokenizer subprocesses stuck in uninterruptible driver calls (nccl collective, cudaMalloc, GPU DMA) that ignore SIGKILL; zombie GPU contexts on multi-GPU runs.

Common situations: Crashed multi-GPU servers where NCCL communicators wedge, containers with PID-namespace quirks, or defective GPU drivers leaving processes in D state.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Error happened when batch testing peer-to-peer access from {
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `
• {name}_block_cnt and {name}_block_idx must be on the same de
• {name}_block tensors must live on CUDA
• {name} must live on CUDA

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9a85fa725409b724. Report an issue: GitHub.