{"record":{"id":"9a85fa725409b724","repo":"sgl-project/sglang","slug":"kill-process-tree-len-alive-process-es-not-re","errorCode":null,"errorMessage":"kill_process_tree: {len(alive)} process(es) not reaped within {wait_timeout}s after SIGKILL; pids={[p.pid for p in alive]}","messagePattern":"kill_process_tree: (.+?) process\\(es\\) not reaped within (.+?)s after SIGKILL; pids=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/common.py","lineNumber":2197,"sourceCode":"    SIGKILL is asynchronous -- children hold GPU context, pinned memory and\n    fds until the kernel reaps them. Raise on timeout so a stuck process\n    surfaces instead of leaving a latent race.\n\n    Polls /proc via is_running()/status() rather than psutil.wait_procs, whose\n    os.pidfd_open path (used for non-child procs) raises OSError(EINVAL) against\n    a just-killed process on some kernels and aborts the whole wait.\n    \"\"\"\n    warn_at = min(10.0, wait_timeout / 2)\n    deadline = time.monotonic() + wait_timeout\n    warn_deadline = time.monotonic() + warn_at\n    warned = False\n    while True:\n        alive = _still_holding_resources(procs)\n        if not alive:\n            return\n        now = time.monotonic()\n        if now >= deadline:\n            raise RuntimeError(\n                f\"kill_process_tree: {len(alive)} process(es) not reaped within \"\n                f\"{wait_timeout}s after SIGKILL; pids={[p.pid for p in alive]}\"\n            )\n        if not warned and now >= warn_deadline:\n            logger.warning(\n                \"kill_process_tree: %d process(es) still alive after %.1fs SIGKILL; \"\n                \"continuing to wait up to %.1fs total. pids=%s\",\n                len(alive),\n                warn_at,\n                wait_timeout,\n                [p.pid for p in alive],\n            )\n            warned = True\n        time.sleep(0.1)\n\n\ndef kill_process_tree(\n    parent_pid,","sourceCodeStart":2179,"sourceCodeEnd":2215,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/common.py#L2179-L2215","documentation":"kill_process_tree sent SIGKILL to a process group but _still_holding_resources still reports processes holding resources after wait_timeout seconds; the API raises instead of returning silently so callers know cleanup is incomplete (typical of unkillable D-state processes stuck in CUDA/driver IO).","triggerScenarios":"Scheduler/tokenizer subprocesses stuck in uninterruptible driver calls (nccl collective, cudaMalloc, GPU DMA) that ignore SIGKILL; zombie GPU contexts on multi-GPU runs.","commonSituations":"Crashed multi-GPU servers where NCCL communicators wedge, containers with PID-namespace quirks, or defective GPU drivers leaving processes in D state.","solutions":["Check nvidia-smi / ps for D-state processes and their wchan; if stuck in kernel IO, only host-level action clears them","Increase wait_timeout passed to kill_process_tree to tolerate slow driver teardown","Upgrade GPU driver/NCCL if wedges recur; use CUDA_LAUNCH_BLOCKING or coredumps to find the stuck call","In containers, ensure PID 1 reaps children (init: true / --pid=host as appropriate)"],"exampleFix":"// before\nkill_process_tree(pid)\n// after\nkill_process_tree(pid, wait_timeout=30.0)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    kill_process_tree(proc.pid, wait_timeout=30.0)\nexcept RuntimeError as e:\n    log_and_alert(e); escalate_to_host_cleanup()","preventionTips":["Use longer wait_timeout on multi-GPU NCCL jobs","Run an init process in containers to reap children","Monitor for D-state processes and alert before they wedge shutdown"],"tags":["process-management","sigkill","subprocess","cuda","zombie-process"],"backgroundTag":"process-cleanup-timeout","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}