{"record":{"id":"adfd707b2fda6a6f","repo":"ZhuLinsen/daily_stock_analysis","slug":"resource-cleanup-failed-adfd70","errorCode":"resource_cleanup_failed","errorMessage":"Codex compatibility-check process group could not be reclaimed","messagePattern":"Codex compatibility-check process group could not be reclaimed","errorType":"exception","errorClass":"CodexAppServerError","httpStatus":null,"severity":"error","filePath":"src/services/agent_backend_status_service.py","lineNumber":78,"sourceCode":"    if process.poll() is None or _probe_process_group_alive(process_group_id):\n        try:\n            os.killpg(process_group_id, signal.SIGKILL)\n        except ProcessLookupError:\n            pass\n        except PermissionError:\n            if process.poll() is None:\n                process.kill()\n        kill_deadline = time.monotonic() + _PROBE_KILL_GRACE_SECONDS\n        if process.poll() is None:\n            try:\n                process.wait(timeout=max(0.0, kill_deadline - time.monotonic()))\n            except subprocess.TimeoutExpired:\n                pass\n        while _probe_process_group_alive(process_group_id) and time.monotonic() < kill_deadline:\n            time.sleep(0.02)\n\n    if process.poll() is None or _probe_process_group_alive(process_group_id):\n        raise CodexAppServerError(\n            \"resource_cleanup_failed\",\n            \"Codex compatibility-check process group could not be reclaimed\",\n        )\n\n\ndef _run_codex_probe(\n    command: list[str],\n    *,\n    timeout: float,\n    capture_output: bool = False,\n) -> subprocess.CompletedProcess:\n    \"\"\"Run one bounded CLI probe without orphaning the native Codex child.\"\"\"\n    process = subprocess.Popen(\n        command,\n        stdout=subprocess.PIPE if capture_output else subprocess.DEVNULL,\n        stderr=subprocess.PIPE if capture_output else subprocess.DEVNULL,\n        text=capture_output,\n        env=controlled_environment(),","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/ZhuLinsen/daily_stock_analysis/blob/5159bd72e8373d215492dff122acc9d389e219c9/src/services/agent_backend_status_service.py#L60-L96","documentation":"Raised by _terminate_probe_process_group (src/services/agent_backend_status_service.py:78) with code 'resource_cleanup_failed' after the Codex compatibility-check probe process group survived SIGTERM, a grace wait, SIGKILL, and a further 2.0s kill grace (_PROBE_KILL_GRACE_SECONDS). The service spawns the Codex CLI in its own process group and must reclaim both the launcher and its native child; if os.killpg(pid, 0) still signals a live group afterwards, this RuntimeError (CodexAppServerError) fires to surface leaked processes instead of silently continuing.","triggerScenarios":"Running a Codex backend compatibility check where the CLI's native child is stuck in an uninterruptible state (D-state on I/O), is being traced/debugged (SIGKILL deferred), or was re-parented into a new process group so killpg misses it; also possible on permission-restricted environments where SIGKILL to the group fails with PermissionError.","commonSituations":"Containers/ci sandboxes with pid limits or seccomp filters blocking signals to process groups; a hung Codex binary (filesystem stall, GPU driver hang); running under strace/gdb which traps signals; zombie children kept alive by a supervisor.","solutions":["Inspect leftover processes: `ps -eo pid,pgid,stat,cmd | grep -i codex` and look for D-state or traced processes; kill them manually (`kill -9 -<pgid>`).","If a debugger/tracer is attached, detach it so SIGKILL can be delivered.","In restricted containers, ensure the process has CAP_KILL over its own children or run without a pid namespace restriction on signals.","Upgrade/replace a Codex CLI version known to hang, and check the probe timeout that precedes cleanup.","Catch CodexAppServerError with code 'resource_cleanup_failed' and report backend status as degraded rather than crashing the status service."],"exampleFix":"# before\nstatus = run_codex_compatibility_check()  # may raise, kills the API call\n\n# after\ntry:\n    status = run_codex_compatibility_check()\nexcept CodexAppServerError as exc:\n    if exc.args[0] == 'resource_cleanup_failed' or 'resource_cleanup_failed' in str(exc):\n        status = {'available': False, 'reason': 'codex probe could not be reclaimed'}\n    else:\n        raise","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"from src.agent.codex_app_server_transport import CodexAppServerError\ntry:\n    result = run_codex_compatibility_check()\nexcept CodexAppServerError as exc:\n    if 'resource_cleanup_failed' in str(exc):\n        # surface degraded status; optionally log leaked pgid for manual reaping\n        result = {'available': False, 'reason': 'probe_process_leak'}\n    else:\n        raise","preventionTips":["Keep the Codex CLI version current; hangs that survive SIGKILL are usually a stale binary or driver stall.","Avoid running probes under debuggers/tracers in production; they defer fatal signals.","In containers, ensure signal permissions over the probe's process group.","Alert an operator when this fires — it means processes are leaking, which compounds."],"tags":["process","cleanup","subprocess","signals","codex"],"backgroundTag":null,"analyzedSha":"5159bd72e8373d215492dff122acc9d389e219c9","analyzedAt":"2026-08-15T01:59:36.292Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}