What causes "Codex compatibility-check process group could not be reclaimed" in ZhuLinsen/daily_stock_analysis?

Inspect leftover processes: `ps -eo pid,pgid,stat,cmd | grep -i codex` and look for D-state or traced processes; kill them manually (`kill -9 - `).

ZhuLinsen/daily_stock_analysis · error · CodexAppServerError

resource_cleanup_failed

resource_cleanup_failed

Error message

Codex compatibility-check process group could not be reclaimed

What it means

Raised by _terminate_probe_process_group (src/services/agent_backend_status_service.py:78) with code 'resource_cleanup_failed' after the Codex compatibility-check probe process group survived SIGTERM, a grace wait, SIGKILL, and a further 2.0s kill grace (_PROBE_KILL_GRACE_SECONDS). The service spawns the Codex CLI in its own process group and must reclaim both the launcher and its native child; if os.killpg(pid, 0) still signals a live group afterwards, this RuntimeError (CodexAppServerError) fires to surface leaked processes instead of silently continuing.

Source

Thrown at src/services/agent_backend_status_service.py:78

    if process.poll() is None or _probe_process_group_alive(process_group_id):
        try:
            os.killpg(process_group_id, signal.SIGKILL)
        except ProcessLookupError:
            pass
        except PermissionError:
            if process.poll() is None:
                process.kill()
        kill_deadline = time.monotonic() + _PROBE_KILL_GRACE_SECONDS
        if process.poll() is None:
            try:
                process.wait(timeout=max(0.0, kill_deadline - time.monotonic()))
            except subprocess.TimeoutExpired:
                pass
        while _probe_process_group_alive(process_group_id) and time.monotonic() < kill_deadline:
            time.sleep(0.02)

    if process.poll() is None or _probe_process_group_alive(process_group_id):
        raise CodexAppServerError(
            "resource_cleanup_failed",
            "Codex compatibility-check process group could not be reclaimed",
        )


def _run_codex_probe(
    command: list[str],
    *,
    timeout: float,
    capture_output: bool = False,
) -> subprocess.CompletedProcess:
    """Run one bounded CLI probe without orphaning the native Codex child."""
    process = subprocess.Popen(
        command,
        stdout=subprocess.PIPE if capture_output else subprocess.DEVNULL,
        stderr=subprocess.PIPE if capture_output else subprocess.DEVNULL,
        text=capture_output,
        env=controlled_environment(),

View on GitHub (pinned to 5159bd72e8)

Solutions

Inspect leftover processes: `ps -eo pid,pgid,stat,cmd | grep -i codex` and look for D-state or traced processes; kill them manually (`kill -9 -<pgid>`).
If a debugger/tracer is attached, detach it so SIGKILL can be delivered.
In restricted containers, ensure the process has CAP_KILL over its own children or run without a pid namespace restriction on signals.
Upgrade/replace a Codex CLI version known to hang, and check the probe timeout that precedes cleanup.
Catch CodexAppServerError with code 'resource_cleanup_failed' and report backend status as degraded rather than crashing the status service.

Example fix

# before
status = run_codex_compatibility_check()  # may raise, kills the API call

# after
try:
    status = run_codex_compatibility_check()
except CodexAppServerError as exc:
    if exc.args[0] == 'resource_cleanup_failed' or 'resource_cleanup_failed' in str(exc):
        status = {'available': False, 'reason': 'codex probe could not be reclaimed'}
    else:
        raise

Defensive patterns

Strategy: try-catch

Try / catch

from src.agent.codex_app_server_transport import CodexAppServerError
try:
    result = run_codex_compatibility_check()
except CodexAppServerError as exc:
    if 'resource_cleanup_failed' in str(exc):
        # surface degraded status; optionally log leaked pgid for manual reaping
        result = {'available': False, 'reason': 'probe_process_leak'}
    else:
        raise

Prevention

Keep the Codex CLI version current; hangs that survive SIGKILL are usually a stale binary or driver stall.
Avoid running probes under debuggers/tracers in production; they defer fatal signals.
In containers, ensure signal permissions over the probe's process group.
Alert an operator when this fires — it means processes are leaking, which compounds.

When it happens

Trigger: Running a Codex backend compatibility check where the CLI's native child is stuck in an uninterruptible state (D-state on I/O), is being traced/debugged (SIGKILL deferred), or was re-parented into a new process group so killpg misses it; also possible on permission-restricted environments where SIGKILL to the group fails with PermissionError.

Common situations: Containers/ci sandboxes with pid limits or seccomp filters blocking signals to process groups; a hung Codex binary (filesystem stall, GPU driver hang); running under strace/gdb which traps signals; zombie children kept alive by a supervisor.

Related errors

AI-assisted analysis of ZhuLinsen/daily_stock_analysis@5159bd72e8 (2026-08-15). Data as JSON: /api/errors/adfd707b2fda6a6f. Report an issue: GitHub.