can1357/oh-my-pi · error · GitCommandError

GitCommandError: git worktree prune failed (incl. 124 timeou

Error message

GitCommandError: git worktree prune failed (incl. 124 timeout)

What it means

When a worktree add fails or times out, the cleanup path runs `git worktree prune` in the pool to clear the dangling registration. If prune also fails (including the 120s timeout surfaced as returncode 124), a GitCommandError is raised — deliberately, so the cleanup event retries instead of recording success while stale metadata still blocks the next `git worktree add`.

Source

Thrown at python/robomp/src/sandbox.py:1218

                if repo_dir.exists():
                    removed = _safe_run(["git", "worktree", "remove", "--force", str(repo_dir)], cwd=pool)
                    if removed.returncode != 0:
                        shutil.rmtree(repo_dir, ignore_errors=True)
                        needs_prune = True
                elif ws_root.exists():
                    # Checkout gone but the workspace root remains -> a prior op
                    # was killed mid-flight and may have left a dangling
                    # registration. A fully-cleaned workspace has no ws_root, so
                    # a plain repeat close prunes nothing.
                    needs_prune = True
                if needs_prune:
                    pruned = _safe_run(["git", "worktree", "prune"], cwd=pool)
                    if pruned.returncode != 0:
                        # Prune is the step that clears the dangling registration.
                        # If it fails (incl. a 124 timeout), report it so the
                        # cleanup event retries instead of recording success with
                        # stale metadata still blocking the next add.
                        raise GitCommandError(
                            ["git", "worktree", "prune"], pruned.returncode, pruned.stdout, pruned.stderr
                        )
            if ws_root.exists():
                shutil.rmtree(ws_root, ignore_errors=True)

    def reclaim_workspace_caches(self, *, repo: str, number: int | str) -> bool:
        """Strip re-creatable dependency caches from an idle workspace.

        Every task run reinstalls ``node_modules`` (see
        ``host_tools.ensure_workspace_dependencies``), so between runs the
        checkout's ``node_modules`` and the workspace-private bun install
        cache are dead weight — multiple GB per issue that would otherwise
        persist until the issue closes, which is exactly how the host runs
        out of disk. ``--continue`` resumes are unaffected: session
        transcripts, context, artifacts and the worktree survive.

        The rename pass runs under the per-repo lock (serialized against
        ``ensure_workspace``); the slow deletes happen after the lock is

View on GitHub (pinned to 9690622007)

Solutions

Retry the cleanup event — the error exists precisely so cleanup retries; transient locks often clear.
Check for a stale index.lock or hung git process in the pool dir and kill/remove it.
Ensure only one orchestrator runs against /data; stop duplicates.
As last resort, remove the pool clone directory and let the next event re-clone.

Defensive patterns

Strategy: retry

Type guard

def is_prune_timeout(exc: BaseException) -> bool:
    return isinstance(exc, GitCommandError) and exc.returncode == 124

Try / catch

try:
    manager.remove_workspace(repo=repo, number=n)
except GitCommandError as e:
    if e.returncode == 124:
        # inspect pool for hung git processes / locks, then re-enqueue cleanup
        ...
    else:
        raise

Prevention

Ensure single-orchestrator ownership of /data
Gracefully stop containers instead of SIGKILL to avoid partial worktree adds
Alert on cleanup events that retry repeatedly — indicates a persistent pool lock

When it happens

Trigger: remove_workspace()/cleanup on a pool where a prior `git worktree add` was killed mid-operation and the follow-up prune fails — e.g. pool repo locked by another process, prune timing out after 120s (returncode 124), or a corrupt pool worktree metadata (.git/worktrees).

Common situations: Container killed (SIGKILL/OOM) during workspace provisioning leaving a partial worktree; two orchestrator instances sharing one /data volume; a hung git process on a stalled filesystem causing the 124 timeout.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/1134ea5c27c3a147. Report an issue: GitHub.