can1357/oh-my-pi · error · GitCommandError

124

124

Error message

git timed out after {timeout:.0f}s

What it means

_run() is the sandbox's internal wrapper around subprocess git invocations with a hard timeout. When git exceeds the timeout, subprocess.TimeoutExpired is caught and re-raised as GitCommandError with the conventional exit code 124 and the message 'git timed out after Ns'. This converts hangs (network stalls, filesystem locks) into a bounded, retryable failure instead of a wedged dispatcher thread.

Source

Thrown at python/robomp/src/sandbox.py:390

def _run(
    cmd: list[str],
    *,
    cwd: Path | None = None,
    timeout: float | None = _DEFAULT_SANDBOX_SUBPROCESS_TIMEOUT,
) -> subprocess.CompletedProcess[str]:
    """Legacy raising helper (still used by a sandbox test). Forwards to subprocess.run."""
    try:
        proc = subprocess.run(
            cmd,
            cwd=str(cwd) if cwd else None,
            check=False,
            capture_output=True,
            text=True,
            timeout=timeout,
        )
    except subprocess.TimeoutExpired as exc:
        raise GitCommandError(cmd, 124, "", f"git timed out after {timeout:.0f}s") from exc
    if proc.returncode != 0:
        raise GitCommandError(cmd, proc.returncode, proc.stdout, proc.stderr)
    return proc


def _worktree_add(add_cmd: list[str], *, pool: Path, repo_dir: Path) -> None:
    """Run `git worktree add`, cleaning partial state on failure.

    A worktree-add killed mid-operation (the 120s `_run` timeout surfaces as
    GitCommandError 124, or any nonzero git failure) can leave a partial
    checkout at `repo_dir` and/or a dangling pool worktree registration. Left
    behind, the event retry hits stale metadata and fails again on the same
    path. Best-effort remove the checkout and prune the pool, then re-raise so the
    retry starts from a clean path. If the prune itself fails (incl. a 124
    timeout), raise that instead — chained from the add error — since a
    dangling registration left behind is exactly what poisons the retry.
    """
    try:

View on GitHub (pinned to 9690622007)

Solutions

  1. Look for a stale lock: check `<pool>/.git/index.lock` and `<pool>/.git/worktrees/*/locked`; remove only if no git process is live, then retry.
  2. Increase the timeout if the repo is legitimately large (check the timeout parameter / settings) — 124 with slow-but-progressing ops means the limit is too tight.
  3. Retry: the dispatcher requeues failed events, and transient network stalls usually clear.
  4. If worktree add specifically hangs, prune stale worktree metadata first: `git worktree prune` in the pool.

Example fix

# before
manager.ensure_workspace(...)  # GitCommandError code=124 after 120s on a huge repo
# after
# raise the timeout via settings/env, or pre-fetch outside the timeout path
transport.fetch_pool(repo=repo, pool_dir=pool)
manager.ensure_workspace(...)  # local-only worktree add now completes quickly
Defensive patterns

Strategy: retry

Try / catch

try:
    manager.ensure_workspace(...)
except GitCommandError as e:
    if getattr(e, "returncode", None) == 124:
        schedule_retry(event, backoff=60)  # transient hang; queue will re-run
    else:
        mark_failed(event, e)

Prevention

When it happens

Trigger: Any sandbox git operation routed through _run — including `git worktree add` via _worktree_add, and `git branch -m` — that exceeds the configured timeout because git is waiting on a network fetch, an index.lock held by another process, or a hung filesystem (NFS, full disk).

Common situations: git worktree add triggering an on-demand fetch from a slow or unreachable remote; a crashed process leaving index.lock in the shared pool clone; disk I/O saturation on the host running many concurrent workspaces.

Understand the failure class

Related errors

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/b92956e61dd7e5f0. Report an issue: GitHub.