What causes "omp task exceeded hard timeout" in can1357/oh-my-pi?

Raise the timeout via ROBOMP_TASK_TIMEOUT (or the per-task-kind override) and/or ROBOMP_TASK_TIMEOUT_HARD_GRACE_SECONDS if legitimate tasks are being cut off.

can1357/oh-my-pi · error · TimeoutError

omp task exceeded hard timeout

Error message

omp task exceeded hard timeout

What it means

run_task's RPC driver raises TimeoutError('omp task exceeded hard timeout') when the threading.Timer armed at soft task_timeout + ROBOMP_TASK_TIMEOUT_HARD_GRACE_SECONDS fires and _hard_stop cancelled the omp subprocess before the turn returned. The flag hard_timeout_fired is checked after _drive_turn returns, so the task ran longer than the hard ceiling and was force-killed.

Source

Thrown at python/robomp/src/worker.py:773

            hard_timer = threading.Timer(hard_timeout_seconds, _hard_stop)
            hard_timer.daemon = True
            hard_timer.start()
            try:
                turn = _drive_turn(
                    client,
                    prompt,
                    task_kind=task_kind,
                    inputs=inputs,
                    bindings=bindings,
                    tools_called=tools_called,
                )
                if turn is None:
                    return None
            finally:
                hard_timer.cancel()
            if hard_timeout_fired.is_set():
                raise TimeoutError("omp task exceeded hard timeout")
            if turn is not None and turn.assistant_message is not None:
                stop_reason = turn.assistant_message.get("stopReason")
                if stop_reason == "error":
                    error_msg = turn.assistant_message.get("errorMessage") or "model returned error"
                    raise RuntimeError(f"omp agent error (stopReason=error): {error_msg}")
            log.info(
                "rpc_done",
                extra={
                    "issue": bindings.issue_key,
                    "task": task_kind,
                    "messages": len(turn.messages),
                    "events": len(turn.events),
                },
            )
            return turn.assistant_text
        finally:
            unregister_cancel_hook()

View on GitHub (pinned to 9690622007)

Solutions

Raise the timeout via ROBOMP_TASK_TIMEOUT (or the per-task-kind override) and/or ROBOMP_TASK_TIMEOUT_HARD_GRACE_SECONDS if legitimate tasks are being cut off.
Inspect the worktree and events.last_error / rpc_hard_timeout log line to see what the agent was doing when killed.
Check the omp gateway/model latency; a slow or repeatedly-erroring provider stretches runs.
Retry the delivery via POST /api/trigger — resumed sessions continue prior reasoning, so the rerun often finishes faster.
If a specific task kind always times out, simplify its prompt or split the work; don't just keep raising limits.

Defensive patterns

Strategy: try-catch

Validate before calling

// before dispatching, check the effective ceiling
const soft = Number(process.env.ROBOMP_TASK_TIMEOUT ?? 0);
const grace = Number(process.env.ROBOMP_TASK_TIMEOUT_HARD_GRACE_SECONDS ?? 0);
if (soft > 0 && soft + grace < expectedMinutes * 60) console.warn('task timeout too low for this task kind');

Try / catch

try {
  result = await runTask(...);
} catch (e) {
  if (e instanceof TimeoutError && e.message.includes('hard timeout')) {
    log.warn('omp task hit hard timeout; consider raising ROBOMP_TASK_TIMEOUT and retrying');
    await requeueDelivery(deliveryId);
  } else throw e;
}

Prevention

Size ROBOMP_TASK_TIMEOUT (and the hard grace) to the slowest realistic task kind.
Watch for 'rpc_hard_timeout' log lines to spot systematically slow tasks.
Ensure the LLM gateway is healthy; provider latency is a common root cause.
Retry failed deliveries — resumed sessions continue prior reasoning and usually finish sooner.
Keep task prompts scoped so the agent cannot loop indefinitely on one step.

When it happens

Trigger: An omp agent turn (fix, triage, review, release CI repair) runs past task_timeout + hard grace — e.g. a runaway model loop, many tool calls, or a hung subprocess whose cancellation hook fired. The timer callback sets the flag and calls _cancel_hook; when the turn eventually returns/None the TimeoutError is raised.

Common situations: Model stuck in retry loops on flaky tools; very large issue requiring long runs; slow LLM gateway; task_timeout set too low for the task class; omp subprocess hung on a blocking bash tool call inside the worktree.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/cd8a1c123c02725f. Report an issue: GitHub.