{"record":{"id":"cd8a1c123c02725f","repo":"can1357/oh-my-pi","slug":"omp-task-exceeded-hard-timeout","errorCode":null,"errorMessage":"omp task exceeded hard timeout","messagePattern":"omp task exceeded hard timeout","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"python/robomp/src/worker.py","lineNumber":773,"sourceCode":"\n            hard_timer = threading.Timer(hard_timeout_seconds, _hard_stop)\n            hard_timer.daemon = True\n            hard_timer.start()\n            try:\n                turn = _drive_turn(\n                    client,\n                    prompt,\n                    task_kind=task_kind,\n                    inputs=inputs,\n                    bindings=bindings,\n                    tools_called=tools_called,\n                )\n                if turn is None:\n                    return None\n            finally:\n                hard_timer.cancel()\n            if hard_timeout_fired.is_set():\n                raise TimeoutError(\"omp task exceeded hard timeout\")\n            if turn is not None and turn.assistant_message is not None:\n                stop_reason = turn.assistant_message.get(\"stopReason\")\n                if stop_reason == \"error\":\n                    error_msg = turn.assistant_message.get(\"errorMessage\") or \"model returned error\"\n                    raise RuntimeError(f\"omp agent error (stopReason=error): {error_msg}\")\n            log.info(\n                \"rpc_done\",\n                extra={\n                    \"issue\": bindings.issue_key,\n                    \"task\": task_kind,\n                    \"messages\": len(turn.messages),\n                    \"events\": len(turn.events),\n                },\n            )\n            return turn.assistant_text\n        finally:\n            unregister_cancel_hook()\n","sourceCodeStart":755,"sourceCodeEnd":791,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/python/robomp/src/worker.py#L755-L791","documentation":"run_task's RPC driver raises TimeoutError('omp task exceeded hard timeout') when the threading.Timer armed at soft task_timeout + ROBOMP_TASK_TIMEOUT_HARD_GRACE_SECONDS fires and _hard_stop cancelled the omp subprocess before the turn returned. The flag hard_timeout_fired is checked after _drive_turn returns, so the task ran longer than the hard ceiling and was force-killed.","triggerScenarios":"An omp agent turn (fix, triage, review, release CI repair) runs past task_timeout + hard grace — e.g. a runaway model loop, many tool calls, or a hung subprocess whose cancellation hook fired. The timer callback sets the flag and calls _cancel_hook; when the turn eventually returns/None the TimeoutError is raised.","commonSituations":"Model stuck in retry loops on flaky tools; very large issue requiring long runs; slow LLM gateway; task_timeout set too low for the task class; omp subprocess hung on a blocking bash tool call inside the worktree.","solutions":["Raise the timeout via ROBOMP_TASK_TIMEOUT (or the per-task-kind override) and/or ROBOMP_TASK_TIMEOUT_HARD_GRACE_SECONDS if legitimate tasks are being cut off.","Inspect the worktree and events.last_error / rpc_hard_timeout log line to see what the agent was doing when killed.","Check the omp gateway/model latency; a slow or repeatedly-erroring provider stretches runs.","Retry the delivery via POST /api/trigger — resumed sessions continue prior reasoning, so the rerun often finishes faster.","If a specific task kind always times out, simplify its prompt or split the work; don't just keep raising limits."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// before dispatching, check the effective ceiling\nconst soft = Number(process.env.ROBOMP_TASK_TIMEOUT ?? 0);\nconst grace = Number(process.env.ROBOMP_TASK_TIMEOUT_HARD_GRACE_SECONDS ?? 0);\nif (soft > 0 && soft + grace < expectedMinutes * 60) console.warn('task timeout too low for this task kind');","typeGuard":null,"tryCatchPattern":"try {\n  result = await runTask(...);\n} catch (e) {\n  if (e instanceof TimeoutError && e.message.includes('hard timeout')) {\n    log.warn('omp task hit hard timeout; consider raising ROBOMP_TASK_TIMEOUT and retrying');\n    await requeueDelivery(deliveryId);\n  } else throw e;\n}","preventionTips":["Size ROBOMP_TASK_TIMEOUT (and the hard grace) to the slowest realistic task kind.","Watch for 'rpc_hard_timeout' log lines to spot systematically slow tasks.","Ensure the LLM gateway is healthy; provider latency is a common root cause.","Retry failed deliveries — resumed sessions continue prior reasoning and usually finish sooner.","Keep task prompts scoped so the agent cannot loop indefinitely on one step."],"tags":["timeout","worker","omp","subprocess"],"backgroundTag":"task-timeout-exceeded","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}