can1357/oh-my-pi · warning · ManualTriageTimeout

{delivery_id} did not reach a terminal state within {timeout

Error message

{delivery_id} did not reach a terminal state within {timeout_seconds:g}s (state={state})

What it means

ManualTriageTimeout raised by await_terminal_state when the delivery event did not reach a terminal state (done/failed/skipped) within the given timeout. The message includes the elapsed timeout and the state last observed, so callers know whether it is still queued or running.

Source

Thrown at python/robomp/src/manual_triage.py:149

    Pure DB polling — the caller MUST NOT spawn its own ``WorkerPool``; the
    long-lived ``serve`` process is the only owner of the dispatcher loop.
    Returns the final row, or ``None`` if the row was deleted while waiting.
    Raises ``ManualTriageTimeout`` if ``timeout`` elapses first.
    """
    deadline = None if timeout is None else time.monotonic() + timeout
    while True:
        row = db.get_event(delivery_id)
        if row is None:
            return None
        if row.state in _TERMINAL_STATES:
            return row

        sleep_for = poll_interval
        if deadline is not None:
            remaining = deadline - time.monotonic()
            if remaining <= 0:
                assert timeout is not None
                raise ManualTriageTimeout(delivery_id, row.state, timeout)
            sleep_for = min(poll_interval, remaining)
        await asyncio.sleep(sleep_for)


__all__ = [
    "InvalidIssueRef",
    "ManualTriageError",
    "ManualTriageConflict",
    "ManualTriageTimeout",
    "await_terminal_state",
    "build_issues_opened_payload",
    "enqueue_manual_triage",
    "manual_delivery_id",
    "parse_issue_ref",
]

View on GitHub (pinned to 9690622007)

Solutions

Increase the `timeout` (or extend the deadline) and poll again
Inspect the event row's state and logs to see if the run is progressing or wedged
If the run is stuck in a non-terminal state, cancel/reset it and re-enqueue

Example fix

// before
await await_terminal_state(db, delivery_id, timeout=30)
// after
await await_terminal_state(db, delivery_id, timeout=300)

Defensive patterns

Strategy: try-catch

Validate before calling

# ensure the delivery exists and is progressing before waiting
row = db.get_event(delivery_id)
if row is None or row.state in ("done", "failed", "skipped"):
    return row.state if row else "missing"

Type guard

TERMINAL = {"done", "failed", "skipped"}
def is_terminal(state: str) -> bool:
    return state in TERMINAL

Try / catch

try:
    await await_terminal_state(db, delivery_id, timeout=timeout_s)
except ManualTriageTimeout as e:
    log.warning("triage %s still %s after %ss", delivery_id, e.args, timeout_s)
    # inspect/cancel stuck run, or re-poll with a longer timeout

Prevention

Set timeout generously relative to expected triage duration
Monitor worker liveness so crashed runs don't stay 'running' forever
Cancel/reset events stuck in non-terminal states before re-enqueueing

When it happens

Trigger: Polling with await_terminal_state (from _go or _wait) while the triage run is slow, stuck, or the poll deadline expires before a terminal transition.

Common situations: Triage agent taking longer than the configured timeout, worker crashed leaving the event in "running" forever, too-short timeout for large repos.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/c1be19ad3fe5d4b5. Report an issue: GitHub.