tinyhumansai/openhuman · error · SubmitError

pool worker job timed out (hard deadline; worker wedged)

Error message

pool worker job timed out (hard deadline; worker wedged)

What it means

Thrown from the pool worker submit path when awaiting the worker's response line exceeds the caller-supplied hard deadline (tokio::time::timeout_at). It fires when the reused/repurposed OS worker process is wedged (hung child process, deadlocked tool run) and never writes a completion line. Because the request bytes were already flushed into the worker's stdin, the job may have started executing, so unlike a pre-flush write failure this is a terminal SubmitError::post and the same job must not be silently re-run.

Source

Thrown at src/openhuman/runtime/pool/worker.rs:246

        // reused idle worker died) → the job did not run → safe to retry.
        self.stdin.write_all(line.as_bytes()).await.map_err(|e| {
            SubmitError::pre(anyhow::Error::new(e).context("writing pool job request"))
        })?;
        // Past this point the request bytes are in the pipe: the job may execute,
        // so any later failure is terminal (never re-run the same job).
        self.stdin.flush().await.map_err(|e| {
            SubmitError::post(anyhow::Error::new(e).context("flushing pool job request"))
        })?;

        // Fixed deadline: `continue`ing over unparseable / mismatched-id lines
        // must NOT reset the wedged-worker timeout, so it bounds the total wait.
        let deadline = hard_timeout.map(|t| tokio::time::Instant::now() + t);
        loop {
            let next = match deadline {
                Some(dl) => match tokio::time::timeout_at(dl, self.responses.next_line()).await {
                    Ok(inner) => inner,
                    Err(_) => {
                        return Err(SubmitError::post(anyhow::anyhow!(
                            "pool worker job timed out (hard deadline; worker wedged)"
                        )))
                    }
                },
                None => self.responses.next_line().await,
            };
            let line = match next {
                Ok(Some(line)) => line,
                Ok(None) => {
                    return Err(SubmitError::post(anyhow::anyhow!(
                        "pool worker closed stdout"
                    )))
                }
                Err(error) => {
                    return Err(SubmitError::post(
                        anyhow::Error::new(error).context("reading pool job response"),
                    ))
                }

View on GitHub (pinned to 7491200858)

Solutions

1. Surface the error to the caller as terminal; do not transparently resubmit the same job, since the wedged worker may still be executing it.
2. Investigate or restart the wedged worker process (kill/reap the pool worker) before issuing new jobs.
3. Tune the hard timeout passed into submit if legitimate jobs of this kind routinely exceed it.

When it happens

Trigger: Thrown at src/openhuman/runtime/pool/worker.rs:246 when the library encounters an invalid state.

Common situations: See trigger scenarios.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

AI-assisted analysis of tinyhumansai/openhuman@7491200858 (2026-08-17). Data as JSON: /api/errors/00174ba5203b0887. Report an issue: GitHub.