{"record":{"id":"00174ba5203b0887","repo":"tinyhumansai/openhuman","slug":"pool-worker-job-timed-out-hard-deadline-worker-w","errorCode":null,"errorMessage":"pool worker job timed out (hard deadline; worker wedged)","messagePattern":"pool worker job timed out \\(hard deadline; worker wedged\\)","errorType":"exception","errorClass":"SubmitError","httpStatus":null,"severity":"error","filePath":"src/openhuman/runtime/pool/worker.rs","lineNumber":246,"sourceCode":"        // reused idle worker died) → the job did not run → safe to retry.\n        self.stdin.write_all(line.as_bytes()).await.map_err(|e| {\n            SubmitError::pre(anyhow::Error::new(e).context(\"writing pool job request\"))\n        })?;\n        // Past this point the request bytes are in the pipe: the job may execute,\n        // so any later failure is terminal (never re-run the same job).\n        self.stdin.flush().await.map_err(|e| {\n            SubmitError::post(anyhow::Error::new(e).context(\"flushing pool job request\"))\n        })?;\n\n        // Fixed deadline: `continue`ing over unparseable / mismatched-id lines\n        // must NOT reset the wedged-worker timeout, so it bounds the total wait.\n        let deadline = hard_timeout.map(|t| tokio::time::Instant::now() + t);\n        loop {\n            let next = match deadline {\n                Some(dl) => match tokio::time::timeout_at(dl, self.responses.next_line()).await {\n                    Ok(inner) => inner,\n                    Err(_) => {\n                        return Err(SubmitError::post(anyhow::anyhow!(\n                            \"pool worker job timed out (hard deadline; worker wedged)\"\n                        )))\n                    }\n                },\n                None => self.responses.next_line().await,\n            };\n            let line = match next {\n                Ok(Some(line)) => line,\n                Ok(None) => {\n                    return Err(SubmitError::post(anyhow::anyhow!(\n                        \"pool worker closed stdout\"\n                    )))\n                }\n                Err(error) => {\n                    return Err(SubmitError::post(\n                        anyhow::Error::new(error).context(\"reading pool job response\"),\n                    ))\n                }","sourceCodeStart":228,"sourceCodeEnd":264,"githubUrl":"https://github.com/tinyhumansai/openhuman/blob/749120085864ce16e0f273c7b86fac7740b39c5b/src/openhuman/runtime/pool/worker.rs#L228-L264","documentation":"Thrown from the pool worker submit path when awaiting the worker's response line exceeds the caller-supplied hard deadline (tokio::time::timeout_at). It fires when the reused/repurposed OS worker process is wedged (hung child process, deadlocked tool run) and never writes a completion line. Because the request bytes were already flushed into the worker's stdin, the job may have started executing, so unlike a pre-flush write failure this is a terminal SubmitError::post and the same job must not be silently re-run.","triggerScenarios":"Thrown at src/openhuman/runtime/pool/worker.rs:246 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Surface the error to the caller as terminal; do not transparently resubmit the same job, since the wedged worker may still be executing it.","Investigate or restart the wedged worker process (kill/reap the pool worker) before issuing new jobs.","Tune the hard timeout passed into submit if legitimate jobs of this kind routinely exceed it."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"749120085864ce16e0f273c7b86fac7740b39c5b","analyzedAt":"2026-08-17T21:21:45.363Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}