{"record":{"id":"d6b9f5f4c32cff6a","repo":"ytti/oxidized","slug":"cmd-timed-out","errorCode":null,"errorMessage":"#{@cmd} timed out","messagePattern":"#(.+?) timed out","errorType":"exception","errorClass":"Timeout::Error","httpStatus":null,"severity":"error","filePath":"lib/oxidized/hook/exec.rb","lineNumber":56,"sourceCode":"  end\n\n  def run_cmd!(env)\n    pid = nil\n    status = nil\n    Timeout.timeout(@timeout) do\n      pid = spawn env, @cmd, unsetenv_others: true\n      pid, status = wait2 pid\n      unless status.exitstatus.zero?\n        msg = \"#{@cmd.inspect} failed with exit value #{status.exitstatus}\"\n        logger.error msg\n        raise msg\n      end\n    end\n  rescue Timeout::Error\n    kill \"TERM\", pid\n    msg = \"#{@cmd} timed out\"\n    logger.error msg\n    raise Timeout::Error, msg\n  end\n\n  def make_env(ctx)\n    env = {\n      \"OX_EVENT\" => ctx.event.to_s\n    }\n    if ctx.node\n      env.merge!(\n        \"OX_NODE_NAME\"      => ctx.node.name.to_s,\n        \"OX_NODE_IP\"        => ctx.node.ip.to_s,\n        \"OX_NODE_FROM\"      => ctx.node.from.to_s,\n        \"OX_NODE_MSG\"       => ctx.node.msg.to_s,\n        \"OX_NODE_GROUP\"     => ctx.node.group.to_s,\n        \"OX_NODE_MODEL\"     => ctx.node.model.class.name,\n        \"OX_REPO_COMMITREF\" => ctx.commitref.to_s,\n        \"OX_REPO_NAME\"      => ctx.node.repo.to_s,\n        \"OX_ERR_TYPE\"       => ctx.node.err_type.to_s,\n        \"OX_ERR_REASON\"     => ctx.node.err_reason.to_s","sourceCodeStart":38,"sourceCodeEnd":74,"githubUrl":"https://github.com/ytti/oxidized/blob/687ed4262d9d21e54662e872e2383386d8498f7b/lib/oxidized/hook/exec.rb#L38-L74","documentation":"The exec hook spawns cmd with Process.spawn inside Timeout.timeout(@timeout), default 60 seconds (lib/oxidized/hook/exec.rb:43-45). If the child process has not exited when the budget runs out, the rescue sends SIGTERM to the child pid, logs the failure, and re-raises Timeout::Error with the message '<cmd> timed out'. In the default synchronous mode the exception propagates out of run_hook and fails the node job; with async: true it is swallowed inside the worker thread and only logged.","triggerScenarios":"Any exec hook command that runs longer than the configured timeout: a script that waits on stdin (the child is spawned with unsetenv_others and no TTY), a curl/git push against an unreachable host, a script blocked on a lock, or a default 60s timeout with a legitimately slow command.","commonSituations":"Backup/push scripts whose runtime grows as the repo grows; scripts that prompt for a password; network-dependent commands during outages; dockerized oxidized shelling out to slow wrappers.","solutions":["Raise the budget in the hook config: add timeout: 300 (or enough seconds for the worst case)","Make the command fast and non-interactive: add internal limits (curl --max-time, ssh -o ConnectTimeout, retries with backoff) and remove any prompt for input","Set async: true when the hook result should not fail the node job (the exception is then only raised inside the thread)","Handle SIGTERM in the script so the killed child can clean up partial state"],"exampleFix":"# before\nhooks:\n  exec_hook:\n    type: exec\n    cmd: /usr/local/bin/push-repo.sh\n\n# after\nhooks:\n  exec_hook:\n    type: exec\n    timeout: 300\n    cmd: /usr/local/bin/push-repo.sh","handlingStrategy":"try-catch","validationCode":"# smoke-test the hook command under the same budget and scrubbed env before enabling it\nrequire 'timeout'\nTimeout.timeout(300) do\n  system({ 'OX_EVENT' => 'post_store' }, '/usr/local/bin/push-repo.sh', unsetenv_others: true) or raise 'hook cmd failed or is too slow'\nend","typeGuard":null,"tryCatchPattern":"begin\n  hook.run_hook(ctx)\nrescue Timeout::Error => e\n  logger.warn \"hook #{hook.class.name} timed out: #{e.message}; continuing without it\"\nend","preventionTips":["Set hook timeouts 2-3x the script's p95 runtime and alert when runs approach the limit","Never let hook scripts read stdin; pass everything through the OX_* environment variables","Put internal limits (curl --max-time, ssh -o ConnectTimeout) on every network call the script makes","Handle SIGTERM in the script so timeouts kill it cleanly"],"tags":["oxidized","exec-hook","timeout","sigterm","subprocess"],"backgroundTag":"command-timeout","analyzedSha":"687ed4262d9d21e54662e872e2383386d8498f7b","analyzedAt":"2026-08-23T11:19:42.084Z","schemaVersion":2},"datasetVersion":"2026-08-23T16:17:53.355Z"}