ytti/oxidized · error · Timeout::Error

#{@cmd} timed out

Error message

#{@cmd} timed out

What it means

The exec hook spawns cmd with Process.spawn inside Timeout.timeout(@timeout), default 60 seconds (lib/oxidized/hook/exec.rb:43-45). If the child process has not exited when the budget runs out, the rescue sends SIGTERM to the child pid, logs the failure, and re-raises Timeout::Error with the message '<cmd> timed out'. In the default synchronous mode the exception propagates out of run_hook and fails the node job; with async: true it is swallowed inside the worker thread and only logged.

Source

Thrown at lib/oxidized/hook/exec.rb:56

  end

  def run_cmd!(env)
    pid = nil
    status = nil
    Timeout.timeout(@timeout) do
      pid = spawn env, @cmd, unsetenv_others: true
      pid, status = wait2 pid
      unless status.exitstatus.zero?
        msg = "#{@cmd.inspect} failed with exit value #{status.exitstatus}"
        logger.error msg
        raise msg
      end
    end
  rescue Timeout::Error
    kill "TERM", pid
    msg = "#{@cmd} timed out"
    logger.error msg
    raise Timeout::Error, msg
  end

  def make_env(ctx)
    env = {
      "OX_EVENT" => ctx.event.to_s
    }
    if ctx.node
      env.merge!(
        "OX_NODE_NAME"      => ctx.node.name.to_s,
        "OX_NODE_IP"        => ctx.node.ip.to_s,
        "OX_NODE_FROM"      => ctx.node.from.to_s,
        "OX_NODE_MSG"       => ctx.node.msg.to_s,
        "OX_NODE_GROUP"     => ctx.node.group.to_s,
        "OX_NODE_MODEL"     => ctx.node.model.class.name,
        "OX_REPO_COMMITREF" => ctx.commitref.to_s,
        "OX_REPO_NAME"      => ctx.node.repo.to_s,
        "OX_ERR_TYPE"       => ctx.node.err_type.to_s,
        "OX_ERR_REASON"     => ctx.node.err_reason.to_s

View on GitHub (pinned to 687ed4262d)

Solutions

1. Raise the budget in the hook config: add timeout: 300 (or enough seconds for the worst case)
2. Make the command fast and non-interactive: add internal limits (curl --max-time, ssh -o ConnectTimeout, retries with backoff) and remove any prompt for input
3. Set async: true when the hook result should not fail the node job (the exception is then only raised inside the thread)
4. Handle SIGTERM in the script so the killed child can clean up partial state

Example fix

# before
hooks:
  exec_hook:
    type: exec
    cmd: /usr/local/bin/push-repo.sh

# after
hooks:
  exec_hook:
    type: exec
    timeout: 300
    cmd: /usr/local/bin/push-repo.sh

# smoke-test the hook command under the same budget and scrubbed env before enabling it
require 'timeout'
Timeout.timeout(300) do
  system({ 'OX_EVENT' => 'post_store' }, '/usr/local/bin/push-repo.sh', unsetenv_others: true) or raise 'hook cmd failed or is too slow'
end

begin
  hook.run_hook(ctx)
rescue Timeout::Error => e
  logger.warn "hook #{hook.class.name} timed out: #{e.message}; continuing without it"
end

When it happens

Trigger: Any exec hook command that runs longer than the configured timeout: a script that waits on stdin (the child is spawned with unsetenv_others and no TTY), a curl/git push against an unreachable host, a script blocked on a lock, or a default 60s timeout with a legitimately slow command.

Common situations: Backup/push scripts whose runtime grows as the repo grows; scripts that prompt for a password; network-dependent commands during outages; dockerized oxidized shelling out to slow wrappers.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• #{self.class.name}: configuration invalid: #{e.message}
• hook.remote_repo is required
• hook.token is required
• hook.channel is required
• hook.jid is required

AI-assisted analysis of ytti/oxidized@687ed4262d (2026-08-23). Data as JSON: /api/errors/d6b9f5f4c32cff6a. Report an issue: GitHub.