hashicorp/nomad · error

exec task timed out: %v

Error message

exec task timed out: %v

What it means

In DriverHandle.ExecStreaming, after streaming stdin/stdout/stderr, the handler waits on doneCh (signaled by the driver when the exec finishes) and races it against ctx.Done(). If the context is cancelled or its deadline expires first, the exec is considered timed out and this error wraps ctx.Err().

Source

Thrown at client/allocrunner/taskrunner/driver_handle.go:114

	if !ok {
		return fmt.Errorf("task driver does not support exec")
	}

	execOpts, doneCh := drivers.StreamToExecOptions(
		ctx, command, tty, stream)

	result, err := d.ExecTaskStreaming(ctx, h.taskID, execOpts)
	if err != nil {
		return err
	}

	execOpts.Stdout.Close()
	execOpts.Stderr.Close()

	select {
	case err = <-doneCh:
	case <-ctx.Done():
		err = fmt.Errorf("exec task timed out: %v", ctx.Err())
	}

	if err != nil {
		return err
	}

	return stream.Send(drivers.NewExecStreamingResponseExit(result.ExitCode))
}

func (h *DriverHandle) Network() *drivers.DriverNetwork {
	return h.net
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Increase the caller's context deadline/timeout for long-running exec commands
2. Verify the exec'd command actually terminates (avoid interactive commands in scripts)
3. Check task resource starvation (CPU limits) that makes the command slow
4. Cancel intentionally and re-run with a larger timeout if this was expected

Example fix

// before
ctx := context.Background()
// after: give the exec more time
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()

// pick a deadline that comfortably exceeds expected command runtime
deadline := time.Now().Add(5 * time.Minute)
if deadline.Before(time.Now().Add(expectedRuntime)) {
    return errors.New("configured exec timeout too small for this command")
}

err := handle.ExecStreaming(ctx, cmd, tty, stream)
if err != nil && strings.Contains(err.Error(), "exec task timed out") {
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
    defer cancel()
    err = handle.ExecStreaming(ctx, cmd, tty, stream) // retry with longer deadline
}

When it happens

Trigger: The exec session in the task did not complete before the caller's context deadline/cancellation; ctx.Done() fires while waiting on doneCh.

Common situations: 'nomad alloc exec' with a command that hangs longer than the client timeout; an interactive exec session left idle past a deadline; slow command inside a resource-starved container.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• operation cancelled: %w
• exec timed out: %v
• command is not present
• rcp.connection_write_timeout must be greater than zero
• rcp.stream_close_timeout must be greater than zero

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d017ada52453e20c. Report an issue: GitHub.