hashicorp/nomad · error

exec timed out: %v

Error message

exec timed out: %v

What it means

After wiring up exec I/O copying, the handler waits on either the exec error channel or the stream context being done. If the gRPC server context finishes first (client disconnect, deadline, cancellation), the exec is reported as timed out with the context's error embedded.

Source

Thrown at plugins/drivers/server.go:355

	execOpts, errCh := StreamToExecOptions(server.Context(),
		msg.Setup.Command, msg.Setup.Tty,
		server)

	result, err := d.ExecTaskStreaming(server.Context(),
		msg.Setup.TaskId, execOpts)

	execOpts.Stdout.Close()
	execOpts.Stderr.Close()

	if err != nil {
		return err
	}

	// wait for copy to be done
	select {
	case err = <-errCh:
	case <-server.Context().Done():
		err = fmt.Errorf("exec timed out: %v", server.Context().Err())
	}

	if err != nil {
		return err
	}

	server.Send(&ExecTaskStreamingResponseMsg{
		Exited: true,
		Result: exitResultToProto(result),
	})

	return err
}

func (b *driverPluginServer) SignalTask(ctx context.Context, req *proto.SignalTaskRequest) (*proto.SignalTaskResponse, error) {
	err := b.impl.SignalTask(req.TaskId, req.Signal)
	if err != nil {
		return nil, err

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped context error: DeadlineExceeded means increase the exec/RPC deadline; Canceled means the client or server canceled the stream.
2. Increase the timeout configured for the exec operation (e.g.Nomad's exec command/API timeout).
3. Ensure the client keeps the stream open until the command finishes instead of disconnecting early.
4. Verify plugin process health and network stability between Nomad client and driver plugin.

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
// after
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) // allow long execs

// Estimate command duration and compare with the RPC deadline before exec
deadline, _ := ctx.Deadline()
if time.Until(deadline) < expectedExecDuration {
	ctx, _ = context.WithDeadline(context.Background(), time.Now().Add(expectedExecDuration*2))
}

err := srv.ExecTaskStreaming(ctx)
if err != nil && strings.HasPrefix(err.Error(), "exec timed out:") {
	cause := ctx.Err() // DeadlineExceeded -> raise deadline; Canceled -> client disconnect
	if errors.Is(cause, context.DeadlineExceeded) { /* retry with larger timeout */ }
}

When it happens

Trigger: The ExecTaskStreaming RPC context is canceled or its deadline expires while the exec is still running; the client drops the connection mid-exec; caller-supplied context timeout is shorter than the command duration.

Common situations: Long-running commands exceeding the RPC deadline; user closes an exec session (alloc exec) abruptly; network partition between Nomad client and plugin; overly tight exec_timeout in tooling wrapping alloc exec.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• exec task timed out: %v
• CSI.ControllerListSnapshots: %v
• failed to receive initial message: %v
• failed to receive initial message: %v
• first message should always be setup

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/63fa7a5cdd779e4d. Report an issue: GitHub.