hashicorp/nomad · error

failed to send input: %w

Error message

failed to send input: %w

What it means

In api/allocations_exec.go:71, the exec session's run loop watches a sendErrCh for failures writing to the exec websocket. When sending stdin/input to the running task fails, it terminates the exec session with exit code -2 and this wrapped error. It indicates the websocket connection used for streaming input to the allocation broke or was closed mid-session.

Source

Thrown at api/allocations_exec.go:71

	sendErrCh := s.startTransmit(ctx, conn)
	exitCh, recvErrCh := s.startReceiving(ctx, conn)

	for {
		select {
		case <-ctx.Done():
			return -2, ctx.Err()
		case exitCode := <-exitCh:
			return exitCode, nil
		case recvErr := <-recvErrCh:
			// drop websocket code, not relevant to user
			if wsErr, ok := recvErr.(*websocket.CloseError); ok && wsErr.Text != "" {
				return -2, errors.New(wsErr.Text)
			}

			return -2, recvErr
		case sendErr := <-sendErrCh:
			return -2, fmt.Errorf("failed to send input: %w", sendErr)
		}
	}
}

func (s *execSession) startConnection() (*websocket.Conn, error) {
	// First, attempt to connect to the node directly, but may fail due to network isolation
	// and network errors.  Fallback to using server-side forwarding instead.
	nodeClient, err := s.client.GetNodeClientWithTimeout(s.alloc.NodeID, ClientConnTimeout, s.q)
	if err == NodeDownErr {
		return nil, NodeDownErr
	}

	q := s.q
	if q == nil {
		q = &QueryOptions{}
	}
	if q.Params == nil {
		q.Params = make(map[string]string)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %w error — a websocket close error reveals whether the node or network closed the connection
2. Re-establish the exec session (retry Exec) — sessions are not resumable
3. Check node health and allocation status; if the node drained, reschedule the allocation elsewhere
4. Keep the connection alive (application-level ping/keepalive) or shorten session duration behind proxies with idle timeouts
5. Verify network connectivity/firewall rules allow websocket traffic to the node's HTTPAddr

Example fix

// before
exitCode, err := allocs.Exec(allocID, task, tty, commands, stdin, stdout, stderr, q)
if err != nil { return err }
// after: handle send failure as a retryable connection loss
exitCode, err := allocs.Exec(allocID, task, tty, commands, stdin, stdout, stderr, q)
if err != nil {
    if strings.Contains(err.Error(), "failed to send input") {
        return fmt.Errorf("exec connection lost, re-establishing session: %w", err)
    }
    return err
}

// pre-flight: confirm the allocation is running on a healthy node
alloc, _, err := client.Allocations().Info(allocID, nil)
if err != nil || alloc.ClientStatus == "failed" { return fmt.Errorf("allocation not viable for exec") }

exit, err := execSession()
if err != nil && strings.Contains(err.Error(), "failed to send input") {
	// transient connection loss: recreate the session with backoff
	return retryWithBackoff(3, execSession)
}

When it happens

Trigger: Calling AllocSTW/Exec/ActionExec on an allocation and writing stdin while the websocket to the node is dead: node goes down mid-session, network interruption, node agent restarts, or the connection was closed server-side.

Common situations: Long-running interactive exec sessions where the target node drains or restarts; laptops sleeping mid-session killing the TCP connection; network isolation between client and node; NAT/firewall idle timeouts dropping the websocket.

Related errors

• websocket closed before receiving exit code: %w
• failed to marshal command: %W
• http addr of node %q (%s) is not advertised
• unsupported scheme: %v
• invalid redirect location %q: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d65bc796c8a69d0b. Report an issue: GitHub.