hashicorp/nomad · error

set mode: %v

Error message

set mode: %v

What it means

NodeRpc writes a single mode byte (pool.RpcNomad) after opening the stream so the client's mux reader dispatches to the Nomad RPC handler; failure writing that byte produces this error. It means the stream broke immediately after opening — the connection to the node is effectively unusable.

Source

Thrown at nomad/client_rpc.go:253

	// Make the RPC
	return NodeRpc(state.Session, method, args, reply)
}

// NodeRpc is used to make an RPC call to a node. The method takes the
// Yamux session for the node and the method to be called.
func NodeRpc(session *yamux.Session, method string, args, reply any) error {
	// Open a new session
	stream, err := session.Open()
	if err != nil {
		return fmt.Errorf("session open: %v", err)
	}
	defer stream.Close()

	// Write the RpcNomad byte to set the mode
	if _, err := stream.Write([]byte{byte(pool.RpcNomad)}); err != nil {
		stream.Close()
		return fmt.Errorf("set mode: %v", err)
	}

	// Make the RPC
	err = msgpackrpc.CallWithCodec(pool.NewClientCodec(stream), method, args, reply)
	if err != nil {
		return err
	}

	return nil
}

// NodeStreamingRpc is used to make a streaming RPC call to a node. The method
// takes the Yamux session for the node and the method to be called. It conducts
// the initial handshake and returns a connection to be used or an error. It is
// the callers responsibility to close the connection if there is no error.
func NodeStreamingRpc(session *yamux.Session, method string) (net.Conn, error) {
	// Open a new session
	stream, err := session.Open()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the command; a fresh connection will be dialed if the old one is dead.
2. Verify node health: `nomad node status <node-id>` should show ready.
3. Check the client's logs for connection resets and fix underlying network instability.
4. If the alloc's client is being shut down, wait for rescheduling and target the new alloc.

Example fix

// before
nomad alloc signal -task web SIGTERM <alloc>  # fails: set mode: connection reset
// after (retry once after checking node)
nomad node status <node> && nomad alloc signal -task web SIGTERM <alloc>

if strings.Contains(err.Error(), "set mode") {
    // stream died immediately; reconnect and retry once
    time.Sleep(500 * time.Millisecond)
    return NodeRpc(session, method, args, reply)
}

When it happens

Trigger: Any NodeRpc caller (Signal, exec/Host streaming, pause-state, GC, profiling) where the stream.Write of the RpcNomad byte fails: connection reset, client disconnected concurrently, or stream closed by peer.

Common situations: Client agent stopping at the exact moment of the call; flaky network dropping the muxed TCP connection; container/node shutdown racing with an admin command like `nomad alloc signal`.

Related errors

• missing AllocID
• <remote node streaming RPC error relayed from ack.Error>
• failed to open a streaming connection: %v
• missing target region
• session open: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c503587c4ae52f1b. Report an issue: GitHub.