hashicorp/nomad · error

session open: %v

Error message

session open: %v

What it means

NodeRpc opens a new yamux stream over the multiplexed session to a node; this error is returned when session.Open() fails, meaning the underlying connection or session to the client cannot accept a new stream. It usually indicates the whole node connection is dead or closing.

Source

Thrown at nomad/client_rpc.go:246

	// Get the connection to the client
	state, ok := s.getNodeConn(nodeID)
	if !ok {
		// Make the RPC via another server
		return findNodeConnAndForward(s, nodeID, method, args, reply)
	}

	// Make the RPC
	return NodeRpc(state.Session, method, args, reply)
}

// NodeRpc is used to make an RPC call to a node. The method takes the
// Yamux session for the node and the method to be called.
func NodeRpc(session *yamux.Session, method string, args, reply any) error {
	// Open a new session
	stream, err := session.Open()
	if err != nil {
		return fmt.Errorf("session open: %v", err)
	}
	defer stream.Close()

	// Write the RpcNomad byte to set the mode
	if _, err := stream.Write([]byte{byte(pool.RpcNomad)}); err != nil {
		stream.Close()
		return fmt.Errorf("set mode: %v", err)
	}

	// Make the RPC
	err = msgpackrpc.CallWithCodec(pool.NewClientCodec(stream), method, args, reply)
	if err != nil {
		return err
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Retry the operation; the server will typically re-establish the node connection.
  2. Check the client agent is running and reachable: `nomad node status <node>` shows ready.
  3. Inspect network devices/firewalls for idle timeouts killing long-lived mux connections; raise idle timeouts or enable keepalives.
  4. If it persists, restart the client agent to reset its RPC multiplexing.

Example fix

// before: single attempt
err := NodeRpc(session, "Exec", args, reply)
// after: retry on session failure
var err error
for i := 0; i < 3; i++ {
    if err = NodeRpc(session, "Exec", args, reply); err == nil || !strings.Contains(err.Error(), "session open") {
        break
    }
    time.Sleep(time.Second)
}
Defensive patterns

Strategy: retry

Validate before calling

node, _, err := client.Nodes().Info(nodeID, nil)
if err != nil || node.Status != "ready" {
    return fmt.Errorf("skip NodeRpc: node %s not ready", nodeID)
}

Try / catch

if strings.Contains(err.Error(), "session open") {
    // dead mux session; brief backoff then retry once
    time.Sleep(time.Second)
    return NodeRpc(session, method, args, reply)
}

Prevention

When it happens

Trigger: Called by forwardProfileClient, Host (fs/exec endpoints), GarbageCollectAll, Signal, SetPauseState, GetPauseState when the yamux session to the node is already closed, the TCP connection dropped, or the session exhausted streams.

Common situations: Client agent restarted or crashed between connection check and RPC; NAT/firewall dropped the idle multiplexed connection; network blip; client in the middle of shutdown while a task command (signal, exec, logs) is issued.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f1fe0dc772a557e7. Report an issue: GitHub.