hashicorp/nomad · error

failed to upgrade connection: %v

Error message

failed to upgrade connection: %v

What it means

Returned by wrapWebsocketHandler when the HTTP-to-websocket upgrade performed before handing off to the wrapped handler fails (e.g. missing/bad Upgrade headers or underlying hijack error); the connection is never upgraded so the handler cannot run.

Source

Thrown at command/agent/websockets.go:60

// already-upgraded connection to the handler. We pass the connection and the
// auth token via request context.
//
// NOTE: Outside of initial setup/upgrade failures, this handler should not
// return any value or error after the connection has been upgraded to a
// websocket. Any value or error should be sent to the websocket connection
// along with a close message before returning.
func (s *HTTPServer) wrapWebsocketHandler(handler handlerFn) handlerFn {
	return func(w http.ResponseWriter, req *http.Request) (any, error) {

		if fips140.Enabled() {
			return "", fmt.Errorf("websockets are disallowed in FIPS-140 mode")
		}

		// Upgrade the connection
		conn, err := s.wsUpgrader.Upgrade(w, req, nil)
		if err != nil {
			// The upgrade failed so return the error.
			return nil, fmt.Errorf("failed to upgrade connection: %v", err)
		}

		// Ensure the underlying websocket connection is closed when we
		// are done. This does not transmit the close message to the
		// client, so that must still be done before returning from
		// this function. Failure to send the close message can cause
		// the client connection to hang for an extended period of time
		// before closing.
		defer conn.Close()

		token, err := s.readWsHandshake(conn.ReadJSON, req)
		if err != nil {
			conn.WriteMessage(websocket.CloseMessage,
				websocket.FormatCloseMessage(toWsCode(400), err.Error()))
			return nil, nil
		}

		// Store connection and token in context for handler to use

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure any proxy in front of Nomad passes the Upgrade and Connection headers (e.g. nginx 'proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"').
2. Check the wrapped error text for 'websocket: the client is not using the websocket protocol' vs origin rejection and fix client headers accordingly.
3. Use the correct client library (nomad websocket exec/logs endpoints with proper Sec-WebSocket-* headers).
4. Verify Origin against the server's wsUpgrader CheckOrigin configuration.

Example fix

// before (nginx)
proxy_pass http://nomad;
// after (nginx)
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://nomad;

u, _ := url.Parse(wsURL)
if u.Scheme != "ws" && u.Scheme != "wss" {
    return fmt.Errorf("expected ws(s) URL, got %s", u.Scheme)
}
conn, resp, err := websocket.DefaultDialer.Dial(wsURL, nil)
if err != nil && resp != nil { return fmt.Errorf("handshake: %d %s", resp.StatusCode, err) }

conn, _, err := dialer.Dial(wsURL, nil)
if err != nil {
    log.Printf("ws upgrade failed (check proxy Upgrade headers / Origin): %v", err)
    return err
}

When it happens

Trigger: s.wsUpgrader.Upgrade(w, req, nil) fails because the request lacks a valid Upgrade: websocket header, the Sec-WebSocket-Key is missing/invalid, the Origin fails the CheckOrigin policy, or the client sent an incompatible websocket version.

Common situations: Proxy/load balancer stripping Upgrade/Connection headers, browser sending an unexpected Origin, an HTTP/2-only client, or a client library not performing the handshake correctly.

Related errors

• failed to send input: %w
• network namespace already exists but was misconfigured
• network already configured but not found in state
• no CNI network config found
• empty response from AWS metadata

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/df6ad35e7653b278. Report an issue: GitHub.