hashicorp/nomad · error

failed to upgrade connection

Error message

failed to upgrade connection: %v

What it means

Returned by wrapWebsocketHandler when the HTTP-to-websocket upgrade performed before handing off to the wrapped handler fails (e.g. missing/bad Upgrade headers or underlying hijack error); the connection is never upgraded so the handler cannot run.

Solutions

  1. Ensure any proxy in front of Nomad passes the Upgrade and Connection headers (e.g. nginx 'proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"').
  2. Check the wrapped error text for 'websocket: the client is not using the websocket protocol' vs origin rejection and fix client headers accordingly.
  3. Use the correct client library (nomad websocket exec/logs endpoints with proper Sec-WebSocket-* headers).
  4. Verify Origin against the server's wsUpgrader CheckOrigin configuration.

Example fix

// before (nginx)
proxy_pass http://nomad;
// after (nginx)
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://nomad;
Defensive patterns

Strategy: try-catch

Validate before calling

u, _ := url.Parse(wsURL)
if u.Scheme != "ws" && u.Scheme != "wss" {
    return fmt.Errorf("expected ws(s) URL, got %s", u.Scheme)
}
conn, resp, err := websocket.DefaultDialer.Dial(wsURL, nil)
if err != nil && resp != nil { return fmt.Errorf("handshake: %d %s", resp.StatusCode, err) }

Try / catch

conn, _, err := dialer.Dial(wsURL, nil)
if err != nil {
    log.Printf("ws upgrade failed (check proxy Upgrade headers / Origin): %v", err)
    return err
}

Prevention

When it happens

Trigger: s.wsUpgrader.Upgrade(w, req, nil) fails because the request lacks a valid Upgrade: websocket header, the Sec-WebSocket-Key is missing/invalid, the Origin fails the CheckOrigin policy, or the client sent an incompatible websocket version.

Common situations: Proxy/load balancer stripping Upgrade/Connection headers, browser sending an unexpected Origin, an HTTP/2-only client, or a client library not performing the handshake correctly.

Related errors


AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/df6ad35e7653b278. Report an issue: GitHub.

Appendix: source

Thrown at command/agent/websockets.go:60

// already-upgraded connection to the handler. We pass the connection and the
// auth token via request context.
//
// NOTE: Outside of initial setup/upgrade failures, this handler should not
// return any value or error after the connection has been upgraded to a
// websocket. Any value or error should be sent to the websocket connection
// along with a close message before returning.
func (s *HTTPServer) wrapWebsocketHandler(handler handlerFn) handlerFn {
	return func(w http.ResponseWriter, req *http.Request) (any, error) {

		if fips140.Enabled() {
			return "", fmt.Errorf("websockets are disallowed in FIPS-140 mode")
		}

		// Upgrade the connection
		conn, err := s.wsUpgrader.Upgrade(w, req, nil)
		if err != nil {
			// The upgrade failed so return the error.
			return nil, fmt.Errorf("failed to upgrade connection: %v", err)
		}

		// Ensure the underlying websocket connection is closed when we
		// are done. This does not transmit the close message to the
		// client, so that must still be done before returning from
		// this function. Failure to send the close message can cause
		// the client connection to hang for an extended period of time
		// before closing.
		defer conn.Close()

		token, err := s.readWsHandshake(conn.ReadJSON, req)
		if err != nil {
			conn.WriteMessage(websocket.CloseMessage,
				websocket.FormatCloseMessage(toWsCode(400), err.Error()))
			return nil, nil
		}

		// Store connection and token in context for handler to use

View on GitHub (pinned to 482b49bf1a)