kubernetes/kubernetes · warning

unable to upgrade: missing upgrade headers in request: %#v

Error message

unable to upgrade: missing upgrade headers in request: %#v

What it means

400 Bad Request returned by spdy.responseUpgrader.UpgradeResponse (upgrade.go:87-93) when the request is not a proper SPDY upgrade: the Connection header does not contain 'Upgrade' OR the Upgrade header does not contain 'SPDY/3.1' (case-insensitive). The full request Header is dumped into the message via %#v for debugging. This runs before any attempt to negotiate the subprotocol or hijack the connection.

Source

Thrown at staging/src/k8s.io/streaming/pkg/httpstream/spdy/upgrade.go:91

// is capable of upgrading HTTP responses using SPDY/3.1 via the spdystream
// package.
//
// If pingPeriod is non-zero, for each incoming connection a background
// goroutine will send periodic Ping frames to the server. Use this to keep
// idle connections through certain load balancers alive longer.
func NewResponseUpgraderWithPings(pingPeriod time.Duration) httpstream.ResponseUpgrader {
	return responseUpgrader{pingPeriod: pingPeriod}
}

// UpgradeResponse upgrades an HTTP response to one that supports multiplexed
// streams. newStreamHandler will be called synchronously whenever the
// other end of the upgraded connection creates a new stream.
func (u responseUpgrader) UpgradeResponse(w http.ResponseWriter, req *http.Request, newStreamHandler httpstream.NewStreamHandler) httpstream.Connection {
	connectionHeader := strings.ToLower(req.Header.Get(httpstream.HeaderConnection))
	upgradeHeader := strings.ToLower(req.Header.Get(httpstream.HeaderUpgrade))
	if !strings.Contains(connectionHeader, strings.ToLower(httpstream.HeaderUpgrade)) || !strings.Contains(upgradeHeader, strings.ToLower(HeaderSpdy31)) {
		errorMsg := fmt.Sprintf("unable to upgrade: missing upgrade headers in request: %#v", req.Header)
		http.Error(w, errorMsg, http.StatusBadRequest)
		return nil
	}

	hijacker, ok := w.(http.Hijacker)
	if !ok {
		errorMsg := "unable to upgrade: unable to hijack response"
		http.Error(w, errorMsg, http.StatusInternalServerError)
		return nil
	}

	w.Header().Add(httpstream.HeaderConnection, httpstream.HeaderUpgrade)
	w.Header().Add(httpstream.HeaderUpgrade, HeaderSpdy31)
	w.WriteHeader(http.StatusSwitchingProtocols)

	conn, bufrw, err := hijacker.Hijack()
	if err != nil {
		runtime.HandleErrorWithContext(req.Context(), err, "Unable to upgrade: error hijacking response")
		return nil

View on GitHub (pinned to b882c60b40)

Solutions

  1. Use kubectl/client-go remotecommand, which sets Connection: Upgrade and Upgrade: SPDY/3.1 correctly for SPDY streaming.
  2. Configure intermediaries to preserve Connection and Upgrade headers end-to-end (these are hop-by-hop by default and often stripped — explicitly allow-list them).
  3. Verify the endpoint expects SPDY; some newer paths prefer websockets — ensure the client matches the server's transport.
  4. Inspect the %#v header dump in the error message to see exactly which headers arrived stripped.
  5. If running over HTTP/2, fall back to HTTP/1.1 for the streaming call or use a websocket-capable path.

Example fix

# before: proxy strips hop-by-hop headers (Connection, Upgrade) -> 400
# nginx default can drop these

# after: explicitly forward the upgrade headers
location / {
    proxy_pass https://kubelet;
    proxy_set_header Connection $http_connection;
    proxy_set_header Upgrade $http_upgrade;
    proxy_http_version 1.1;
}

# client side: ensure both upgrade headers are sent
req.Header.Set("Connection", "Upgrade")
req.Header.Set("Upgrade", "SPDY/3.1")
req.Header.Set("X-Stream-Protocol-Version", "v4.channel.k8s.io")
Defensive patterns

Strategy: validation

Validate before calling

// Client: set both upgrade headers required by the SPDY upgrader.
req.Header.Set(httpstream.HeaderConnection, httpstream.HeaderUpgrade)
req.Header.Set(httpstream.HeaderUpgrade, spdy.HeaderSpdy31) // "SPDY/3.1"
req.Header.Set(httpstream.HeaderProtocolVersion, "v4.channel.k8s.io")
// And configure intermediaries to forward Connection + Upgrade.
// nginx:
//   proxy_set_header Connection $http_connection;
//   proxy_set_header Upgrade $http_upgrade;
//   proxy_http_version 1.1;

Type guard

func isSpdyUpgradeRequest(r *http.Request) bool {
    conn := strings.ToLower(r.Header.Get(httpstream.HeaderConnection))
    up := strings.ToLower(r.Header.Get(httpstream.HeaderUpgrade))
    return strings.Contains(conn, strings.ToLower(httpstream.HeaderUpgrade)) &&
        strings.Contains(up, strings.ToLower(spdy.HeaderSpdy31))
}

Try / catch

// Client: 400 'missing upgrade headers' -> a hop stripped them; reconfigure the proxy and retry.
if resp.StatusCode == 400 && strings.Contains(body, "missing upgrade headers") {
    return fmt.Errorf("intermediary stripped Connection/Upgrade; fix proxy config and retry")
}

Prevention

When it happens

Trigger: A request to a SPDY streaming endpoint (exec/attach/port-forward on kubelet/apiserver) that lacks the standard HTTP upgrade headers; a client that set X-Stream-Protocol-Version but forgot Connection: Upgrade / Upgrade: SPDY/3.1; a proxy that strips or rewrites the Connection/Upgrade headers; an HTTP/2 client where upgrade semantics differ.

Common situations: Intermediaries (cloud LBs, ingress, service meshes) that drop the Connection header because it is hop-by-hop per RFC 7230; a websocket client hitting a SPDY-only endpoint; client-go/kubectl version mismatch; an HTTP/2-only path where the SPDY/3.1 upgrade cannot be expressed.

Related errors

AI-assisted analysis of kubernetes/kubernetes@b882c60b40 (2026-08-07). Data as JSON: /api/errors/18a2d935704356b5. Report an issue: GitHub.