kubernetes/kops · error

incorrect RequestHash

Error message

incorrect RequestHash

What it means

The SHA-256 hash of the HTTP request body received by kops-controller does not match the RequestHash claim inside the signed token. The token commits to the exact body bytes that were signed on the node; a mismatch means the body was altered in transit or the client hashed different bytes than it sent. This binds the token to one specific request, preventing body-substitution replay.

Source

Thrown at pkg/bootstrap/pkibootstrap/pkiverifier/verifier.go:99

	tokenData := &pkibootstrap.AuthTokenData{}
	if err := json.Unmarshal(token.Data, tokenData); err != nil {
		return nil, nil, fmt.Errorf("unmarshalling authorization token data: %w", err)
	}

	// Guard against replay attacks
	if tokenData.Audience != pkibootstrap.AudienceNodeAuthentication {
		return nil, nil, fmt.Errorf("incorrect Audience")
	}
	timeSkew := math.Abs(time.Since(time.Unix(tokenData.Timestamp, 0)).Seconds())
	if timeSkew > float64(v.opt.MaxTimeSkew) {
		return nil, nil, fmt.Errorf("incorrect Timestamp %v", tokenData.Timestamp)
	}

	// Verify the token has signed the body content.
	requestHash := sha256.Sum256(body)
	if !bytes.Equal(requestHash[:], tokenData.RequestHash) {
		return nil, nil, fmt.Errorf("incorrect RequestHash")
	}

	return token, tokenData, nil
}

// Can generate keys with
// openssl ecparam -name prime256v1 -genkey -noout -out ec-priv-key.pem
// openssl ec -in ec-priv-key.pem -pubout > ec-pub-key.pem
// Note that golang doesn't support secp256k1: https://groups.google.com/g/golang-nuts/c/Mbkug5t3ZYA

func (v *verifier) VerifyToken(ctx context.Context, rawRequest *http.Request, authToken string, body []byte) (*bootstrap.VerifyResult, error) {
	// Reminder: we shouldn't trust any data we get from the client until we've checked the signature (and even then...)
	// Thankfully the GCE SDK does seem to escape the parameters correctly, for example.

	token, tokenData, err := v.parseTokenData(pkibootstrap.AuthenticationTokenPrefix, authToken, body)
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Remove or bypass anything between the node and kops-controller that transforms the request body (recompression, JSON re-encoding, WAF rewriting).
2. On the client, sign exactly the []byte sent: pass the same body slice to both CreateToken and the HTTP request writer.
3. Check handler middleware: if the body is read before VerifyToken, reset it with io.Copy(body, r.Body) / r.Body = http.NoBody pattern or buffer-and-restore so the verifier hashes the original bytes.
4. Align nodeup and kops-controller versions so the request body format matches what the token was minted for.
5. Debug by sha256-summing the body the controller receives and comparing with the token's requestHash claim (base64-decode the Data field).

Example fix

// before: client signs one payload, sends another
payload, _ := json.Marshal(req)
token, _ := auth.CreateToken(payload)
http.Post(url, "application/json", bytes.NewReader(prettyJSON))
// after: sign and send the identical bytes
payload, _ := json.Marshal(req)
token, _ := auth.CreateToken(payload)
http.Post(url, "application/json", bytes.NewReader(payload))

// On the client: verify the bytes you sign are exactly the bytes you send
payload, err := json.Marshal(req)
if err != nil {
	return err
}
token, err := authenticator.CreateToken(payload)
if err != nil {
	return err
}
hash := sha256.Sum256(payload)
_ = hash // log/hash compare after send if debugging; ensure the request body is bytes.NewReader(payload), not a re-marshaled copy

func requestHashMatches(body []byte, d *pkibootstrap.AuthTokenData) bool {
	h := sha256.Sum256(body)
	return d != nil && bytes.Equal(h[:], d.RequestHash)
}

result, err := verifier.VerifyToken(ctx, req, authToken, body)
if err != nil {
	if strings.Contains(err.Error(), "incorrect RequestHash") {
		// body was mutated in transit or client signed different bytes; do not blind-retry
		klog.Errorf("bootstrap request hash mismatch: %v", err)
		return nil, fmt.Errorf("request body does not match signed token: %w", err)
	}
	return nil, err
}

When it happens

Trigger: parseTokenData (verifier.go:97-100) raises this when sha256.Sum256(body) != tokenData.RequestHash: a proxy/load-balancer rewrites or re-compresses the POST body, the client signs a different payload than the one sent (e.g. signs the marshaled struct then sends a re-marshaled/vendored variant), Content-Encoding/transformation middleware mutates the body, or the body was read and not rewound before VerifyToken is called.

Common situations: Ingress controllers or service meshes that transparently recompress (gzip) or normalize JSON bodies; client code that signs the canonical JSON but sends pretty-printed JSON; double-reading the request body in a handler middleware so the verifier sees empty bytes; version skew where the request schema changed between nodeup and kops-controller.

Related errors

• unmarshalling authorization token data: %w
• incorrect Audience
• incorrect Timestamp %v
• failed to verify claim signature for node
• unable to SSH to %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/502c41a458c0a0b5. Report an issue: GitHub.