grpc/grpc-go · error

invalid header mutation: %v

Error message

invalid header mutation: %v

What it means

A header mutation received from an external processing server has an invalid header key (extconfig.go:137-139). validateHeaderKey rejects keys that are empty, pseudo-headers (starting with ':'), 'host', start with 'grpc-', are not all lowercase, or exceed 16384 bytes. This is a data-plane error, not a configuration error.

Source

Thrown at internal/xds/httpfilter/extconfig.go:139

// ignored.
//
// The input metadata must not be nil.
func (hmr *HeaderMutationRules) ApplyAdditions(hvos []*v3corepb.HeaderValueOption, input metadata.MD) error {
	if hmr == nil {
		hmr = &HeaderMutationRules{}
	}
	if input == nil {
		return fmt.Errorf("input metadata is nil")
	}
	if hmr.DisallowAll {
		return nil
	}

	for _, hvo := range hvos {
		header := hvo.GetHeader()
		key := header.GetKey()
		if err := validateHeaderKey(key); err != nil {
			return fmt.Errorf("invalid header mutation: %v", err)
		}

		value := header.GetValue()
		if strings.HasSuffix(key, "-bin") {
			value = string(header.GetRawValue())
		}
		if len(value) > maxHeaderSize {
			return fmt.Errorf("invalid header mutation: value for header key %q exceeds the maximum length of %d bytes", key, maxHeaderSize)
		}
		// ValidatePair rejects values carrying bytes outside %x20-%x7E. It
		// skips the value check for "-bin" keys, whose values the transport
		// base64 encodes.
		if err := imetadata.ValidatePair(key, value); err != nil {
			return fmt.Errorf("invalid header mutation: %v", err)
		}

		if !hmr.allow(key) {
			if hmr.DisallowIsError {

View on GitHub (pinned to 0c51461d27)

Solutions

1. Fix the external processing server to only send valid, lowercase, non-reserved header keys
2. Ensure header keys do not start with ':', are not 'host', and do not start with 'grpc-'
3. Convert all header keys to lowercase before sending mutations
4. Validate header key length does not exceed 16384 bytes

Example fix

// before — external auth server sends uppercase and reserved headers
resp := &extprocpb.HeaderMutation{
    Set: []HeaderValueOption{
        {Header: &HeaderValue{Key: "Content-Type", Value: "application/json"}},
        {Header: &HeaderValue{Key: ":path", Value: "/new-path"}},
    },
}

// after — lowercase, non-reserved keys only
resp := &extprocpb.HeaderMutation{
    Set: []HeaderValueOption{
        {Header: &HeaderValue{Key: "content-type", Value: "application/json"}},
        // ':path' removed — pseudo-headers cannot be mutated
    },
}

// Validate header keys before applying mutations from an external server.
func validateMutationKeys(hvos []*v3corepb.HeaderValueOption) error {
    for _, hvo := range hvos {
        key := hvo.GetHeader().GetKey()
        if key == "" || key[0] == ':' || key == "host" ||
            strings.HasPrefix(key, "grpc-") || key != strings.ToLower(key) {
            return fmt.Errorf("invalid header key %q from external server", key)
        }
    }
    return nil
}

// In the external processing/auth server response handler:
err := hmr.ApplyAdditions(hvos, md)
if err != nil && strings.Contains(err.Error(), "invalid header mutation") {
    // Log and drop the invalid mutation rather than failing the RPC.
    log.Printf("dropping invalid header mutation from external server: %v", err)
    err = nil
}

When it happens

Trigger: ApplyAdditions iterates over HeaderValueOption entries from an external authorization server's response, and validateHeaderKey returns an error for one of the header keys. The error wraps the specific validation failure (e.g. 'header key "Content-Type" is not lowercase').

Common situations: External auth server tries to set ':path' or ':authority' (pseudo-headers); server sends 'Content-Type' or 'Authorization' (uppercase) instead of 'content-type'; server sends an empty key; server sends a 'grpc-status' header (reserved grpc- prefix); server sends a header key longer than 16KB.

Related errors

• httpfilter: %v
• extauthz: missing default_value in filter_enabled
• extauthz: error parsing config %v: unknown type %T, want *an
• extauthz: failed to unmarshal config: %v
• extauthz: empty grpc_service provided in config %v

AI-assisted analysis of grpc/grpc-go@0c51461d27 (2026-08-11). Data as JSON: /api/errors/561c00583f9c6e77. Report an issue: GitHub.