weaviate/weaviate · error

failed to decode auth broker response

Error message

failed to decode auth broker response: %w

What it means

After a 200 response, fetchCredentials decodes the body into AuthBrokerCredentialValue. If the JSON is malformed or truncated, the decode error is wrapped with this message, so the failure is in the broker response payload, not the HTTP layer.

Solutions

1. Inspect the raw broker response body for unexpected HTML or schema drift
2. Check any proxy/ingress between Weaviate and the broker for interference
3. Verify broker version compatibility with AuthBrokerCredentialValue field names

// Verify the broker returns JSON, not an HTML page behind a proxy
resp, _ := http.Get(brokerURL)
ct := resp.Header.Get("Content-Type")
if !strings.Contains(ct, "application/json") {
    return fmt.Errorf("broker returned non-JSON content type %q", ct)
}

creds, err := fetchCredentials(ctx)
if err != nil && strings.Contains(err.Error(), "failed to decode") {
    // log the raw response body for diagnosis, then fail
    return fmt.Errorf("broker payload invalid: %w", err)
}

When it happens

Trigger: JSON decoding of resp.Body fails — empty body, HTML error page behind a proxy returning 200, or truncated response.

Common situations: Reverse proxy/ingress intercepts and returns a 200 HTML login page; broker version returns a different JSON schema; network truncation of a partial response.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• auth broker returned non-200 status
• failed to decode auth broker response
• marshal body
• marshal body
• adjust geo property type: marshal geo property map

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/e8f9e59c4be35ab1. Report an issue: GitHub.

Appendix: source

Thrown at usecases/modulecomponents/awscommon/auth_broker.go:171

	if err != nil {
		return nil, fmt.Errorf("%w: %w", ErrRetryableAuthBroker, err)
	}
	defer func() {
		_, _ = io.Copy(io.Discard, resp.Body)
		resp.Body.Close()
	}()

	if resp.StatusCode >= 500 || resp.StatusCode == http.StatusTooManyRequests {
		return nil, fmt.Errorf("%w: auth broker returned status %d", ErrRetryableAuthBroker, resp.StatusCode)
	}

	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("auth broker returned non-200 status: %d", resp.StatusCode)
	}

	var creds AuthBrokerCredentialValue
	if err := json.NewDecoder(resp.Body).Decode(&creds); err != nil {
		return nil, fmt.Errorf("failed to decode auth broker response: %w", err)
	}

	if creds.AccessKeyID == "" || creds.SecretAccessKey == "" || creds.SessionToken == "" || creds.Expiration.IsZero() {
		return nil, errors.New("auth broker response missing required fields (access_key_id, secret_access_key, session_token, expiration)")
	}

	return &creds, nil
}

func (b *AuthBrokerCredentials) readIdentityToken() (string, error) {
	tok, err := os.ReadFile(b.identityTokenPath)
	if err != nil {
		return "", fmt.Errorf("failed to read web identity token from %q: %w", b.identityTokenPath, err)
	}
	// An empty file most likely means we caught kubelet mid-rotation. Fail
	// clearly at this layer rather than sending "Authorization: Bearer " to
	// the broker and getting an opaque 401 back.
	trimmed := strings.TrimSpace(string(tok))

View on GitHub (pinned to 75aa4b6d11)