weaviate/weaviate · error

auth broker returned non-200 status

Error message

auth broker returned non-200 status: %d

What it means

fetchCredentials returns this plain (non-wrapped) error when the auth broker responds with any status that is neither 200 nor retryable (5xx/429). It is deliberately not retryable, so fetchCredentialsWithRetry fails fast.

Solutions

1. Check the status code in the message: 401/403 means fix the IRSA token or service-account role binding
2. 404: correct the auth broker URL/path in configuration
3. If 401 due to rotation, trigger a credential re-fetch after the token file is refreshed

// Pre-flight: ensure the broker endpoint is reachable and returns 200
resp, err := http.Get(brokerURL + "/health")
if err != nil || resp.StatusCode != http.StatusOK {
    return fmt.Errorf("auth broker unhealthy (status %v)", resp)
}

creds, err := fetchCredentials(ctx)
if err != nil {
    // non-200: not retryable — inspect code in message and fix config/token
    return fmt.Errorf("credential fetch failed permanently: %w", err)
}

When it happens

Trigger: fetchCredentials gets a response with StatusCode < 500, != 429, and != 200 — e.g. 401 (rejected bearer token), 403, 404 (wrong path).

Common situations: Stale or empty web identity token file after kubelet rotation yields 401; misconfigured broker URL path yields 404; service account lacks permission for the role (403).

Related errors

• failed to decode auth broker response
• amazon
• auth broker credentials fetch aborted
• auth broker identity file
• auth broker returned non-200 status

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/876397ebe0c04de1. Report an issue: GitHub.

Appendix: source

Thrown at usecases/modulecomponents/awscommon/auth_broker.go:166

	}

	req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", identityToken))

	resp, err := b.client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("%w: %w", ErrRetryableAuthBroker, err)
	}
	defer func() {
		_, _ = io.Copy(io.Discard, resp.Body)
		resp.Body.Close()
	}()

	if resp.StatusCode >= 500 || resp.StatusCode == http.StatusTooManyRequests {
		return nil, fmt.Errorf("%w: auth broker returned status %d", ErrRetryableAuthBroker, resp.StatusCode)
	}

	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("auth broker returned non-200 status: %d", resp.StatusCode)
	}

	var creds AuthBrokerCredentialValue
	if err := json.NewDecoder(resp.Body).Decode(&creds); err != nil {
		return nil, fmt.Errorf("failed to decode auth broker response: %w", err)
	}

	if creds.AccessKeyID == "" || creds.SecretAccessKey == "" || creds.SessionToken == "" || creds.Expiration.IsZero() {
		return nil, errors.New("auth broker response missing required fields (access_key_id, secret_access_key, session_token, expiration)")
	}

	return &creds, nil
}

func (b *AuthBrokerCredentials) readIdentityToken() (string, error) {
	tok, err := os.ReadFile(b.identityTokenPath)
	if err != nil {
		return "", fmt.Errorf("failed to read web identity token from %q: %w", b.identityTokenPath, err)

View on GitHub (pinned to 75aa4b6d11)