weaviate/weaviate · error

failed to read certificate from %s: %w

Error message

failed to read certificate from %s: %w

What it means

Once the certificate bytes are obtained (HTTP or S3 source), loadCertPool reads the body; an io.ReadAll / io.Copy failure wraps as 'failed to read certificate from <url>'. This means the response started successfully but the body could not be fully consumed.

Source

Thrown at usecases/auth/authentication/oidc/middleware.go:409

// S3 URI, or inline PEM string) and returns a certificate pool containing it.
// Note: HTTP URL fetches use the default http.Client, so the certificate URL
// must be reachable without custom TLS settings. Certificate and SkipTLSVerify
// are mutually exclusive, so this function is only called when SkipTLSVerify
// is false.
func (c *Client) loadCertPool() (*x509.CertPool, error) {
	var certificate, certificateSource string
	if strings.HasPrefix(c.Config.Certificate.Get(), "http") {
		resp, err := http.Get(c.Config.Certificate.Get())
		if err != nil {
			return nil, fmt.Errorf("failed to get certificate from %s: %w", c.Config.Certificate.Get(), err)
		}
		defer resp.Body.Close()
		if resp.StatusCode != http.StatusOK {
			return nil, fmt.Errorf("failed to download certificate from %s: http status: %v", c.Config.Certificate.Get(), resp.StatusCode)
		}
		certBytes, err := io.ReadAll(resp.Body)
		if err != nil {
			return nil, fmt.Errorf("failed to read certificate from %s: %w", c.Config.Certificate.Get(), err)
		}
		certificate = string(certBytes)
		certificateSource = c.Config.Certificate.Get()
	} else if strings.HasPrefix(c.Config.Certificate.Get(), "s3://") {
		parts := strings.TrimPrefix(c.Config.Certificate.Get(), "s3://")
		segments := strings.SplitN(parts, "/", 2)
		if len(segments) != 2 {
			return nil, fmt.Errorf("invalid S3 URI, must contain bucket and key: %s", c.Config.Certificate.Get())
		}
		region := os.Getenv("AWS_REGION")
		if region == "" {
			region = os.Getenv("AWS_DEFAULT_REGION")
		}
		creds := credentials.NewIAM("")
		// check if we are able to get the credentials using AWS IAM
		if _, err := creds.GetWithContext(nil); err != nil {
			// if IAM doesn't work, check environment settings for creds, set anonymous access if none found
			creds = credentials.NewEnvAWS()

View on GitHub (pinned to 75aa4b6d11)

Solutions

  1. Retry the fetch; if intermittent, it's a network/proxy stability issue.
  2. If the source is S3, verify the object exists and the IAM credentials have s3:GetObject.
  3. Check proxy/LB timeouts and increase body-size/timeout limits.
  4. Fall back to a locally mounted certificate file to remove the network dependency.

Example fix

// before (S3 read failing mid-stream)
AUTHENTICATION_OIDC_CERTIFICATE=s3://my-bucket/ca.pem
// after (local mount)
AUTHENTICATION_OIDC_CERTIFICATE=/etc/weaviate/certs/ca.pem
Defensive patterns

Strategy: retry

Validate before calling

// Prefetch the full body to validate readability before configuring
resp, err := http.Get(certURL)
if err != nil { return err }
body, err := io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil { return fmt.Errorf("cert body not readable: %w", err) }
if len(body) == 0 { return fmt.Errorf("empty certificate") }

Try / catch

const maxRetries = 3
for i := 0; i < maxRetries; i++ {
	if _, err := client.Init(ctx); err == nil || !strings.Contains(err.Error(), "failed to read certificate") {
		break
	}
	time.Sleep(time.Second << i)
}

Prevention

When it happens

Trigger: loadCertPool's io.ReadAll(resp.Body) (HTTP path) or io.Copy into a buffer (S3 path) returns an error mid-stream: connection reset while streaming, truncated response, S3 object read failure.

Common situations: Flaky network or proxy dropping long bodies; S3 object removed mid-download or credentials lacking read permission surfacing as a read error via minio; server closing connections early under load.

Understand the failure class

Related errors

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/829a94bc92582a60. Report an issue: GitHub.