thanos-io/thanos · error

client credentials

Error message

client credentials

What it means

The client TLS certificate manager loads the mTLS key pair via tls.LoadX509KeyPair on demand and on mtime change. Failure (missing/unreadable/mismatched/malformed files) is wrapped as "client credentials".

Solutions

1. Confirm cert and key match: compare modulus/sha of openssl x509 -noout -modulus -in cert and openssl pkey -in key -pubout.
2. Fix file permissions so the process can read the key (chmod 0600, correct owner).
3. Rotate both files atomically (write to temp then rename) to avoid mid-rotation mismatches.
4. Verify the cert/key paths in config point at the intended files.

Example fix

// before
client, err := tls.StoreClientTLSCredentials(logger, cert, key, ca, ...) // key path stale after rotation
// after
// re-issue cert+key as a matched pair and update both paths
openssl x509 -noout -modulus -in client.crt | openssl md5
openssl rsa -noout -modulus -in client.key | openssl md5  # must match

func validateClientPair(certPath, keyPath string) error {
    if _, err := tls.LoadX509KeyPair(certPath, keyPath); err != nil {
        return fmt.Errorf("client cert/key invalid: %w", err)
    }
    return nil
}

cert, err := mgr.getClientCertificate(cri)
if err != nil {
    level.Error(logger).Log("msg", "client cert reload failed; keeping previous cert", "err", err)
    return m.lastGoodCert, nil
}

When it happens

Trigger: getClientCertificate is called during a TLS handshake (initial or after cert/key mtime change) and LoadX509KeyPair fails on certPath/keyPath.

Common situations: Client cert rotated and key regenerated so the pair no longer matches; key file has a passphrase or wrong permissions; only cert was mounted; paths from env vars pointing to wrong files; cert/key written non-atomically during rotation.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• could not get organization field from client cert
• could not get organizationalUnit field from client cert
• could not get commonName field from client cert
• could not get required certificate field from client cert
• when a client CA is used a server key and certificate must…

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/d81209c0b388d33d. Report an issue: GitHub.

Appendix: source

Thrown at pkg/tls/options.go:222

}

func (m *clientTLSManager) getClientCertificate(*tls.CertificateRequestInfo) (*tls.Certificate, error) {
	m.mtx.Lock()
	defer m.mtx.Unlock()

	statCert, err := os.Stat(m.certPath)
	if err != nil {
		return nil, err
	}
	statKey, err := os.Stat(m.keyPath)
	if err != nil {
		return nil, err
	}

	if m.cert == nil || !statCert.ModTime().Equal(m.certModTime) || !statKey.ModTime().Equal(m.keyModTime) {
		cert, err := tls.LoadX509KeyPair(m.certPath, m.keyPath)
		if err != nil {
			return nil, errors.Wrap(err, "client credentials")
		}
		m.certModTime = statCert.ModTime()
		m.keyModTime = statKey.ModTime()
		m.cert = &cert
	}

	return m.cert, nil
}

type validOption struct {
	tlsOption map[string]uint16
}

func (validOption validOption) joinString() string {
	var keys []string

	for key := range validOption.tlsOption {
		keys = append(keys, key)

View on GitHub (pinned to 35b8b99117)