JuliusBrussee/caveman · error

%s: load system certificate pool: %w

Error message

%s: load system certificate pool: %w

What it means

Thrown by rootsWithCAFile in shared/platform/chhttp/chhttp.go:167 after the custom CA bundle file was read successfully but x509.SystemCertPool() failed. The function builds a trust pool by starting from the OS trust store and appending the bundle's certificates, so the system pool is a hard prerequisite. The error is prefixed with the CA-file environment variable name so the operator knows which knob produced it.

Source

Thrown at shared/platform/chhttp/chhttp.go:167

// Appending (rather than replacing) keeps a public managed endpoint verifiable
// while a private CA is trusted for the internal one.
//
// The bundle is parsed block by block instead of via CertPool.AppendCertsFromPEM,
// which reports success as soon as ONE certificate parses and silently drops the
// rest. A secret mount that is truncated mid-bundle, or corrupt after the first
// entry, would then be half-trusted: the endpoints whose issuer survived keep
// verifying and the ones whose issuer was dropped fail later, at the first
// telemetry flush, looking like a network fault. Any unusable certificate block —
// or a trailing PEM header with no complete block behind it — fails the whole
// bundle CLOSED at boot instead.
func rootsWithCAFile(path string) (*x509.CertPool, error) {
	bundle, err := os.ReadFile(path)
	if err != nil {
		return nil, fmt.Errorf("%s: %w", caFileEnv, err)
	}
	roots, err := x509.SystemCertPool()
	if err != nil {
		return nil, fmt.Errorf("%s: load system certificate pool: %w", caFileEnv, err)
	}
	added := 0
	rest := bundle
	for {
		var block *pem.Block
		block, rest = pem.Decode(rest)
		if block == nil {
			break
		}
		if block.Type != "CERTIFICATE" {
			continue
		}
		cert, err := x509.ParseCertificate(block.Bytes)
		if err != nil {
			return nil, fmt.Errorf("%s (%s): certificate %d is unparseable, so the bundle is incomplete and must not be half-trusted: %w", caFileEnv, path, added+1, err)
		}
		roots.AddCert(cert)
		added++

View on GitHub (pinned to 27d5a3981a)

Solutions

  1. Install the OS certificate package in the image (apk add ca-certificates / apt-get install ca-certificates) or copy /etc/ssl/certs from a builder stage.
  2. Check SSL_CERT_FILE and SSL_CERT_DIR in the process environment and make sure they point to existing, readable files/directories.
  3. Verify the service user has read permission on the system trust store.
  4. As a last resort on scratch images, set SSL_CERT_FILE to the custom bundle itself so SystemCertPool reads a file that exists.

Example fix

# before (Dockerfile)
FROM scratch
COPY myapp /myapp

# after
FROM golang:1.23 AS build
RUN go build -o /myapp ./...
FROM alpine:3.20
RUN apk add --no-cache ca-certificates
COPY --from=build /myapp /myapp
Defensive patterns

Strategy: validation

Validate before calling

// before setting the CA-file env var, prove the system pool loads
if _, err := x509.SystemCertPool(); err != nil {
    log.Fatalf("system trust store unusable: %v (check SSL_CERT_FILE/SSL_CERT_DIR and image certs)", err)
}

Try / catch

roots, err := chhttpTrustPool(path)
if err != nil {
    // fail closed: do not fall back to a default transport
    return fmt.Errorf("boot: TLS trust config: %w", err)
}

Prevention

When it happens

Trigger: Setting the CA-file env var (caFileEnv, e.g. CLICKHOUSE_CA_FILE) to a readable file while the process cannot load the operating system trust store: SSL_CERT_FILE or SSL_CERT_DIR points to a missing/unreadable file, or the container image ships no /etc/ssl/certs at all (scratch/distroless without ca-certificates).

Common situations: Minimal Docker images (FROM scratch or distroless) that never installed the ca-certificates package; a stray SSL_CERT_FILE env var left pointing at a file deleted during image slimming; hardened hosts where the trust store is not readable by the service user.

Understand the failure class

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/4956a7b724305cd3. Report an issue: GitHub.