JuliusBrussee/caveman · error

%s: %w

Error message

%s: %w

What it means

Thrown by rootsWithCAFile when os.ReadFile fails on the path in CLICKHOUSE_TLS_CA_FILE; the underlying error (no such file, permission denied, etc.) is wrapped with the env var name for context. The ClickHouse TLS config fails closed — an unreadable CA bundle is an error, never a silent fallback to the ambient trust store.

Source

Thrown at shared/platform/chhttp/chhttp.go:163

	return cfg, nil
}

// rootsWithCAFile returns the system pool with the PEM bundle at path appended.
// Appending (rather than replacing) keeps a public managed endpoint verifiable
// while a private CA is trusted for the internal one.
//
// The bundle is parsed block by block instead of via CertPool.AppendCertsFromPEM,
// which reports success as soon as ONE certificate parses and silently drops the
// rest. A secret mount that is truncated mid-bundle, or corrupt after the first
// entry, would then be half-trusted: the endpoints whose issuer survived keep
// verifying and the ones whose issuer was dropped fail later, at the first
// telemetry flush, looking like a network fault. Any unusable certificate block —
// or a trailing PEM header with no complete block behind it — fails the whole
// bundle CLOSED at boot instead.
func rootsWithCAFile(path string) (*x509.CertPool, error) {
	bundle, err := os.ReadFile(path)
	if err != nil {
		return nil, fmt.Errorf("%s: %w", caFileEnv, err)
	}
	roots, err := x509.SystemCertPool()
	if err != nil {
		return nil, fmt.Errorf("%s: load system certificate pool: %w", caFileEnv, err)
	}
	added := 0
	rest := bundle
	for {
		var block *pem.Block
		block, rest = pem.Decode(rest)
		if block == nil {
			break
		}
		if block.Type != "CERTIFICATE" {
			continue
		}
		cert, err := x509.ParseCertificate(block.Bytes)
		if err != nil {

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Check the wrapped os.ReadFile error: 'no such file or directory' means fix the path/mount, 'permission denied' means fix file ownership/mode.
2. Verify the mount exists at process start: in k8s confirm the volume is mounted and the secret key produces that filename; locally, ls the exact path.
3. Ensure the env value has no surrounding quotes or stray whitespace (the code trims spaces, but shell quoting can still bite depending on how env is injected).
4. If you meant to drop custom CAs and use system roots, unset CLICKHOUSE_TLS_CA_FILE entirely.

Example fix

# before
CLICKHOUSE_TLS_CA_FILE=/etc/clickhouse/ca.pem   # secret mounted elsewhere

# after — match the actual mount path
CLICKHOUSE_TLS_CA_FILE=/etc/secrets/clickhouse/ca.pem
# kustomization: verify volumes[].mountPath and the secret key filename line up

if p := os.Getenv("CLICKHOUSE_TLS_CA_FILE"); p != "" {
    if _, err := os.Stat(p); err != nil {
        return fmt.Errorf("CLICKHOUSE_TLS_CA_FILE %q is not readable before start: %w", p, err)
    }
}

Handle at startup: read the wrapped os.ReadFile error, fix the mount/path/permissions, and restart. Do not catch and fall back to system roots — that fallback is exactly what the fail-closed design forbids.

When it happens

Trigger: Setting CLICKHOUSE_TLS_CA_FILE to a path that doesn't exist in the container, isn't mounted yet, or has restrictive permissions; a trailing newline/whitespace or quotes around the path in the env value can also produce a non-existent path.

Common situations: Secret not mounted at the expected path in k8s (volume mount name/path mismatch); running the binary locally with a path from the deploy manifest; a rotated secret whose new file path differs; env value copied with surrounding quotes.

Related errors

• production refuses %s=true: it disables chain and hostname v
• production requires an https:// CLICKHOUSE_URL (TLS only); C
• %s is not set; cannot encrypt/decrypt secrets
• %s is not valid base64: %w
• secretbox: production requires CAVE_KMS_PROVIDER=scaleway

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/50eb7c52cf4b4ad3. Report an issue: GitHub.