production refuses %s=true: it disables chain and hostname v

Error message

production refuses %s=true: it disables chain and hostname verification, so any in-path party can read or forge telemetry — use %s instead

What it means

Thrown by tlsClientConfig in the ClickHouse telemetry HTTP package when CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY=true is set while the process runs in production (per env.IsProduction / CAVE_ENV). Disabling TLS verification would let any on-path party read or forge telemetry, so production refuses to boot the client with it; the error message points at CLICKHOUSE_TLS_SERVER_NAME as the safe alternative for mismatched-hostname setups.

Source

Thrown at shared/platform/chhttp/chhttp.go:125

// definitions of "prod" would let one gate fire while the other stayed asleep.
func production() bool {
	return env.IsProduction()
}

// tlsClientConfig builds the ClickHouse TLS configuration from the environment,
// or (nil, nil) when no CLICKHOUSE_TLS_* knob is set — in which case callers keep
// the stock transport (system roots, hostname verified against the URL host).
//
// It fails CLOSED: a prod skip-verify request, an unreadable CA file, or a CA
// file with no parseable certificate all return an error rather than quietly
// downgrading to the ambient trust store.
func tlsClientConfig() (*tls.Config, error) {
	serverName := strings.TrimSpace(env.String(serverNameEnv, ""))
	caFile := strings.TrimSpace(env.String(caFileEnv, ""))
	skipVerify := env.Bool(skipVerifyEnv, false)

	if skipVerify && production() {
		return nil, fmt.Errorf("production refuses %s=true: it disables chain and hostname verification, so any in-path party can read or forge telemetry — use %s instead", skipVerifyEnv, serverNameEnv)
	}
	if serverName == "" && caFile == "" && !skipVerify {
		return nil, nil
	}

	cfg := &tls.Config{
		MinVersion: tls.VersionTLS12,
		ServerName: serverName,
		// #nosec G402 -- refused in production above; non-prod only, and loudly
		// warned about by ValidateProduction at startup.
		InsecureSkipVerify: skipVerify,
	}
	if caFile != "" {
		roots, err := rootsWithCAFile(caFile)
		if err != nil {
			return nil, err
		}
		cfg.RootCAs = roots

View on GitHub (pinned to 27d5a3981a)

Solutions

Remove CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY from the production environment.
If the certificate hostname doesn't match the URL host, set CLICKHOUSE_TLS_SERVER_NAME=<expected certificate name> instead.
For a private CA, install the CA bundle via CLICKHOUSE_TLS_CA_FILE (system pool + bundle, still verified) rather than skipping verification.
Keep skip-verify only in non-prod environments and expect a startup warning from ValidateProduction there.

Example fix

# before (production env)
CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY=true

# after
# cert hostname != URL host:
CLICKHOUSE_TLS_SERVER_NAME=clickhouse.internal.corp
# or trust a private CA while keeping verification:
CLICKHOUSE_TLS_CA_FILE=/etc/secrets/clickhouse-ca.pem

Defensive patterns

Strategy: validation

Validate before calling

if os.Getenv("CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY") == "true" && isProdEnv() {
    return errors.New("refuse to start: insecure TLS skip-verify is not allowed in production")
}

Try / catch

Do not catch-and-continue: this error is a deliberate boot-time refusal. Let startup fail and fix the environment, otherwise you are running the exact risk the guard exists to prevent.

Prevention

Keep production env files free of CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY; audit ConfigMaps/secrets during promotion from staging.
For hostname mismatches use CLICKHOUSE_TLS_SERVER_NAME; for private CAs use CLICKHOUSE_TLS_CA_FILE.
Add a deploy-time check that refuses configs combining production CAVE_ENV with skip-verify.

When it happens

Trigger: Setting CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY=true in an environment where CAVE_ENV marks the process production (e.g. CAVE_ENV=production), and any ClickHouse client path that builds the TLS config (boot or first telemetry client construction).

Common situations: A dev workaround (self-signed cert on a local ClickHouse) copied into the production env file; a private CA whose hostname doesn't match the URL host, worked around with skip-verify instead of CLICKHOUSE_TLS_SERVER_NAME; k8s ConfigMap promoting staging env wholesale.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/a73c0b2a0e1666e6. Report an issue: GitHub.