What causes "Failed to read CA bytes" in apache/pulsar?

Fix filesystem permissions so the worker process user can read brokerClientTrustCertsFilePath (see the wrapped IOException cause).

apache/pulsar · error · IllegalStateException

Failed to read CA bytes

Error message

Failed to read CA bytes

What it means

WorkerConfig.getTlsTrustChainBytes() lazily loads the TLS trust certificate chain from brokerClientTrustCertsFilePath. If the file exists but cannot be read (permission denied, I/O error, file removed between the exists() check and read), it wraps the IOException in IllegalStateException('Failed to read CA bytes'). Returns null only when the path is empty or the file does not exist.

Source

Thrown at pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java:959

                    ? this.getWorkerPort() : this.getWorkerPortTls());
        }
        return this.workerId;
    }

    public String getWorkerHostname() {
        if (isBlank(this.workerHostname)) {
            this.workerHostname = unsafeLocalhostResolve();
        }
        return this.workerHostname;
    }

    public byte[] getTlsTrustChainBytes() {
        if (StringUtils.isNotEmpty(getBrokerClientTrustCertsFilePath())
                && Files.exists(Paths.get(getBrokerClientTrustCertsFilePath()))) {
            try {
                return Files.readAllBytes(Paths.get(getBrokerClientTrustCertsFilePath()));
            } catch (IOException e) {
                throw new IllegalStateException("Failed to read CA bytes", e);
            }
        } else {
            return null;
        }
    }

    public String getWorkerWebAddress() {
        return String.format("http://%s:%d", this.getWorkerHostname(), this.getWorkerPort());
    }

    public String getWorkerWebAddressTls() {
        return String.format("https://%s:%d", this.getWorkerHostname(), this.getWorkerPortTls());
    }

    public static String unsafeLocalhostResolve() {
        try {
            // Get the fully qualified hostname
            return InetAddress.getLocalHost().getCanonicalHostName();

View on GitHub (pinned to 820761864e)

Solutions

Fix filesystem permissions so the worker process user can read brokerClientTrustCertsFilePath (see the wrapped IOException cause).
Verify the configured path points to a readable PEM trust-cert file, not a directory or broken symlink.
Use an unreadable-proof mount mode for the TLS secret (e.g. defaultMode 0444 in Kubernetes).
Alternatively embed the trust chain via a config value instead of a file path if filesystem access is unreliable in your deployment.

Example fix

# before (K8s secret mount)
volumes:
- name: tls-trust
  secret:
    secretName: ca-cert
    defaultMode: 0400   # worker runs as non-root -> unreadable
# after
volumes:
- name: tls-trust
  secret:
    secretName: ca-cert
    defaultMode: 0444

Defensive patterns

Strategy: validation

Validate before calling

Path trustCerts = Paths.get(workerConfig.getBrokerClientTrustCertsFilePath());
if (trustCerts.toString() != null && !trustCerts.toString().isBlank()) {
    if (!Files.isRegularFile(trustCerts) || !Files.isReadable(trustCerts)) {
        throw new IllegalStateException("TLS trust cert file missing or unreadable: " + trustCerts);
    }
    Files.readAllBytes(trustCerts); // pre-flight read to surface permission issues early
}

Try / catch

try {
    byte[] caBytes = workerConfig.getTlsTrustChainBytes();
} catch (IllegalStateException e) {
    if (e.getMessage() != null && e.getMessage().equals("Failed to read CA bytes")) {
        // check cause IOException: fix file permissions/path of brokerClientTrustCertsFilePath
    }
    throw e;
}

Prevention

Mount TLS secrets readable by the worker user (e.g. K8s defaultMode 0444).
Pre-flight check brokerClientTrustCertsFilePath in deployment/entrypoint scripts.
Avoid replacing cert files in place; use atomic rename of fully-readable files.
Keep the exists()-then-read race in mind: mount stable paths, not symlinks to rotating targets.

When it happens

Trigger: Accessing getTlsTrustChainBytes() during worker/client construction when brokerClientTrustCertsFilePath is set and the file exists but Files.readAllBytes fails — e.g. unreadable permissions, the path is a directory-like special file, or an I/O error mid-read.

Common situations: TLS cert file mounted with root-only permissions while the worker runs as another user; cert file deleted/replaced between exists() and read in a container; wrong file mounted at the configured path (e.g. a symlink to an unreadable target); Kubernetes secret mounted with restrictive mode.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/dcf2a841de6c90f2. Report an issue: GitHub.