elastic/elasticsearch · critical · RuntimeException

Failed to initialise TLS context for OTel log export

Error message

Failed to initialise TLS context for OTel log export

What it means

OtelSdkExportLogsSupplier.buildProvider() builds an SSLContext for the OTel log exporter whenever certificate_authorities or certificate/key are configured. Any GeneralSecurityException raised while loading PEM material, constructing the trust/key managers, or initialising SSLContext.getInstance("TLS") is wrapped and rethrown as RuntimeException at supplier.get() (initial install or cert hot-reload). This aborts the OTel logs appender setup.

Source

Thrown at modules/apm/src/main/java/org/elasticsearch/telemetry/apm/internal/export/otelsdk/OtelSdkExportLogsSupplier.java:249

        OtlpGrpcLogRecordExporterBuilder exporterBuilder = OtlpGrpcLogRecordExporter.builder()
            .setEndpoint(OtelSdkSettings.TELEMETRY_LOGS_ENDPOINT.get(settings))
            .setTimeout(OtelSdkSettings.TELEMETRY_EXPORT_SEND_TIMEOUT.get(settings).toDuration())
            .setConnectTimeout(OtelSdkSettings.TELEMETRY_EXPORT_CONNECT_TIMEOUT.get(settings).toDuration())
            .setRetryPolicy(OtelSdkSettings.OTLP_RETRY_POLICY);
        List<String> cas = OtelSdkSettings.TELEMETRY_LOGS_SSL_CERTIFICATE_AUTHORITIES.get(settings);
        if (cas.isEmpty() == false || cert.isEmpty() == false) {
            try {
                SslTrustConfig trustConfig = cas.isEmpty() ? DefaultJdkTrustConfig.DEFAULT_INSTANCE : new PemTrustConfig(cas, configDir);
                X509ExtendedTrustManager trustManager = trustConfig.createTrustManager();
                KeyManager[] keyManagers = null;
                if (cert.isEmpty() == false) {
                    keyManagers = new KeyManager[] { new PemKeyConfig(cert, key, new char[0], configDir).createKeyManager() };
                }
                SSLContext sslContext = SSLContext.getInstance("TLS");
                sslContext.init(keyManagers, new TrustManager[] { trustManager }, null);
                exporterBuilder.setSslContext(sslContext, trustManager);
            } catch (GeneralSecurityException e) {
                throw new RuntimeException("Failed to initialise TLS context for OTel log export", e);
            }
        }
        int maxQueueSize = OtelSdkSettings.TELEMETRY_LOGS_MAX_QUEUE_SIZE.get(settings);
        return SdkLoggerProvider.builder()
            .setResource(OtelSdkResource.get(settings))
            .addLogRecordProcessor(BatchLogRecordProcessor.builder(exporterBuilder.build()).setMaxQueueSize(maxQueueSize).build())
            .build();
    }

    /**
     * Rebuild the OTel logs export with fresh TLS material and swap it into the running appender
     * atomically to avoid dropped records.
     *
     * <p>{@link ElasticsearchOtelAppender#setOpenTelemetry} is a volatile write guarded by a
     * {@code ReadWriteLock} inside the appender, so new audit events switch to the new channel
     * without a gap. The old {@link SdkLoggerProvider} is closed after the swap: its
     * {@code BatchLogRecordProcessor} flushes any buffered records through the still-valid old
     * channel (rotation happens before cert expiry) before shutting down the old gRPC connection.

View on GitHub (pinned to db6a809a66)

Solutions

1. Validate each PEM file independently: openssl x509 -in cert.pem -noout; openssl pkey -in key.pem -noout; confirm cert and key share a modulus.
2. Verify the configured paths are readable by the ES process user.
3. Ensure files are PEM-encoded (BEGIN CERTIFICATE / BEGIN PRIVATE KEY), not DER or PKCS#12.
4. If a CA chain is used, confirm it chains to the collector's presented cert.
5. Re-issue or re-upload the cert/key pair during a maintenance window to avoid a half-replaced state.

Example fix

// before
telemetry.logs.ssl.certificate: certs/client.crt
telemetry.logs.ssl.key: certs/client.key  // wrong file / mismatched
// after
openssl x509 -modulus -in certs/client.crt -noout | openssl md5
openssl rsa  -modulus -in certs/client.key -noout | openssl md5
// hashes must match; redeploy matching pair

// Pre-validate PEM material the same way the supplier will
static String validatePem(Path cert, Path key, List<Path> cas) throws Exception {
  var cf = java.security.cert.CertificateFactory.getInstance("X.509");
  try (var in = Files.newInputStream(cert)) { cf.generateCertificate(in); }
  java.security.KeyPair kp = null;
  try (var in = Files.newInputStream(key)) {
    var kpObj = java.security.KeyPairGenerator.getInstance("RSA"); // placeholder; use BouncyCastle/PEMParser in real code
  }
  for (Path ca : cas) try (var in = Files.newInputStream(ca)) { cf.generateCertificate(in); }
  return "ok";
}

// Wrap node start / supplier refresh so a TLS failure is surfaced, not swallowed
try {
 otelLogsSupplier.get();
} catch (RuntimeException e) {
  if (e.getCause() instanceof GeneralSecurityException) {
    alertOps("OTel logs TLS init failed: " + e.getCause().getMessage());
    // do NOT continue with insecure fallback; fail closed
    throw e;
  } else throw e;
}

When it happens

Trigger: Configuring telemetry.logs.ssl.certificate / .key / .certificate_authorities with: a malformed or non-PEM file, a cert/key pair that does not match, a CA file that cannot be parsed, an unreadable path (permissions), or an unsupported algorithm/provider on the JVM.

Common situations: Self-signed mTLS between ES and an OTLP collector; cert rotation that uploaded a half-written file; pointing to a PKCS#12 file when PEM is required; running on a JVM with restricted crypto providers.

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• telemetry.logs.ssl.certificate and telemetry.logs.ssl.key mu
• failed to initialize a TrustManager for the system keystore
• Cannot specify more than one trust method (CA=%s, trustStore
• could not create the default ssl context
• Cannot combine trust configurations [{}]

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/c884775872e563f0. Report an issue: GitHub.