apache/skywalking · critical · IllegalStateException

admin-server: failed to build admin gRPC client SSL context

Error message

admin-server: failed to build admin gRPC client SSL context

What it means

After the empty-path check passes, the provider builds a Netty client SslContext from the configured CA bundle via AdminClusterChannelManagerImpl.clientSslContext(). Any failure there (file not found, unreadable, malformed PEM, unsupported PKCS format) is wrapped in this IllegalStateException with the original exception as cause, aborting module start.

Source

Thrown at oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModuleProvider.java:187

        // in plaintext at a port the server is only willing to handshake. Fail fast at
        // boot rather than letting the cluster silently break at first reconcile.
        SslContext clientSslContext = null;
        if (moduleConfig.isGRPCSslEnabled()) {
            if (moduleConfig.getGRPCSslTrustedCAsPath() == null
                || moduleConfig.getGRPCSslTrustedCAsPath().isEmpty()) {
                throw new IllegalStateException(
                    "admin-server: gRPCSslEnabled=true but gRPCSslTrustedCAsPath is empty. "
                        + "The admin-internal gRPC bus needs a CA bundle on every node so "
                        + "peer channels can establish TLS to the server's cert. Set "
                        + "SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH (or "
                        + "admin-server.gRPCSslTrustedCAsPath in application.yml) on every "
                        + "OAP, or set gRPCSslEnabled=false everywhere.");
            }
            try {
                clientSslContext = AdminClusterChannelManagerImpl.clientSslContext(
                    moduleConfig.getGRPCSslTrustedCAsPath());
            } catch (final Exception e) {
                throw new IllegalStateException(
                    "admin-server: failed to build admin gRPC client SSL context", e);
            }
        }
        peerChannelManager = new AdminClusterChannelManagerImpl(
            () -> getManager().find(ClusterModule.NAME).provider()
                              .getService(ClusterNodesQuery.class),
            moduleConfig.getGRPCPort(),
            moduleConfig.getInternalCommunicationTimeout(),
            clientSslContext);
        registerServiceImplementation(AdminClusterChannelManager.class, peerChannelManager);
    }

    @Override
    public void start() {
        // Routes are added by feature modules in their start() phase via the
        // HTTPHandlerRegister service exposed above. Channel manager is
        // already registered in prepare() with a lazy ClusterNodesQuery
        // supplier; nothing else to do here.

View on GitHub (pinned to 102af09b4a)

Solutions

1. Read the chained cause in the stack trace (FileNotFoundException, PEMException, etc.) — it names the real problem
2. Verify the path exists inside the container (kubectl exec / docker run -- ls) and is readable by the OAP user
3. Validate the file is a PEM CA bundle (openssl x509 -in ca.pem -noout -text) and re-issue if malformed
4. Ensure mounts (secret volumes, configmaps) actually project the file at that path

# Pre-flight: verify the CA bundle parses before OAP start
openssl x509 -in "$SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH" -noout || { echo 'bad CA bundle'; exit 1; }

This throws IllegalStateException at module start with the underlying SSL exception as cause — inspect e.getCause() (PEMException, FileNotFoundException) to pinpoint the file problem; do not catch-and-continue in production, fix the mount/file.

When it happens

Trigger: gRPCSslTrustedCAsPath points to a path that does not exist in the container, is not readable by the OAP process, is an empty or corrupted PEM, or contains a certificate format the TLS stack rejects. The cause chain carries the underlying exception.

Common situations: Docker/K8s deployments where the TLS secret was not mounted at the configured path or mounted with wrong permissions; rotating certs and truncating the file; PEM with only a private key instead of CA certificates; wrong architecture binary issues after image changes.

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• admin-server: gRPCSslEnabled=true but gRPCSslTrustedCAsPath
• admin-server: failed to start gRPC server
• admin-server: port must be > 0 when the module is enabled, g
• admin-server: gRPCPort must be > 0 when the module is enable
• hierarchy-definition.yml not found.

AI-assisted analysis of apache/skywalking@102af09b4a (2026-08-14). Data as JSON: /api/errors/c15c6b262fd43e60. Report an issue: GitHub.