apache/skywalking · critical · IllegalStateException

admin-server: gRPCSslEnabled=true but gRPCSslTrustedCAsPath

Error message

admin-server: gRPCSslEnabled=true but gRPCSslTrustedCAsPath is empty. The admin-internal gRPC bus needs a CA bundle on every node so peer channels can establish TLS to the server's cert. Set SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH (or admin-server.gRPCSslTrustedCAsPath in application.yml) on every OAP, or set gRPCSslEnabled=false everywhere.

What it means

When admin-server's internal gRPC bus has TLS enabled (gRPCSslEnabled=true), every node acts as both server and client; peers dial each other's gRPC port with the trusted-CAs bundle to verify the server cert. This check fails fast at boot when gRPCSslTrustedCAsPath is null/empty, because a client without a CA bundle would either fail or silently downgrade the security model at first reconcile.

Source

Thrown at oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModuleProvider.java:175

        grpcServer.initialize();
        registerServiceImplementation(GRPCHandlerRegister.class,
                                      new GRPCHandlerRegisterImpl(grpcServer));

        // Peer channel manager: register IMMEDIATELY in prepare() so the
        // framework's requiredCheck (which fires before any provider's
        // start()) sees this service. ClusterNodesQuery isn't available
        // here yet — pass a supplier that resolves it lazily on first
        // reconcile, which fires from notifyAfterCompleted() after every
        // module's start() has run.
        // When the server side is TLS-enabled, the client side MUST also use TLS — a
        // missing or empty trusted-CAs path on the client would leave us dialling peers
        // in plaintext at a port the server is only willing to handshake. Fail fast at
        // boot rather than letting the cluster silently break at first reconcile.
        SslContext clientSslContext = null;
        if (moduleConfig.isGRPCSslEnabled()) {
            if (moduleConfig.getGRPCSslTrustedCAsPath() == null
                || moduleConfig.getGRPCSslTrustedCAsPath().isEmpty()) {
                throw new IllegalStateException(
                    "admin-server: gRPCSslEnabled=true but gRPCSslTrustedCAsPath is empty. "
                        + "The admin-internal gRPC bus needs a CA bundle on every node so "
                        + "peer channels can establish TLS to the server's cert. Set "
                        + "SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH (or "
                        + "admin-server.gRPCSslTrustedCAsPath in application.yml) on every "
                        + "OAP, or set gRPCSslEnabled=false everywhere.");
            }
            try {
                clientSslContext = AdminClusterChannelManagerImpl.clientSslContext(
                    moduleConfig.getGRPCSslTrustedCAsPath());
            } catch (final Exception e) {
                throw new IllegalStateException(
                    "admin-server: failed to build admin gRPC client SSL context", e);
            }
        }
        peerChannelManager = new AdminClusterChannelManagerImpl(
            () -> getManager().find(ClusterModule.NAME).provider()
                              .getService(ClusterNodesQuery.class),

View on GitHub (pinned to 102af09b4a)

Solutions

1. Set gRPCSslTrustedCAsPath (or SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH) to the CA bundle that signed the peers' server certs, on EVERY OAP node
2. Ensure the CA file actually contains the CA(s) used to issue gRPCSslCertChainPath certs on all nodes
3. If TLS is not required for the admin bus, set gRPCSslEnabled: false on all nodes and keep the bus on a private network

Example fix

# before (application.yml)
admin-server:
  default:
    gRPCPort: ${SW_ADMIN_SERVER_GRPC_PORT:18080}
    gRPCSslEnabled: true
# after
admin-server:
  default:
    gRPCPort: ${SW_ADMIN_SERVER_GRPC_PORT:18080}
    gRPCSslEnabled: true
    gRPCSslCertChainPath: /etc/sw/tls/server-cert.pem
    gRPCSslKeyPath: /etc/sw/tls/server-key.pem
    gRPCSslTrustedCAsPath: /etc/sw/tls/ca.pem

# Pre-flight: TLS on => CA bundle must exist and be non-empty on every node
if [ "${SW_ADMIN_SERVER_GRPC_SSL_ENABLED:-false}" = "true" ]; then
  ca="${SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH:?gRPCSslEnabled=true requires SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH}"
  [ -s "$ca" ] || { echo "CA bundle empty or missing: $ca"; exit 1; }
fi

When it happens

Trigger: Setting gRPCSslEnabled: true on the admin-server provider without setting gRPCSslTrustedCAsPath; or setting the cert/key paths for the server side only. The message names both remediation routes: set SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH on every OAP, or disable TLS everywhere.

Common situations: Partial TLS rollout — enabling SSL on some nodes only; copying a config template that has gRPCSslCertChainPath/gRPCSslKeyPath but not the CA bundle path; environment variable misspelling so the default empty string survives.

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• admin-server: port must be > 0 when the module is enabled, g
• admin-server: gRPCPort must be > 0 when the module is enable
• admin-server: failed to build admin gRPC client SSL context
• decorate() should be invoked after service()
• Failed to compile MAL expression for metric: {}, expression:

AI-assisted analysis of apache/skywalking@102af09b4a (2026-08-14). Data as JSON: /api/errors/4cb3b2595305d895. Report an issue: GitHub.