kubesphere/kubesphere · critical

failed to build handler chain: %v

Error message

failed to build handler chain: %v

What it means

During APIServer PrepareRun, the server assembles its full HTTP handler chain via buildHandlerChain (authn/authz, filters, auditing, etc.). If that assembly returns an error, PrepareRun wraps and re-raises it as 'failed to build handler chain', aborting server startup before it can serve requests.

Source

Thrown at pkg/apiserver/apiserver.go:135

	s.container.RecoverHandler(func(panicReason interface{}, httpWriter http.ResponseWriter) {
		logStackOnRecover(panicReason, httpWriter)
	})
	s.installDynamicResourceAPI()
	s.installKubeSphereAPIs()
	s.installMetricsAPI()
	s.installHealthz()
	s.installLivez()
	if err := s.installOpenAPI(); err != nil {
		return err
	}

	for _, ws := range s.container.RegisteredWebServices() {
		klog.V(2).Infof("%s", ws.RootPath())
	}

	combinedHandler, err := s.buildHandlerChain(s.container, stopCh)
	if err != nil {
		return fmt.Errorf("failed to build handler chain: %v", err)
	}
	s.Server.Handler = filters.WithGlobalFilter(combinedHandler)
	return nil
}

func (s *APIServer) installOpenAPI() error {
	s.OpenAPIConfig = &restfulspec.Config{
		WebServices:                   s.container.RegisteredWebServices(),
		PostBuildSwaggerObjectHandler: openapicontroller.EnrichSwaggerObject,
	}

	openapiV2Services, err := openapiv2.BuildAndRegisterAggregator(s.OpenAPIConfig, s.container)
	if err != nil {
		klog.Errorf("failed to install openapi v2 service : %s", err)
	}
	s.openAPIV2Service = openapiV2Services
	openapiV3Services, err := openapiv3.BuildAndRegisterAggregator(s.OpenAPIConfig, s.container)
	if err != nil {

View on GitHub (pinned to 04a29b5c60)

Solutions

1. Read the wrapped inner error in the log (%v payload) — it names the actual failing component; fix that configuration first.
2. Validate the ks-apiserver ConfigMap/options: check TLS certs/keys, authentication (oauthOptions, identity providers), authorization, and auditing settings for syntactic and semantic correctness.
3. Verify external dependencies referenced by the handler chain (webhook URLs, Redis/DB for auditing/session, token issuers) are reachable and correctly configured.
4. If introduced after an upgrade, diff the options against the release's sample configuration and remove/rename deprecated keys.
5. Reproduce with klog verbosity raised (klog.V(2) shows web services/handler setup) to pinpoint which chain component errors.

Example fix

// before (ks-apiserver.yaml): bad cert path
secretReference:
  tlsCertFile: /etc/kubesphere/tls.crt
// after
secretReference:
  tlsCertFile: /etc/kubesphere/certs/tls.crt
// and ensure the file exists before starting the server

// pre-flight config check before starting the server
for _, f := range []string{tlsCertFile, tlsKeyFile} {
    if _, err := os.Stat(f); err != nil {
        return fmt.Errorf("missing required file %s: %w", f, err)
    }
}
if _, err := url.Parse(auditWebhookURL); err != nil {
    return fmt.Errorf("invalid audit webhook url: %w", err)
}

if err := server.PrepareRun(stopCh); err != nil {
    klog.Fatalf("apiserver prepare failed: %v", err) // inspect wrapped inner error for the failing component
}

When it happens

Trigger: Calling APIServer.PrepareRun (during ks-apiserver startup) when buildHandlerChain fails, typically due to a misconfigured or unloadable dependency in the chain: invalid options (bad TLS/authn/authz settings, invalid insecure/port config), failure constructing an auditing or authentication filter, or an error initializing a registered filter dependency.

Common situations: Invalid ks-apiserver.yaml options (e.g., wrong client secret/oauth options, bad TLS cert paths); auditing backend misconfiguration (bad webhook/elasticsearch URL); identity provider or token authenticator misconfiguration; version-upgrade leaving stale option keys in the ConfigMap.

Related errors

• the Identity provider was not found
• the Identity provider was Disabled
• redirect URL is not allowed
• invalid service port number
• invalid service host

AI-assisted analysis of kubesphere/kubesphere@04a29b5c60 (2026-09-03). Data as JSON: /api/errors/5b2034056647bca7. Report an issue: GitHub.