thanos-io/thanos · error

setup gRPC server

Error message

setup gRPC server

What it means

tls.NewServerConfig failed while building the gRPC server TLS configuration. This validates that the supplied TLS cert/key files exist, parse as X.509, and that a client CA (if given) loads. On any of these failures the sidecar aborts with 'setup gRPC server'.

Solutions

1. Check the paths in --grpc-cert/--grpc-key/--grpc-client-ca-file exist and are readable (ls, openssl x509 -in cert)
2. Verify cert and key match: compare modulus/fingerprint of both files
3. Refresh the mounted Kubernetes Secret and restart the sidecar
4. Remove the TLS flags temporarily to confirm TLS config is the failing part, then re-add corrected files

Example fix

// before
- --grpc-cert=/etc/thanos/tls/server.crt
- --grpc-key=/etc/thanos/tls/server.key   # stale key
// after
- --grpc-cert=/etc/tls/server.crt
- --grpc-key=/etc/tls/server.key
# with both files refreshed from the same cert issuance

for _, p := range []string{tlsCert, tlsKey, tlsCA} {
    if p != "" {
        if _, err := os.Stat(p); err != nil {
            return fmt.Errorf("TLS file missing: %s", p)
        }
    }
}

tlsCfg, err := tls.NewServerConfig(logger, cert, key, ca, ...)
if err != nil {
    return errors.Wrap(err, "setup gRPC server")
}

When it happens

Trigger: errors.Wrap in runSidecar: --grpc-cert and --grpc-key provided but a file is missing/unreadable, key/cert mismatch, invalid PEM, or --grpc-client-ca-file points to a non-existent/invalid CA bundle.

Common situations: Secret mounted at the wrong path after a Helm change; cert renewed but key not updated (mismatch); CA bundle typo; permissions preventing the thanos user from reading the key.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• setup gRPC server
• setup gRPC server
• server credentials
• building gRPC client
• setup gRPC server

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5a4e106689e8ed01. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/sidecar.go:326

			return reloader.Watch(ctx)
		}, func(error) {
			cancel()
		})
	}

	// Setup the gRPC server.
	{
		c := promclient.NewWithTracingClient(logger, httpClient, clientconfig.ThanosUserAgent)

		promStore, err := store.NewPrometheusStore(logger, reg, c, conf.prometheus.url, component.Sidecar, m.Labels, m.Timestamps, m.Version)
		if err != nil {
			return errors.Wrap(err, "create Prometheus store")
		}

		tlsCfg, err := tls.NewServerConfig(log.With(logger, "protocol", "gRPC"),
			conf.grpc.tlsSrvCert, conf.grpc.tlsSrvKey, conf.grpc.tlsSrvClientCA, conf.grpc.tlsMinVersion, conf.grpc.tlsCiphers, conf.grpc.tlsCurves)
		if err != nil {
			return errors.Wrap(err, "setup gRPC server")
		}

		exemplarSrv := exemplars.NewPrometheus(conf.prometheus.url, c, m.Labels)

		infoSrv := info.NewInfoServer(
			component.Sidecar.String(),
			info.WithLabelSetFunc(func() []labelpb.ZLabelSet {
				return promStore.LabelSet()
			}),
			info.WithStoreInfoFunc(func() (*infopb.StoreInfo, error) {
				if httpProbe.IsReady() {
					mint, maxt := m.Timestamps()
					return &infopb.StoreInfo{
						MinTime:                      mint,
						MaxTime:                      maxt,
						SupportsSharding:             true,
						SupportsWithoutReplicaLabels: true,
						TsdbInfos:                    promStore.TSDBInfos(),

View on GitHub (pinned to 35b8b99117)