thanos-io/thanos · critical

setup gRPC server

Error message

setup gRPC server

What it means

While setting up the gRPC server, runReceive calls tls.NewServerConfig to build the server TLS configuration from the certificate, key, client CA, min version, ciphers and curves flags. Any failure loading or validating these TLS materials is wrapped with 'setup gRPC server', so receive cannot serve gRPC and exits.

Solutions

1. Check the underlying wrapped error — it names which TLS input failed (cert, key, CA, cipher, etc.).
2. Verify cert/key/CA file paths passed via --grpc-server.tls-* exist and are readable by the process.
3. Confirm the certificate and key are a valid matching PEM pair (openssl x509/x509 -modulus check) and not expired.
4. Use supported values for --grpc-server.tls-min-version, tls-ciphers and tls-curves for your Go version.
5. If TLS is not needed, omit the TLS flags entirely so tls.NewServerConfig is not asked to load files.

Example fix

// before
thanos receive --grpc-server.tls-cert=/missing/cert.pem --grpc-server.tls-key=/missing/key.pem
// after
thanos receive --grpc-server.tls-cert=/etc/thanos/tls/cert.pem --grpc-server.tls-key=/etc/thanos/tls/key.pem --grpc-server.tls-client-ca=/etc/thanos/tls/ca.pem

for _, p := range []string{tlsCert, tlsKey, tlsClientCA} {
  if p != "" {
    if _, err := tls.LoadX509KeyPair(p, p); p == tlsClientCA {
      if _, err := os.ReadFile(p); err != nil { return err }
    } else if _, err := os.Stat(p); err != nil { return err }
  }
}
// plus: openssl x509 -in cert.pem -checkend 0 and verify key pair match

if err := run(); err != nil {
  if strings.Contains(err.Error(), "setup gRPC server") {
    log.Errorf("check TLS cert/key/CA paths and validity: %v", err)
  }
}

When it happens

Trigger: Starting `thanos receive` with --grpc-server.tls-cert/--grpc-server.tls-key (and optionally --grpc-server.tls-client-ca) where the cert/key files do not exist, are unreadable, are invalid/expired PEM, do not form a matching pair, or a min-version/cipher/curve name is unknown.

Common situations: TLS secret mounted at the wrong path; expired certificates; key without cert or mismatched pair; unsupported TLS version/cipher string on the deployed Go version; missing client CA file when mTLS is required.

Related errors

• setup gRPC server
• setup gRPC server
• server credentials
• building gRPC client
• setup gRPC server

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5abf6e353c7ac3e9. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/receive.go:362

			httpserver.WithGracePeriod(time.Duration(*conf.httpGracePeriod)),
			httpserver.WithTLSConfig(*conf.httpTLSConfig),
		)
		g.Add(func() error {
			statusProber.Healthy()
			return srv.ListenAndServe()
		}, func(err error) {
			statusProber.NotReady(err)
			defer statusProber.NotHealthy(err)

			srv.Shutdown(err)
		})
	}

	level.Debug(logger).Log("msg", "setting up gRPC server")
	{
		tlsCfg, err := tls.NewServerConfig(log.With(logger, "protocol", "gRPC"), conf.grpcConfig.tlsSrvCert, conf.grpcConfig.tlsSrvKey, conf.grpcConfig.tlsSrvClientCA, conf.grpcConfig.tlsMinVersion, conf.grpcConfig.tlsCiphers, conf.grpcConfig.tlsCurves)
		if err != nil {
			return errors.Wrap(err, "setup gRPC server")
		}

		if conf.lazyRetrievalMaxBufferedResponses <= 0 {
			return errors.New("--receive.lazy-retrieval-max-buffered-responses must be > 0")
		}
		options := []store.ProxyStoreOption{
			store.WithProxyStoreDebugLogging(debugLogging),
			store.WithMatcherCache(cache),
			store.WithoutDedup(),
			store.WithLazyRetrievalMaxBufferedResponsesForProxy(conf.lazyRetrievalMaxBufferedResponses),
		}

		proxy := store.NewProxyStore(
			logger,
			reg,
			dbs.TSDBLocalClients,
			comp,
			labels.Labels{},

View on GitHub (pinned to 35b8b99117)