thanos-io/thanos · error

dialing connection

Error message

dialing connection

What it means

EndpointSet.newEndpointRef creates the gRPC channel (grpc.NewClient) to a discovered endpoint spec. If channel construction fails, the error is wrapped as "dialing connection" and the endpoint is not added to the set.

Solutions

1. Fix the endpoint address: gRPC targets should be host:port (or dns:///host:port), no scheme like http://.
2. Use grpc.NewClient with a proper resolver prefix for DNS (dns:///host:port) if needed.
3. Validate spec.dialOpts (transport credentials) are constructed without error at startup.
4. Log spec.Addr() and validate with net.SplitHostPort before creating the ref.

Example fix

// before
spec.Addr() == "http://sidecar:10901" // invalid for gRPC

// after
spec.Addr() == "sidecar:10901"
// or
target := "dns:///sidecar:10901"
conn, err := grpc.NewClient(target, spec.dialOpts...)

host, port, err := net.SplitHostPort(spec.Addr())
if err != nil {
	return fmt.Errorf("invalid gRPC endpoint address %q: %w", spec.Addr(), err)
}
if strings.HasPrefix(spec.Addr(), "http") {
	return fmt.Errorf("gRPC address %q must not include http scheme", spec.Addr())
}

func validGRPCAddr(addr string) bool {
	_, _, err := net.SplitHostPort(addr)
	return err == nil && !strings.Contains(addr, "://")
}

When it happens

Trigger: grpc.NewClient(spec.Addr(), spec.dialOpts...) returns an error — typically an invalid target address string (unparsable host:port, bad scheme/resolver) or invalid dial options (bad credentials/TLS config).

Common situations: Malformed gRPC endpoint address in store/endpoint configuration (e.g. including http:// scheme in a gRPC address, IPv6 without brackets); invalid TLS credentials supplied via dial options; wrong port.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• failed to dial peer
• no query API server reachable
• failed to get tsdb status from prometheus
• retrieving targets
• error retrieving rules

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a2617244ecbd83e5. Report an issue: GitHub.

Appendix: source

Thrown at pkg/query/endpointset.go:672

	mtx      *sync.RWMutex
	cc       *grpc.ClientConn
	addr     string
	isStrict bool

	created  time.Time
	metadata *endpointMetadata
	status   *EndpointStatus

	logger log.Logger
}

// newEndpointRef creates a new endpointRef with a gRPC channel to the given the IP address.
// The call to newEndpointRef will return an error if establishing the channel fails.
func (e *EndpointSet) newEndpointRef(spec *GRPCEndpointSpec) (*endpointRef, error) {
	conn, err := grpc.NewClient(spec.Addr(), spec.dialOpts...)
	if err != nil {
		return nil, errors.Wrap(err, "dialing connection")
	}

	return &endpointRef{
		logger:   e.logger,
		created:  e.now(),
		addr:     spec.Addr(),
		isStrict: spec.isStrictStatic,
		cc:       conn,
		mtx:      &sync.RWMutex{},
	}, nil
}

// update sets the metadata and status of the endpoint ref based on the info response value and error.
func (er *endpointRef) update(now nowFunc, metadata *endpointMetadata, err error) {
	er.mtx.Lock()
	defer er.mtx.Unlock()

	er.updateMetadata(metadata, err)

View on GitHub (pinned to 35b8b99117)