thanos-io/thanos · error

fetching metric metadata from metadata client

Error message

fetching metric metadata from metadata client %v

What it means

receive() opens the metadata stream to the upstream client inside a span; a failure to create/execute the MetricMetadata call is wrapped with this message naming the client. The error is either returned directly (ABORT strategy) or converted into a warning response for the caller.

Solutions

1. Check health of the upstream metadata service and its connectivity
2. Inspect the wrapped error for the gRPC status (unavailable, deadline exceeded, etc.) and fix accordingly
3. Retry with backoff if transient; with ABORT strategy the error propagates, with WARNING it is sent as a warning response
4. Increase request/context timeout if deadline exceeded

// verify upstream reachable before querying
conn, err := grpc.DialContext(ctx, addr, grpc.WithBlock())
if err != nil { return errors.New("metadata service unreachable") }

_, err := client.MetricMetadata(ctx, req)
if err != nil {
    if status.Code(err) == codes.Unavailable || status.Code(err) == codes.DeadlineExceeded {
        // retry with backoff
    }
}

When it happens

Trigger: stream.client.MetricMetadata fails at stream setup — upstream unavailable, context deadline exceeded, or connection refused.

Common situations: Vizier metadata service down during a PxL metadata query; network outage between proxy and store; context canceled because the caller aborted.

Related errors

• proxy MetricMetadata
• fetching tsdb statistics from status client
• receiving tsdb statistics from status client
• no query API server reachable
• failed to get tsdb status from prometheus

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a0f1703842a05f60. Report an issue: GitHub.

Appendix: source

Thrown at pkg/metadata/proxy.go:108

type metricMetadataStream struct {
	client  metadatapb.MetadataClient
	request *metadatapb.MetricMetadataRequest
	channel chan<- *metadatapb.MetricMetadata
	server  metadatapb.Metadata_MetricMetadataServer
}

func (stream *metricMetadataStream) receive(ctx context.Context) error {
	var (
		err         error
		metadataCli metadatapb.Metadata_MetricMetadataClient
	)

	tracing.DoInSpan(ctx, "receive_metadata_stream_request", func(ctx context.Context) {
		metadataCli, err = stream.client.MetricMetadata(ctx, stream.request)
	})

	if err != nil {
		err = errors.Wrapf(err, "fetching metric metadata from metadata client %v", stream.client)

		if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
			return err
		}

		if serr := stream.server.Send(metadatapb.NewWarningMetadataResponse(err)); serr != nil {
			return serr
		}
		// Not an error if response strategy is warning.
		return nil
	}

	for {
		resp, err := metadataCli.Recv()
		if err == io.EOF {
			return nil
		}

View on GitHub (pinned to 35b8b99117)