thanos-io/thanos · error

proxy MetricMetadata

Error message

proxy MetricMetadata

What it means

MetricMetadata on the proxy forwards a metadata request to the MetadataStore service and wraps any failure from that proxied call with 'proxy MetricMetadata'. It means the underlying store/Vizier metadata RPC failed.

Solutions

1. Check Vizier health (px get vizier / vizier pod status) and reconnect if unhealthy
2. Inspect the wrapped inner error for the true gRPC status and address it
3. Retry the request once connectivity is restored
4. Verify network/DNS between the proxy and the metadata service

// check Vizier health before querying
healthy, err := checkVizierHealth(ctx)
if err != nil || !healthy { return errors.New("vizier unavailable") }

if _, err := pxClient.MetricMetadata(ctx, req); err != nil {
    var serr interface{ GRPCStatus() *status.Status }
    if errors.As(err, &serr) {
        log.Printf("metadata proxy failure: %s", serr.GRPCStatus().Code())
    }
    // retry with backoff
}

When it happens

Trigger: Calling cloud-proxy MetricMetadata when the backing metadata service is unreachable, returns a gRPC error, or the streaming server errors mid-response.

Common situations: Vizier unhealthy/disconnected in Pixie Cloud; network partition between cloud proxy and Vizier; metadata service crash or timeout.

Related errors

• fetching metric metadata from metadata client
• fetching tsdb statistics from status client
• receiving tsdb statistics from status client
• no query API server reachable
• failed to get tsdb status from prometheus

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/68c6bb50f7c635ce. Report an issue: GitHub.

Appendix: source

Thrown at pkg/metadata/metadata.go:56

	span, ctx := tracing.StartSpan(ctx, "metadata_grpc_request")
	defer span.Finish()

	srv := &metadataServer{ctx: ctx, metric: req.Metric, limit: int(req.Limit)}

	if req.Limit >= 0 {
		if req.Metric != "" {
			srv.metadataMap = make(map[string][]metadatapb.Meta, 1)
		} else if req.Limit <= 100 {
			srv.metadataMap = make(map[string][]metadatapb.Meta, req.Limit)
		} else {
			srv.metadataMap = make(map[string][]metadatapb.Meta)
		}
	} else {
		srv.metadataMap = make(map[string][]metadatapb.Meta)
	}

	if err := rr.proxy.MetricMetadata(req, srv); err != nil {
		return nil, nil, errors.Wrap(err, "proxy MetricMetadata")
	}

	return srv.metadataMap, srv.warnings, nil
}

type metadataServer struct {
	// This field just exist to pseudo-implement the unused methods of the interface.
	metadatapb.Metadata_MetricMetadataServer
	ctx context.Context

	metric string
	limit  int

	warnings    annotations.Annotations
	metadataMap map[string][]metadatapb.Meta
	mu          sync.Mutex
}

View on GitHub (pinned to 35b8b99117)