What causes "send metric metadata response" in thanos-io/thanos?

Check the wrapped inner error to see whether it is 'no metadata' vs a canceled stream

Unknown

Unknown

Error message

send metric metadata response

What it means

In the metadata proxy's MetricMetadata handler, srv.Send is invoked per metadata batch inside a span; any send failure is converted to a gRPC status.Error(codes.Unknown, ...) wrapped with this message. It means the server could not push a metadata response onto the client stream.

Solutions

Check the wrapped inner error to see whether it is 'no metadata' vs a canceled stream
Ensure the client keeps the stream open until results are consumed
Upgrade to matching cloud/Vizier versions to avoid empty metadata messages
Avoid concurrent use of the stream server (data race) — serialize Send calls

Defensive patterns

Strategy: try-catch

Try / catch

_, err := client.MetricMetadata(ctx, req)
if err != nil {
    st, ok := status.FromError(err)
    if ok && st.Code() == codes.Unknown && strings.Contains(st.Message(), "send metric metadata response") {
        // stream broke mid-response; retry or reconnect
    }
}

Prevention

Do not cancel the client stream before consuming all results
Serialize access to the stream server to avoid data races (see TestProxyDataRace)
Keep gRPC keepalives configured for long metadata scans

When it happens

Trigger: srv.Send returns an error while streaming metadata batches — e.g. the 'no metadata' guard, or the client canceled/closed the stream mid-flight.

Common situations: Client disconnects during a large metadata scan; warnings/nil payloads triggering metadataServer.Send errors; seen in TestProxyDataRace when concurrent sends misbehave.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/7f6ac7d977259063. Report an issue: GitHub.

Appendix: source

Thrown at pkg/metadata/proxy.go:83

		_ = g.Wait()
		close(respChan)
	}()

	for resp := range respChan {
		metas = append(metas, resp)
	}

	if err := g.Wait(); err != nil {
		level.Error(s.logger).Log("err", err)
		return err
	}

	for _, t := range metas {
		tracing.DoInSpan(srv.Context(), "send_metadata_response", func(_ context.Context) {
			err = srv.Send(metadatapb.NewMetricMetadataResponse(t))
		})
		if err != nil {
			return status.Error(codes.Unknown, errors.Wrap(err, "send metric metadata response").Error())
		}
	}

	return nil
}

type metricMetadataStream struct {
	client  metadatapb.MetadataClient
	request *metadatapb.MetricMetadataRequest
	channel chan<- *metadatapb.MetricMetadata
	server  metadatapb.Metadata_MetricMetadataServer
}

func (stream *metricMetadataStream) receive(ctx context.Context) error {
	var (
		err         error
		metadataCli metadatapb.Metadata_MetricMetadataClient
	)

View on GitHub (pinned to 35b8b99117)