thanos-io/thanos · warning

error sending proto response

Error message

error sending proto response: %v

What it means

SerializeProtoResponse writes a snappy-compressed (or raw) protobuf payload to an http.ResponseWriter. If the underlying Write fails, it also attempts an http.Error fallback and returns this wrapped error. It signals that the client never received a complete proto response because the connection or writer broke mid-send.

Solutions

1. Inspect server logs for the wrapped %v cause (broken pipe, connection reset, timeout) to confirm client-side disconnects vs server misconfiguration.
2. Verify no handler code path writes to the ResponseWriter before SerializeProtoResponse (double-write commits headers early).
3. Check proxy/load-balancer idle and read timeouts versus slow query durations.
4. Catch the returned error in the handler and rely on it being logged; the 500 fallback is already sent to the client.

Example fix

// before
if err := util.SerializeProtoResponse(w, resp, compression); err != nil {
    level.Error(logger).Log("msg", "serialize failed", "err", err)
}
// after
if err := util.SerializeProtoResponse(w, resp, compression); err != nil {
    if errors.Is(err, syscall.EPIPE) || errors.Is(err, syscall.ECONNRESET) {
        level.Debug(logger).Log("msg", "client disconnected while sending proto response", "err", err)
    } else {
        level.Error(logger).Log("msg", "serialize failed", "err", err)
    }
}

// before writing, ensure headers not yet sent and response body supports writes
if w.Header().Get("Content-Type") == "" { w.Header().Set("Content-Type", contentTypeProtobuf) }

if err := util.SerializeProtoResponse(w, resp, comp); err != nil {
    var opErr *net.OpError
    if errors.As(err, &opErr) { // client-side network failure: log at debug, no retry on committed response
        level.Debug(logger).Log("err", err)
    } else {
        level.Error(logger).Log("err", err)
    }
}

When it happens

Trigger: Calling SerializeProtoResponse (directly or via an HTTP handler that returns protobuf data) when the client has disconnected, the connection has timed out, or the ResponseWriter has already been used/committed before this call.

Common situations: Slow clients hitting server write timeouts; clients canceling queries (common with Grafana/Prometheus query cancellations); reverse proxies closing upstream connections; handlers that already wrote headers or another error before serializing the proto response.

Related errors

• error starting web server
• failed to validate prometheus flags
• error marshaling proto response
• send request to
• perform request against

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/ecc95e49f56bb82d. Report an issue: GitHub.

Appendix: source

Thrown at internal/cortex/util/http.go:268

	return nil, false
}

// SerializeProtoResponse serializes a protobuf response into an HTTP response.
func SerializeProtoResponse(w http.ResponseWriter, resp proto.Message, compression CompressionType) error {
	data, err := proto.Marshal(resp)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return fmt.Errorf("error marshaling proto response: %v", err)
	}

	switch compression {
	case NoCompression:
	case RawSnappy:
		data = snappy.Encode(nil, data)
	}

	if _, err := w.Write(data); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return fmt.Errorf("error sending proto response: %v", err)
	}
	return nil
}

View on GitHub (pinned to 35b8b99117)