thanos-io/thanos · error

request.At

Error message

request.At

What it means

This error wraps a failure to decode the next series from a Cap'n Proto write request inside CapNProtoWriter.Write. The receive writer streams WriteRequest messages; wreq.At(&series) fails when the wire payload is malformed or does not match the expected capnp schema. It means the incoming replication/forwarding message is corrupt or schema-incompatible.

Solutions

1. Upgrade/align Thanos versions across all Receive peers so the capnp schema matches
2. Check network proxies/load balancers for frame truncation or corruption
3. Log the raw failing request and compare against the expected writecapnp schema
4. Retry the replication from the sender; the error is per-series and the writer skips to the next series after reporting

// sender side: verify request size and non-nil before sending
if req == nil || len(req.Timeseries) == 0 {
	return errors.New("empty write request")
}

err := writer.Write(ctx, wreq)
var wrapped interface{ Cause() error }
if errors.As(err, &wrapped) {
	log.Printf("capnp decode failed: %v", errors.Cause(err))
	// request sender re-replicate this batch
}

When it happens

Trigger: wreq.Next() yields a message and wreq.At(&series) fails during decode — corrupt frames on the wire, truncated request, or sender/receiver capnp schema version mismatch.

Common situations: Mixed Thanos versions in a hashring where the capnp write protocol differs; network middleware truncating or mangling frames; a buggy sender serializing invalid label/sample data.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• invalid duration
• bad cached type
• failed to dial peer
• failed writing to peer
• rpc failed

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/27303f443b1df243. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/capnproto_writer.go:75

	}
	getRef := app.(storage.GetRef)
	var (
		ref          storage.SeriesRef
		errorTracker = &writeErrorTracker{}
	)
	app = &ReceiveAppender{
		tLogger:        tLogger,
		tooFarInFuture: r.opts.TooFarInFutureTimeWindow,
		Appender:       app,
	}

	var (
		series  writecapnp.Series
		builder labels.ScratchBuilder
	)
	for wreq.Next() {
		if err := wreq.At(&series); err != nil {
			return errors.Wrap(err, "request.At")
		}

		// Check if time series labels are valid. If not, skip the time series
		// and report the error.
		if err := validateLabels(series.Labels); err != nil {
			lset := &labelpb.ZLabelSet{Labels: labelpb.ZLabelsFromPromLabels(series.Labels)}
			errorTracker.addLabelsError(err, lset, tLogger)
			continue
		}

		var lset labels.Labels
		// Check if the TSDB has cached reference for those labels.
		ref, lset = getRef.GetRef(series.Labels, series.Labels.Hash())
		if ref == 0 {
			// NOTE(GiedriusS): do a deep copy because the labels are reused in the capnp message.
			// Creation of new series is much rarer compared to adding extra samples
			// to an existing series.
			builder.Reset()

View on GitHub (pinned to 35b8b99117)