thanos-io/thanos · error

marshal samples

Error message

marshal samples

What it means

BuildInto calls marshalSamples to copy []prompb.Sample (timestamp/value pairs) into the capnp TimeSeries struct. A failure there — typically capnp list allocation or field-write failure — is wrapped as 'marshal samples'.

Solutions

1. Limit samples per request / chunk the series before BuildInto
2. Downsample or truncate sample history per request
3. Increase capnp buffer limits if configurable
4. Check capnp library version for known list-size limits

Example fix

// before
err := marshalSamples(tsc, ts.Samples) // unbounded
// after
if len(ts.Samples) > maxSamplesPerSeries {
    ts.Samples = ts.Samples[len(ts.Samples)-maxSamplesPerSeries:]
}
err := marshalSamples(tsc, ts.Samples)

if len(ts.Samples) > maxSamplesPerSeries {
    return errors.New("too many samples in one series for a single message")
}

if err := BuildInto(tuple, tenant, tsreq, builder); err != nil && strings.Contains(err.Error(), "marshal samples") {
    // chunk series into smaller batches and retry
    return splitAndRetry(tsreq)
}

When it happens

Trigger: marshalSamples fails while allocating the sample list or writing float64/int64 fields, e.g. with an enormous number of samples per series in one message.

Common situations: Very long time ranges batched into a single request; memory pressure on the sender; capnp message size limits hit mid-marshal.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• set tenant
• new labels
• marshal labels
• marshal histograms
• marshal exemplars

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e008e99bda7bbb6d. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writecapnp/marshal.go:87

		return errors.Wrap(err, "set tenant")
	}

	series, err := wr.NewTimeSeries(int32(len(tsreq)))
	if err != nil {
		return err
	}
	for i, ts := range tsreq {
		tsc := series.At(i)

		lblsc, err := tsc.NewLabels(int32(len(ts.Labels)))
		if err != nil {
			return errors.Wrap(err, "new labels")
		}
		if err := marshalLabels(lblsc, ts.Labels, builder); err != nil {
			return errors.Wrap(err, "marshal labels")
		}
		if err := marshalSamples(tsc, ts.Samples); err != nil {
			return errors.Wrap(err, "marshal samples")
		}
		if err := marshalHistograms(tsc, ts.Histograms); err != nil {
			return errors.Wrap(err, "marshal histograms")
		}
		if err := marshalExemplars(tsc, ts.Exemplars, builder); err != nil {
			return errors.Wrap(err, "marshal exemplars")
		}
	}

	return nil
}

func BuildIntoSingleTenantWriteRequest(wr WriteRequest, tenant string, tsreq []prompb.TimeSeries) error {
	if err := wr.SetTenant(tenant); err != nil {
		return errors.Wrap(err, "set tenant")
	}

	series, err := wr.NewTimeSeries(int32(len(tsreq)))

View on GitHub (pinned to 35b8b99117)