thanos-io/thanos · error

marshal histograms

Error message

marshal histograms

What it means

BuildInto calls marshalHistograms to convert prompb.Histogram data (native histograms) into capnp structures. Failure during histogram list allocation or field encoding is wrapped as 'marshal histograms'.

Solutions

1. Verify sender and receiver support the same native histogram capnp schema (upgrade/downgrade to match)
2. Reduce histogram bucket resolution/count
3. Chunk batches to reduce allocation pressure
4. Update the capnp/prompb dependencies to compatible versions

Example fix

// before
 histograms forwarded as-is
// after
if len(h.BucketCounts) > maxBuckets { merge adjacent buckets }
err := marshalHistograms(tsc, ts.Histograms)

for _, h := range ts.Histograms {
    if len(h.GetPositiveCounts())+len(h.GetNegativeCounts()) > maxBuckets {
        return errors.New("histogram bucket count too large")
    }
}

if err := BuildInto(tuple, tenant, tsreq, builder); err != nil && strings.Contains(err.Error(), "marshal histograms") {
    // check schema compatibility, reduce bucket resolution, retry
    return retryWithCoarserHistograms(tsreq)
}

When it happens

Trigger: marshalHistograms fails allocating/encoding histogram buckets or counts — common with high-bucket-count native histograms or many histograms per series.

Common situations: Sender and receiver on mismatched versions where native histogram schemas differ; extremely fine histogram bucket resolutions; capnp arena exhaustion.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• set tenant
• new labels
• marshal labels
• marshal samples
• marshal exemplars

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b450948055e7af31. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writecapnp/marshal.go:90

	series, err := wr.NewTimeSeries(int32(len(tsreq)))
	if err != nil {
		return err
	}
	for i, ts := range tsreq {
		tsc := series.At(i)

		lblsc, err := tsc.NewLabels(int32(len(ts.Labels)))
		if err != nil {
			return errors.Wrap(err, "new labels")
		}
		if err := marshalLabels(lblsc, ts.Labels, builder); err != nil {
			return errors.Wrap(err, "marshal labels")
		}
		if err := marshalSamples(tsc, ts.Samples); err != nil {
			return errors.Wrap(err, "marshal samples")
		}
		if err := marshalHistograms(tsc, ts.Histograms); err != nil {
			return errors.Wrap(err, "marshal histograms")
		}
		if err := marshalExemplars(tsc, ts.Exemplars, builder); err != nil {
			return errors.Wrap(err, "marshal exemplars")
		}
	}

	return nil
}

func BuildIntoSingleTenantWriteRequest(wr WriteRequest, tenant string, tsreq []prompb.TimeSeries) error {
	if err := wr.SetTenant(tenant); err != nil {
		return errors.Wrap(err, "set tenant")
	}

	series, err := wr.NewTimeSeries(int32(len(tsreq)))
	if err != nil {
		return err
	}

View on GitHub (pinned to 35b8b99117)