thanos-io/thanos · error

marshal exemplars

Error message

marshal exemplars

What it means

In BuildInto (writecapnp marshal), marshalExemplars failed while copying a timeseries' exemplars into the capnproto replication message — typically a capnproto list/allocation error (e.g. message capacity exhausted for very large exemplar sets) — so the whole WriteRequest replication payload cannot be built.

Solutions

1. Limit the number of exemplars per series before marshaling
2. Truncate/sanitize exemplar label values (e.g. long trace ids)
3. Chunk the WriteRequest into smaller batches
4. Check capnp library version for text-field size constraints

Example fix

// before
exemplars := ts.Exemplars
// after
if len(ts.Exemplars) > maxExemplars {
    ts.Exemplars = ts.Exemplars[:maxExemplars]
}
err := marshalExemplars(tsc, ts.Exemplars, builder)

if len(ts.Exemplars) > maxExemplarsPerSeries {
    return errors.New("too many exemplars per series")
}
for _, e := range ts.Exemplars {
    for _, l := range e.GetLabels() { if len(l.GetValue()) > 4096 { return errors.New("exemplar label too long") } }
}

if err := BuildInto(tuple, tenant, tsreq, builder); err != nil && strings.Contains(err.Error(), "marshal exemplars") {
    // retry with exemplars stripped
    return rebuildWithoutExemplars(tsreq)
}

When it happens

Trigger: marshalExemplars fails during exemplar list allocation or while writing exemplar label strings — typically arena exhaustion or invalid exemplar label data.

Common situations: Exemplar-heavy workloads (traces attached to many samples); very long trace_id span_id label values; batches near capnp size limits.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• set tenant
• new labels
• marshal labels
• marshal samples
• marshal histograms

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/bd7bc6fc90161ddb. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writecapnp/marshal.go:93

	}
	for i, ts := range tsreq {
		tsc := series.At(i)

		lblsc, err := tsc.NewLabels(int32(len(ts.Labels)))
		if err != nil {
			return errors.Wrap(err, "new labels")
		}
		if err := marshalLabels(lblsc, ts.Labels, builder); err != nil {
			return errors.Wrap(err, "marshal labels")
		}
		if err := marshalSamples(tsc, ts.Samples); err != nil {
			return errors.Wrap(err, "marshal samples")
		}
		if err := marshalHistograms(tsc, ts.Histograms); err != nil {
			return errors.Wrap(err, "marshal histograms")
		}
		if err := marshalExemplars(tsc, ts.Exemplars, builder); err != nil {
			return errors.Wrap(err, "marshal exemplars")
		}
	}

	return nil
}

func BuildIntoSingleTenantWriteRequest(wr WriteRequest, tenant string, tsreq []prompb.TimeSeries) error {
	if err := wr.SetTenant(tenant); err != nil {
		return errors.Wrap(err, "set tenant")
	}

	series, err := wr.NewTimeSeries(int32(len(tsreq)))
	if err != nil {
		return err
	}
	builder := symboltable.NewBuilder()
	for i, ts := range tsreq {
		tsc := series.At(i)

View on GitHub (pinned to 35b8b99117)