thanos-io/thanos · error

new labels

Error message

new labels

What it means

Inside BuildInto, for each incoming prometheus TimeSeries, tsc.NewLabels(n) allocates a capnp list of n Label entries. When capnp cannot allocate that list in the message arena, the error is wrapped as 'new labels'.

Solutions

1. Split the WriteRequest into smaller batches before calling BuildInto
2. Drop or normalize labels with very high cardinality (e.g. via relabeling)
3. Increase the capnp message size limit/arena if the library exposes it
4. Check for capnp library version notes on list allocation limits

Example fix

// before
err := BuildInto(tuple, tenant, allSeries, builder)
// after
for chunk := range slices.Chunk(allSeries, 1000) {
    if err := BuildInto(tuple, tenant, chunk, builder); err != nil {
        return err
    }
}

func maxLabelCount(series []prompb.TimeSeries) int {
    m := 0
    for _, ts := range series { if len(ts.Labels) > m { m = len(ts.Labels) } }
    return m
}
// call BuildInto only if maxLabelCount(tsreq) <= 128

if err := BuildInto(tuple, tenant, tsreq, builder); err != nil && strings.Contains(err.Error(), "new labels") {
    // split the batch in half and retry each half
    return splitAndRetry(tsreq)
}

When it happens

Trigger: A time series with a very large number of labels, or a batch so large the capnp arena/segment limit is exhausted while allocating the label list for series index i.

Common situations: Users scraping targets with hundreds of labels; huge WriteRequest batches exceeding capnp message size limits; memory pressure on the sender.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• set tenant
• marshal labels
• marshal samples
• marshal histograms
• marshal exemplars

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8cfdad3655fd01b5. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writecapnp/marshal.go:81

	}
	return wr, nil
}

func BuildInto(wr *TimeSeriesTenantTuple, tenant string, tsreq []prompb.TimeSeries, builder *symboltable.Builder) error {
	if err := wr.SetTenant(tenant); err != nil {
		return errors.Wrap(err, "set tenant")
	}

	series, err := wr.NewTimeSeries(int32(len(tsreq)))
	if err != nil {
		return err
	}
	for i, ts := range tsreq {
		tsc := series.At(i)

		lblsc, err := tsc.NewLabels(int32(len(ts.Labels)))
		if err != nil {
			return errors.Wrap(err, "new labels")
		}
		if err := marshalLabels(lblsc, ts.Labels, builder); err != nil {
			return errors.Wrap(err, "marshal labels")
		}
		if err := marshalSamples(tsc, ts.Samples); err != nil {
			return errors.Wrap(err, "marshal samples")
		}
		if err := marshalHistograms(tsc, ts.Histograms); err != nil {
			return errors.Wrap(err, "marshal histograms")
		}
		if err := marshalExemplars(tsc, ts.Exemplars, builder); err != nil {
			return errors.Wrap(err, "marshal exemplars")
		}
	}

	return nil
}

View on GitHub (pinned to 35b8b99117)