thanos-io/thanos · error

marshal symbols

Error message

marshal symbols

What it means

Wrap of the error from marshalSymbols, which writes the accumulated string symbol table (offsets list plus packed data) into the capnp WriteRequest. Failure means symbols.NewOffsets(len) or symbols.SetData failed; the raw capnp cause is wrapped and indicates an allocation problem on the message.

Solutions

1. Inspect the wrapped capnp cause for which allocation (NewOffsets vs SetData) failed.
2. Reduce label cardinality or split the write into multiple requests.
3. Build the WriteRequest on a fresh capnp message/arena for each call.
4. Retry; if reproducible with normal payloads, check capnp library version.

Example fix

// before
err := BuildIntoSingleTenantWriteRequest(wr, tenant, highCardinalitySeries)
// after: split by cardinality budget
for _, part := range splitSeriesByUniqueStrings(highCardinalitySeries, 10000) {
    err = BuildIntoSingleTenantWriteRequest(wr, tenant, part)
}

unique := map[string]struct{}{}
for _, ts := range series {
    for _, l := range ts.Labels { unique[l.Name+"\x00"+l.Value] = struct{}{} }
}
if len(unique) > maxSymbolsPerRequest { return errors.New("label cardinality too high for one request") }

if err := BuildIntoSingleTenantWriteRequest(wr, tenant, series); err != nil {
    if strings.Contains(err.Error(), "marshal symbols") {
        // split series into smaller requests and retry
    }
}

When it happens

Trigger: BuildIntoSingleTenantWriteRequest calling marshalSymbols after collecting all label/exemplar strings, when the capnp message cannot allocate the offsets list or data bytes (arena exhausted, invalid message).

Common situations: Very high label cardinality (many unique label names/values) in one write request exhausting the segment; reusing a finalized capnp message.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• new symbols
• set tenant
• new labels
• marshal labels
• marshal samples

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b978c097029bdf12. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writecapnp/marshal.go:136

			return errors.Wrap(err, "marshal labels")
		}
		if err := marshalSamples(tsc, ts.Samples); err != nil {
			return errors.Wrap(err, "marshal samples")
		}
		if err := marshalHistograms(tsc, ts.Histograms); err != nil {
			return errors.Wrap(err, "marshal histograms")
		}
		if err := marshalExemplars(tsc, ts.Exemplars, builder); err != nil {
			return errors.Wrap(err, "marshal exemplars")
		}
	}

	symbols, err := wr.NewSymbols()
	if err != nil {
		return errors.Wrap(err, "new symbols")
	}
	if err := marshalSymbols(builder, symbols); err != nil {
		return errors.Wrap(err, "marshal symbols")
	}
	return nil
}

func marshalSymbols(builder *symboltable.Builder, symbols Symbols) error {
	offsets, err := symbols.NewOffsets(builder.Len())
	if err != nil {
		return err
	}
	data := make([]byte, builder.SymbolsSize)
	for k, entry := range builder.Table {
		end := entry.Start + uint32(len(k))
		copy(data[entry.Start:end], k)
		offsets.Set(int(entry.Index), end)
	}

	return symbols.SetData(data)
}

View on GitHub (pinned to 35b8b99117)