thanos-io/thanos · warning

add series

Error message

add %d series

What it means

Aggregated error from Writer.Write produced by writeErrorTracker.collectErrors: N series were dropped because their label sets were out of order (labelpb.ErrOutOfOrderLabels wrapped as 'add %d series'). Labels are out of order when not sorted lexicographically by name as Prometheus requires, so the whole series is rejected before appending.

Solutions

1. Sort the series' labels by name before sending (labels.Labels.Sort or sort.Slice by Name).
2. Check which series are dropped via the receiver's warn log 'Error on series with out-of-order labels'.
3. Fix the producing exporter/library to emit sorted, unique labels per Prometheus write spec.
4. Upgrade the sending Prometheus - it sorts labels before remote write.

Example fix

// before
lbls := []labelpb.ZLabel{{Name: "z", Value: "1"}, {Name: "a", Value: "2"}}
// after
sort.Slice(lbls, func(i, j int) bool { return lbls[i].Name < lbls[j].Name })

sort.Slice(series[i].Labels, func(a, b int) bool { return series[i].Labels[a].Name < series[i].Labels[b].Name })
if err := labelpb.ValidateLabels(series[i].Labels); err != nil {
    return err // drop or fix before sending
}

if err := writer.Write(ctx, tenant, series); err != nil {
    var we receive.WriteErrors // inspect aggregated errors
    if strings.Contains(err.Error(), labelpb.ErrOutOfOrderLabels.Error()) {
        // fix sender labeling; series were dropped
    }
}

When it happens

Trigger: Remote-write payload contains time series whose Labels slice is not sorted by label name; ValidateLabels returns ErrOutOfOrderLabels for one or more series and the count is wrapped at the end of Write.

Common situations: Custom senders/SDKs building label sets manually without sorting; exporters emitting unsorted labels; middleware rewriting labels and breaking sort order.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• failed to parse the template
• failed to execute the template
• get tenant appendable
• get appender
• commit samples

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a091b814004e05e7. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/writer_errors.go:122

	case errors.Is(err, storage.ErrOutOfOrderExemplar):
		a.numExemplarsOutOfOrder++
		level.Debug(exLogger).Log("msg", "Out of order exemplar")
	case errors.Is(err, storage.ErrDuplicateExemplar):
		a.numExemplarsDuplicate++
		level.Debug(exLogger).Log("msg", "Duplicate exemplar")
	case errors.Is(err, storage.ErrExemplarLabelLength):
		a.numExemplarsLabelLength++
		level.Debug(exLogger).Log("msg", "Label length for exemplar exceeds max limit", "limit", exemplar.ExemplarMaxLabelSetLength)
	default:
		level.Debug(exLogger).Log("msg", "Error ingesting exemplar", "err", err)
	}
}

func (a *writeErrorTracker) collectErrors(tLogger log.Logger) writeErrors {
	var errs writeErrors
	if a.numLabelsOutOfOrder > 0 {
		level.Warn(tLogger).Log("msg", "Error on series with out-of-order labels", "numDropped", a.numLabelsOutOfOrder)
		errs.Add(errors.Wrapf(labelpb.ErrOutOfOrderLabels, "add %d series", a.numLabelsOutOfOrder))
	}
	if a.numLabelsDuplicates > 0 {
		level.Warn(tLogger).Log("msg", "Error on series with duplicate labels", "numDropped", a.numLabelsDuplicates)
		errs.Add(errors.Wrapf(labelpb.ErrDuplicateLabels, "add %d series", a.numLabelsDuplicates))
	}
	if a.numLabelsEmpty > 0 {
		level.Warn(tLogger).Log("msg", "Error on series with empty label name or value", "numDropped", a.numLabelsEmpty)
		errs.Add(errors.Wrapf(labelpb.ErrEmptyLabels, "add %d series", a.numLabelsEmpty))
	}

	if a.numSamplesOutOfOrder > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting out-of-order samples", "numDropped", a.numSamplesOutOfOrder)
		errs.Add(errors.Wrapf(storage.ErrOutOfOrderSample, "add %d samples", a.numSamplesOutOfOrder))
	}
	if a.numSamplesDuplicates > 0 {
		level.Warn(tLogger).Log("msg", "Error on ingesting samples with different value but same timestamp", "numDropped", a.numSamplesDuplicates)
		errs.Add(errors.Wrapf(storage.ErrDuplicateSampleForTimestamp, "add %d samples", a.numSamplesDuplicates))
	}

View on GitHub (pinned to 35b8b99117)