thanos-io/thanos · error

out of order labels

Error message

out of order labels

What it means

ErrOutOfOrderLabels is a sentinel error declared in pkg/store/labelpb signaling that a label set's labels are not sorted in lexicographic order by name, which the columnar/capnp proto encoding requires. Writers like the receive component's capnproto writer return it when consecutive label names compare in the wrong order during validation or writing.

Solutions

1. Sort the label set by label name before writing (labels.New builds a sorted set; use labels.Labels.Range after sorting).
2. Ensure relabeling/external-label injection keeps labels lexicographically ordered.
3. Use compare/merge helpers (labels.Labels.Compare) in custom writers to enforce ordering.

Example fix

// before
lbls := labels.FromStrings("zone", "a", "app", "b") // unsorted
// after
lbls := labels.FromStrings("app", "b", "zone", "a") // sorted by name

func labelsSorted(lbls labels.Labels) bool {
	return lbls.Validate(func(l labels.Label) error { return nil }) == nil &&
		func() bool {
			prev := ""
			ok := true
			lbls.Range(func(l labels.Label) error {
				if l.Name <= prev { ok = false }
				prev = l.Name
				return nil
			})
			return ok
		}()
}

func isOutOfOrderLabels(err error) bool { return errors.Is(err, labelpb.ErrOutOfOrderLabels) }

if err := writer.Write(ctx, series); err != nil {
	if errors.Is(err, labelpb.ErrOutOfOrderLabels) {
		// sort labels and retry once
		sorted := labels.NewBuilder(series.Labels).Labels()
		series.Labels = sorted
		return writer.Write(ctx, series)
	}
	return err
}

When it happens

Trigger: Writing a TimeSeries whose labels are not sorted by name (e.g. [zone=a, env=b]) via the capnproto writer (pkg/receive/capnproto_writer.go validateLabels/addLabelsError), or decoding series data that violates the ordering invariant.

Common situations: Custom relabeling or external code appending labels in arbitrary order before remote-write ingestion; hand-built labels.Labels not passed through labels.New(); forwarding writes between Thanos Receive clients with differently-ordered label sets.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• add series
• label set contains a label with empty name or value
• label set contains duplicate label names
• unsupported format for label
• invalid label name

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/7289c5d4c9b0da71. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/labelpb/label.go:26

import (
	"encoding/json"
	"fmt"
	"io"
	"sort"
	"strings"
	"sync"
	"unique"
	"unsafe"

	"github.com/VictoriaMetrics/easyproto"
	"github.com/cespare/xxhash/v2"
	"github.com/pkg/errors"
	"github.com/prometheus/prometheus/model/labels"
)

var (
	ErrOutOfOrderLabels = errors.New("out of order labels")
	ErrEmptyLabels      = errors.New("label set contains a label with empty name or value")
	ErrDuplicateLabels  = errors.New("label set contains duplicate label names")

	sep = []byte{'\xff'}
)

func noAllocString(buf []byte) string {
	return *(*string)(unsafe.Pointer(&buf))
}

func noAllocBytes(buf string) []byte {
	return *(*[]byte)(unsafe.Pointer(&buf))
}

// ZLabelsFromPromLabels converts Prometheus labels to slice of labelpb.ZLabel in type unsafe manner.
// It reuses the same memory. Caller should abort using passed labels.Labels.
func ZLabelsFromPromLabels(lset labels.Labels) []ZLabel {
	return *(*[]ZLabel)(unsafe.Pointer(&lset))

View on GitHub (pinned to 35b8b99117)