What causes "label set contains a label with empty name or value" in thanos-io/thanos?

Drop labels whose value is empty before writing (relabel action: labeldrop or keep only non-empty).

label set contains a label with empty name or value

Error message

label set contains a label with empty name or value

What it means

ErrEmptyLabels is a sentinel error indicating a label set contains a label with an empty name or an empty value (or the whole set is empty in the receive writer path, where an empty set is also rejected). The capnproto writer's validateLabels returns it to reject writes that would produce ambiguous or unrepresentable series.

Solutions

Drop labels whose value is empty before writing (relabel action: labeldrop or keep only non-empty).
Ensure __name__ / metric name is non-empty in the write payload.
For intentionally empty series, attach at least __name__ so the label set is non-empty.

Example fix

// before
metric_relabel_configs:
  - action: replace
    source_labels: [missing]
    target_label: zone
    # produces zone=""
// after
metric_relabel_configs:
  - action: labeldrop
    regex: zone_tmp  # never create empty-valued labels

Defensive patterns

Strategy: validation

Validate before calling

func hasEmptyLabels(lbls labels.Labels) error {
	if lbls.Len() == 0 {
		return errors.New("empty label set")
	}
	return lbls.Validate(func(l labels.Label) error {
		if l.Name == "" || l.Value == "" {
			return fmt.Errorf("empty name or value in %q=%q", l.Name, l.Value)
		}
		return nil
	})
}

Type guard

func isEmptyLabelsErr(err error) bool { return errors.Is(err, labelpb.ErrEmptyLabels) }

Try / catch

if err := writer.Write(ctx, series); err != nil {
	if errors.Is(err, labelpb.ErrEmptyLabels) {
		// drop empty-valued labels client-side and re-send
		b := labels.NewBuilder(series.Labels)
		b.Sort()
		series.Labels = dropEmpty(b.Labels())
		return writer.Write(ctx, series)
	}
	return err
}

Prevention

Use metric_relabel_configs to drop labels with empty values before remote-write.
Never target labels from source_labels that may be absent with replace action.
Validate exemplar label sets client-side before sending.

When it happens

Trigger: Remote-writing a series with an empty label name ({}=value), an empty label value (foo=""), or a series with zero labels through pkg/receive/capnproto_writer.go validateLabels (lbls.Len() == 0) or exemplar label validation.

Common situations: Prometheus relabel_configs that drop label values leaving foo=""; client libraries emitting empty metric names; dropping __name__ via metric_relabel_configs; sending exemplars with empty label sets.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5d282935f3dba12b. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/labelpb/label.go:27

import (
	"encoding/json"
	"fmt"
	"io"
	"sort"
	"strings"
	"sync"
	"unique"
	"unsafe"

	"github.com/VictoriaMetrics/easyproto"
	"github.com/cespare/xxhash/v2"
	"github.com/pkg/errors"
	"github.com/prometheus/prometheus/model/labels"
)

var (
	ErrOutOfOrderLabels = errors.New("out of order labels")
	ErrEmptyLabels      = errors.New("label set contains a label with empty name or value")
	ErrDuplicateLabels  = errors.New("label set contains duplicate label names")

	sep = []byte{'\xff'}
)

func noAllocString(buf []byte) string {
	return *(*string)(unsafe.Pointer(&buf))
}

func noAllocBytes(buf string) []byte {
	return *(*[]byte)(unsafe.Pointer(&buf))
}

// ZLabelsFromPromLabels converts Prometheus labels to slice of labelpb.ZLabel in type unsafe manner.
// It reuses the same memory. Caller should abort using passed labels.Labels.
func ZLabelsFromPromLabels(lset labels.Labels) []ZLabel {
	return *(*[]ZLabel)(unsafe.Pointer(&lset))
}

View on GitHub (pinned to 35b8b99117)