prometheus/prometheus · error

invalid source label name in label_join(): %s

Error message

invalid source label name in label_join(): %s

What it means

evalLabelJoin validates every source label argument (args[3..]) with IsValidLabelName before evaluating the inner vector; an invalid one panics with 'invalid source label name in label_join(): <src>', converted by the engine into a query error. Note the check validates the NAME of the source label, not its value.

Source

Thrown at promql/functions.go:2541

func funcVector(vectorVals []Vector, _ Matrix, _ parser.Expressions, enh *EvalNodeHelper) (Vector, annotations.Annotations) {
	return append(enh.Out,
		Sample{
			Metric: labels.Labels{},
			F:      vectorVals[0][0].F,
		}), nil
}

// label_join function operates only on series; does not look at timestamps or values.
func (ev *evaluator) evalLabelJoin(ctx context.Context, args parser.Expressions) (parser.Value, annotations.Annotations) {
	var (
		dst       = stringFromArg(args[1])
		sep       = stringFromArg(args[2])
		srcLabels = make([]string, len(args)-3)
	)
	for i := 3; i < len(args); i++ {
		src := stringFromArg(args[i])
		if !model.UTF8Validation.IsValidLabelName(src) {
			panic(fmt.Errorf("invalid source label name in label_join(): %s", src))
		}
		srcLabels[i-3] = src
	}
	if !model.UTF8Validation.IsValidLabelName(dst) {
		panic(fmt.Errorf("invalid destination label name in label_join(): %s", dst))
	}

	val, ws := ev.eval(ctx, args[0])
	matrix := val.(Matrix)
	srcVals := make([]string, len(srcLabels))
	lb := labels.NewBuilder(labels.EmptyLabels())

	for i, el := range matrix {
		for i, src := range srcLabels {
			srcVals[i] = el.Metric.Get(src)
		}
		strval := strings.Join(srcVals, sep)
		lb.Reset(el.Metric)

View on GitHub (pinned to 44d6a0e0b1)

Solutions

1. Use plain valid label names for every source argument of label_join
2. If the list is generated, validate each src against a label-name check before building the query
3. Remove quotes/whitespace artifacts introduced by templating or copy-paste

Example fix

# before
label_join(up, "target", "/", "job", "instance ")

# after
label_join(up, "target", "/", "job", "instance")

for (const src of srcLabels) {
  if (!/^[a-zA-Z_][a-zA-Z0-9_]*$/.test(src)) throw new Error(`invalid source label: ${src}`);
}

const isValidLabelName = (s: string): boolean => /^[a-zA-Z_][a-zA-Z0-9_]*$/.test(s);

When it happens

Trigger: Calling label_join(v, dst, sep, src1, src2, ...) where any src argument is not a valid label name — contains invalid characters/UTF-8, or was mangled by templating (e.g. "instance " with a trailing space or a ${var} that rendered empty/malformed).

Common situations: Dynamic source-label lists from dashboard variables; refactoring label_replace queries into label_join and reusing a regex fragment as a label name; quoting mistakes where the src becomes '"job"' with embedded quotes.

Related errors

• invalid destination label name in label_join(): %s
• invalid destination label name in label_replace(): %s
• invalid smoothing factor. Expected: 0 < sf < 1, got: %f
• invalid trend factor. Expected: 0 < tf < 1, got: %f
• invalid regular expression in label_replace(): %s

AI-assisted analysis of prometheus/prometheus@44d6a0e0b1 (2026-08-15). Data as JSON: /api/errors/5f37f9a97ea9b7e2. Report an issue: GitHub.