metric label '%s' is invalid: keys may only contain alphanum

Error message

metric label '%s' is invalid: keys may only contain alphanumeric characters or '_'

What it means

ValidateMetricLabels checks that every metric label KEY is a valid Prometheus metric-style name (alphanumerics and '_', no ':'), reusing IsValidMetricName. Label keys with hyphens, dots, or other characters are rejected during template validation, since Prometheus forbids such label keys.

Source

Thrown at workflow/metrics/util.go:46

		if metric.Gauge.Realtime != nil && *metric.Gauge.Realtime {
			if strings.Contains(metric.Gauge.Value, "resourcesDuration.") {
				return errors.New("'resourcesDuration.*' metrics cannot be used in real-time")
			}
		}
	}
	if metric.Counter != nil && metric.Counter.Value == "" {
		return errors.New("missing counter.value")
	}
	if metric.Histogram != nil && metric.Histogram.Value == "" {
		return errors.New("missing histogram.value")
	}
	return nil
}

func ValidateMetricLabels(metrics map[string]string) error {
	for name := range metrics {
		if !IsValidMetricName(name) {
			return fmt.Errorf(invalidMetricLabelError, name)
		}
	}
	return nil
}

View on GitHub (pinned to 35bff19146)

Solutions

Rename label keys to use only [a-zA-Z0-9_] characters
Replace '.'/'-'/'/' in label keys with underscores
Move non-conforming metadata to label VALUES (values are free-form) rather than keys
Run argo lint locally to catch invalid label keys before submitting

Example fix

# before
labels:
  app.kubernetes.io/name: myapp
# after
labels:
  app_kubernetes_io_name: myapp

Defensive patterns

Strategy: validation

Validate before calling

func validLabelKeys(labels map[string]string) bool {
	for k := range labels {
		if !metrics.IsValidMetricName(k) { return false }
	}
	return true
}

Prevention

Use only [a-zA-Z0-9_] for metric label keys
Never copy Kubernetes annotation/label keys directly into metric labels
Put free-form data in label values, not keys
Run argo lint locally and in CI before submitting

When it happens

Trigger: A workflow step/node template's metrics contain a label key like 'job-id:' or 'pod.name' — validateTemplate calls ValidateMetricLabels and returns this error before submission or reconciliation.

Common situations: Users copying Kubernetes-style label keys (which allow dots, dashes, slashes) into Prometheus metric labels; auto-generated label keys derived from annotation keys containing '/' or '.'; typos like trailing colons.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/3338d49fb4ee013a. Report an issue: GitHub.