metric name is invalid: names may only contain alphanumeric

Error message

metric name is invalid: names may only contain alphanumeric characters or '_'

What it means

UpsertCustomMetric validates every custom metric name with IsValidMetricName, which applies Prometheus legacy name rules (alphanumerics and '_', no leading digits issues, and ':' explicitly disallowed). Names with hyphens, dots, spaces, or other characters are rejected before any instrument is created.

Source

Thrown at workflow/metrics/metrics_custom.go:213

	}
	err = m.createCustomMetric(metricSpec)
	if err != nil {
		return nil, err
	}
	m.attachCustomMetricToWorkflow(metricSpec, ownerKey)
	inst := m.GetInstrument(metricSpec.Name)
	if inst == nil {
		return nil, fmt.Errorf("failed to create new metric %s", metricSpec.Name)
	}
	if customUserData(inst, false) == nil {
		inst.SetUserdata(newUserData())
	}
	return inst, nil
}

func (m *Metrics) UpsertCustomMetric(ctx context.Context, metricSpec *wfv1.Prometheus, ownerKey string, valueFunc RealTimeValueFunc) error {
	if !IsValidMetricName(metricSpec.Name) {
		return fmt.Errorf("%s", invalidMetricNameError)
	}
	baseMetric, err := m.ensureBaseMetric(metricSpec, ownerKey)
	if err != nil {
		return err
	}
	metricValue := getOrCreateValue(baseMetric, metricSpec.GetKey(), metricSpec.Labels)
	metricValue.mutex.Lock()
	defer metricValue.mutex.Unlock()
	metricValue.lastUpdated = time.Now()

	metricType := metricSpec.GetMetricType()
	switch {
	case metricSpec.IsRealtime():
		metricValue.rtValueFunc = valueFunc

	case metricType == wfv1.MetricTypeGauge:
		val, err := strconv.ParseFloat(metricSpec.Gauge.Value, 64)
		if err != nil {

View on GitHub (pinned to 35bff19146)

Solutions

Rename the metric to contain only [a-zA-Z0-9_] characters (Prometheus legacy rules)
Replace hyphens/dots with underscores, e.g. my-metric -> my_metric
Remove any ':' characters from the name
Run workflow validation (argo lint) before submitting to catch the name early

Example fix

// before
metrics:
  - name: my-metric.duration
    gauge: {value: "1"}
// after
metrics:
  - name: my_metric_duration
    gauge: {value: "1"}

Defensive patterns

Strategy: validation

Validate before calling

// reuse the library's own check before submitting
if !metrics.IsValidMetricName(name) {
	return fmt.Errorf("metric name %q must match [a-zA-Z0-9_] and contain no ':'", name)
}

Prevention

Run argo lint in CI on every workflow/template
Use only underscores as separators in metric names
Never copy names with hyphens/dots/colons from other systems
Validate generated names (e.g. derived from labels) programmatically before submit

When it happens

Trigger: Calling UpsertCustomMetric (or submitting a workflow whose metrics section contains a name failing model.LegacyValidation.IsValidMetricName or containing ':') — e.g. name: my-metric or my.metric.

Common situations: Users porting names from other monitoring systems that allow hyphens/dots; auto-generated names derived from pod/step labels containing dashes; typos like colons copied from PromQL recording-rule names.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/86497276f9d3d88e. Report an issue: GitHub.