argoproj/argo-workflows · error

help for metric %s is already set to %s, it cannot be change

Error message

help for metric %s is already set to %s, it cannot be changed

What it means

When a workflow (or a retry/resubmit of it) re-declares a Prometheus custom metric whose name already exists, matchExistingMetric verifies the new spec is consistent with the already-registered OpenTelemetry instrument. If metricSpec.Help differs from the existing instrument's description, the help text of a registered metric cannot be changed, so the executor fails the metric (and the workflow is marked Error).

Source

Thrown at workflow/metrics/metrics_custom.go:144

// GetCustomMetric returns a custom (or any) metric from it's key
// This is exported for legacy testing only
func (m *Metrics) GetCustomMetric(key string) *telemetry.Instrument {
	// It's okay to return nil metrics in this function
	return m.GetInstrument(key)
}

// CustomMetricExists returns if metric exists from its key
// This is exported for testing only
func (m *Metrics) CustomMetricExists(key string) bool {
	return m.GetCustomMetric(key) != nil
}

// TODO labels on custom metrics
func (m *Metrics) matchExistingMetric(metricSpec *wfv1.Prometheus) (*telemetry.Instrument, error) {
	key := metricSpec.Name
	if inst := m.GetInstrument(key); inst != nil {
		if inst.GetDescription() != metricSpec.Help {
			return nil, fmt.Errorf("help for metric %s is already set to %s, it cannot be changed", metricSpec.Name, inst.GetDescription())
		}
		wantedType := metricSpec.GetMetricType()
		switch inst.GetOtel().(type) {
		case *metric.Float64ObservableGauge:
			if wantedType != wfv1.MetricTypeGauge && !metricSpec.IsRealtime() {
				return nil, fmt.Errorf("found existing gauge for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64ObservableCounter:
			if wantedType != wfv1.MetricTypeCounter {
				return nil, fmt.Errorf("found existing counter for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64Histogram:
			if wantedType != wfv1.MetricTypeHistogram {
				return nil, fmt.Errorf("found existing histogram for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		default:
			return nil, fmt.Errorf("found unwanted type %s for custom metric %s of type %s", reflect.TypeOf(inst.GetOtel()), metricSpec.Name, wantedType)
		}

View on GitHub (pinned to 35bff19146)

Solutions

  1. Give each distinct help text a distinct metric name
  2. Make the help string identical across all declarations of the same metric name (including retries of prior runs)
  3. If the change was intentional, run in a fresh executor pod / workflow name so the instrument is created anew
  4. Check metric help templating (e.g. {{workflow.name}}) isn't producing varying strings

Example fix

// before
- name: duration
  help: How long the step took
- name: duration
  help: Step duration  # mismatch
// after
- name: duration
  help: How long the step took
- name: duration_steps
  help: Step duration
Defensive patterns

Strategy: validation

Validate before calling

// ensure help text is constant per metric name across the spec
helpByName := map[string]string{}
for _, m := range spec.Prometheus {
    if h, seen := helpByName[m.Name]; seen && h != m.Help {
        return fmt.Errorf("metric %s has conflicting help: %q vs %q", m.Name, h, m.Help)
    }
    helpByName[m.Name] = m.Help
}

Try / catch

if err != nil && strings.Contains(err.Error(), "is already set to") {
    // fix spec help/name mismatch and resubmit as a new workflow
}

Prevention

When it happens

Trigger: A workflow spec defines two Prometheus metrics with the same `name` but different `help` strings, or a retried/resubmitted workflow changes the help text of a metric name that was already registered in the executor's meter.

Common situations: Copy-pasting a metric block and editing help but not name; retrying a workflow whose updated spec tweaked the help string; templates parameterized such that help text varies per step while name stays constant.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/cd662f775cecc8a6. Report an issue: GitHub.