argoproj/argo-workflows · error

found existing counter for custom metric %s of type %s

Error message

found existing counter for custom metric %s of type %s

What it means

A custom Prometheus metric name already exists as a Float64ObservableCounter and the new spec requests a different metric type. Since an OpenTelemetry counter instrument is immutable in type for a given name, the conflicting declaration is rejected and the metric fails.

Source

Thrown at workflow/metrics/metrics_custom.go:154

	return m.GetCustomMetric(key) != nil
}

// TODO labels on custom metrics
func (m *Metrics) matchExistingMetric(metricSpec *wfv1.Prometheus) (*telemetry.Instrument, error) {
	key := metricSpec.Name
	if inst := m.GetInstrument(key); inst != nil {
		if inst.GetDescription() != metricSpec.Help {
			return nil, fmt.Errorf("help for metric %s is already set to %s, it cannot be changed", metricSpec.Name, inst.GetDescription())
		}
		wantedType := metricSpec.GetMetricType()
		switch inst.GetOtel().(type) {
		case *metric.Float64ObservableGauge:
			if wantedType != wfv1.MetricTypeGauge && !metricSpec.IsRealtime() {
				return nil, fmt.Errorf("found existing gauge for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64ObservableCounter:
			if wantedType != wfv1.MetricTypeCounter {
				return nil, fmt.Errorf("found existing counter for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64Histogram:
			if wantedType != wfv1.MetricTypeHistogram {
				return nil, fmt.Errorf("found existing histogram for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		default:
			return nil, fmt.Errorf("found unwanted type %s for custom metric %s of type %s", reflect.TypeOf(inst.GetOtel()), metricSpec.Name, wantedType)
		}
		return inst, nil
	}
	return nil, nil
}

func (m *Metrics) ensureBaseMetric(metricSpec *wfv1.Prometheus, ownerKey string) (*telemetry.Instrument, error) {
	// Fast path: check if metric already exists and is fully initialized (double-checked locking).
	metric, err := m.matchExistingMetric(metricSpec)
	if err != nil {
		return nil, err

View on GitHub (pinned to 35bff19146)

Solutions

  1. Ensure every declaration sharing a metric name uses kind: Counter, or rename the differently-typed metric
  2. Align the spec across retries — don't change metric kinds for an already-emitted metric name
  3. Run `argo lint` to catch duplicate-name/different-type conflicts before submit
  4. If migrating metric types intentionally, adopt a new metric name (e.g. append a version suffix) rather than reusing the old one

Example fix

// before
- name: hits
  kind: Counter
- name: hits
  kind: Distribution  # counter clash
// after
- name: hits
  kind: Counter
- name: hits_hist
  kind: Distribution
Defensive patterns

Strategy: validation

Validate before calling

kindByName := map[string]wfv1.MetricType{}
for _, m := range spec.Prometheus {
    if t, seen := kindByName[m.Name]; seen && t != m.GetMetricType() {
        return fmt.Errorf("metric %s declared as %v and %v", m.Name, t, m.GetMetricType())
    }
    kindByName[m.Name] = m.GetMetricType()
}

Try / catch

if err != nil && strings.Contains(err.Error(), "found existing counter") {
    // use a distinct metric name for the differently-typed metric
}

Prevention

When it happens

Trigger: Workflow spec declares the same metric name as `Counter` in one place and `Gauge`, `Histogram`/`Distribution` in another (counter case triggers when wantedType != MetricTypeCounter); typically across steps, retries, or a changed template.

Common situations: Retrying a workflow whose spec changed a metric from Counter to Histogram; copy-pasted metric blocks with edited kind but the same name; parameterized help/name collisions between parallel steps.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/7e2a3cf02d7a460f. Report an issue: GitHub.