argoproj/argo-workflows · error

found existing gauge for custom metric %s of type %s

Error message

found existing gauge for custom metric %s of type %s

What it means

A custom Prometheus metric name already exists but is registered as an OpenTelemetry Float64ObservableGauge, while the new spec requests a different metric type (and is not realtime, which is gauge-compatible). OpenTelemetry instruments cannot change type under the same name, so the mismatch is rejected.

Source

Thrown at workflow/metrics/metrics_custom.go:150

// CustomMetricExists returns if metric exists from its key
// This is exported for testing only
func (m *Metrics) CustomMetricExists(key string) bool {
	return m.GetCustomMetric(key) != nil
}

// TODO labels on custom metrics
func (m *Metrics) matchExistingMetric(metricSpec *wfv1.Prometheus) (*telemetry.Instrument, error) {
	key := metricSpec.Name
	if inst := m.GetInstrument(key); inst != nil {
		if inst.GetDescription() != metricSpec.Help {
			return nil, fmt.Errorf("help for metric %s is already set to %s, it cannot be changed", metricSpec.Name, inst.GetDescription())
		}
		wantedType := metricSpec.GetMetricType()
		switch inst.GetOtel().(type) {
		case *metric.Float64ObservableGauge:
			if wantedType != wfv1.MetricTypeGauge && !metricSpec.IsRealtime() {
				return nil, fmt.Errorf("found existing gauge for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64ObservableCounter:
			if wantedType != wfv1.MetricTypeCounter {
				return nil, fmt.Errorf("found existing counter for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64Histogram:
			if wantedType != wfv1.MetricTypeHistogram {
				return nil, fmt.Errorf("found existing histogram for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		default:
			return nil, fmt.Errorf("found unwanted type %s for custom metric %s of type %s", reflect.TypeOf(inst.GetOtel()), metricSpec.Name, wantedType)
		}
		return inst, nil
	}
	return nil, nil
}

func (m *Metrics) ensureBaseMetric(metricSpec *wfv1.Prometheus, ownerKey string) (*telemetry.Instrument, error) {

View on GitHub (pinned to 35bff19146)

Solutions

  1. Use a unique metric name per type, or align all declarations to the same type
  2. Keep `realtime: true` if you must reuse the gauge name with realtime semantics, or drop realtime and use consistent Gauge specs
  3. Make base-metric upgrades (e.g. changing built-in metric types) explicit by using a new metric name
  4. Validate the spec with `argo lint` before submitting

Example fix

// before
- name: duration
  kind: Gauge
- name: duration
  kind: Histogram  # type clash
// after
- name: duration
  kind: Gauge
- name: duration_hist
  kind: Histogram
Defensive patterns

Strategy: validation

Validate before calling

kindByName := map[string]wfv1.MetricType{}
for _, m := range spec.Prometheus {
    if t, seen := kindByName[m.Name]; seen && t != m.GetMetricType() && !m.IsRealtime() {
        return fmt.Errorf("metric %s declared as %v and %v", m.Name, t, m.GetMetricType())
    }
    kindByName[m.Name] = m.GetMetricType()
}

Try / catch

if err != nil && strings.Contains(err.Error(), "found existing gauge") {
    // rename one of the metrics or unify kinds, then resubmit
}

Prevention

When it happens

Trigger: Workflow spec declares the same metric name twice with different `kind`/type (e.g. first `Gauge`, later `Histogram`, `Counter`, or `Distribution`), or a retry redeclares a gauge metric as a counter.

Common situations: Editing a metric's type between a step and its retry; two templates emitting a same-named metric with different types; changing `realtime: true` off so the gauge exemption no longer applies.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/bdcd35fe93a48db0. Report an issue: GitHub.