argoproj/argo-workflows · error

found existing histogram for custom metric %s of type %s

Error message

found existing histogram for custom metric %s of type %s

What it means

Argo Workflows reuses a single OpenTelemetry instrument per custom metric name within the controller's Metrics registry. When a metric with the same name already exists as a Float64Histogram but the new metricSpec asks for a different type (gauge or counter), matchExistingMetric refuses to upsert, because OTel instruments cannot change type after creation. This protects the OTel SDK from duplicate-registration errors and keeps Prometheus series consistent.

Source

Thrown at workflow/metrics/metrics_custom.go:158

func (m *Metrics) matchExistingMetric(metricSpec *wfv1.Prometheus) (*telemetry.Instrument, error) {
	key := metricSpec.Name
	if inst := m.GetInstrument(key); inst != nil {
		if inst.GetDescription() != metricSpec.Help {
			return nil, fmt.Errorf("help for metric %s is already set to %s, it cannot be changed", metricSpec.Name, inst.GetDescription())
		}
		wantedType := metricSpec.GetMetricType()
		switch inst.GetOtel().(type) {
		case *metric.Float64ObservableGauge:
			if wantedType != wfv1.MetricTypeGauge && !metricSpec.IsRealtime() {
				return nil, fmt.Errorf("found existing gauge for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64ObservableCounter:
			if wantedType != wfv1.MetricTypeCounter {
				return nil, fmt.Errorf("found existing counter for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64Histogram:
			if wantedType != wfv1.MetricTypeHistogram {
				return nil, fmt.Errorf("found existing histogram for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		default:
			return nil, fmt.Errorf("found unwanted type %s for custom metric %s of type %s", reflect.TypeOf(inst.GetOtel()), metricSpec.Name, wantedType)
		}
		return inst, nil
	}
	return nil, nil
}

func (m *Metrics) ensureBaseMetric(metricSpec *wfv1.Prometheus, ownerKey string) (*telemetry.Instrument, error) {
	// Fast path: check if metric already exists and is fully initialized (double-checked locking).
	metric, err := m.matchExistingMetric(metricSpec)
	if err != nil {
		return nil, err
	}
	if metric != nil {
		if customUserData(metric, false) != nil {
			m.attachCustomMetricToWorkflow(metricSpec, ownerKey)

View on GitHub (pinned to 35bff19146)

Solutions

  1. Rename one of the metrics so each name has exactly one type across all workflow templates
  2. Change the new metricSpec to declare the same type as the existing histogram (use histogram: instead of gauge:/counter:)
  3. Wait for/terminate the old workflow so the histogram instrument is no longer live, then resubmit with the new type
  4. If running multiple controllers, scope metric names per controller to avoid cross-controller collisions

Example fix

// before
metrics:
  - name: job_duration
    histogram:
      value: "1"
// after (renamed to avoid type collision with existing counter of same name)
metrics:
  - name: job_duration_hist
    histogram:
      value: "1"
Defensive patterns

Strategy: validation

Validate before calling

// before submitting a workflow, ensure no running/sibling workflow declares the same metric name with a different type
func metricTypesConsistent(specs []wfv1.Prometheus) error {
	seen := map[string]wfv1.MetricType{}
	for _, m := range specs {
		t := m.GetMetricType()
		if prev, ok := seen[m.Name]; ok && prev != t {
			return fmt.Errorf("metric %s declared as %v and %v", m.Name, prev, t)
		}
		seen[m.Name] = t
	}
	return nil
}

Prevention

When it happens

Trigger: Calling UpsertCustomMetric (via controller UpsertCustomMetric) with a metricSpec whose Name collides with an already-registered histogram instrument while metricSpec.GetMetricType() is not MetricTypeHistogram — e.g. two workflow templates emit a custom metric with the same name but one declares it as Histogram and the other as Gauge/Counter, or a template was edited to change the metric type without renaming it.

Common situations: Two teams copying a template and changing the metric type but keeping the name; a user switching a metric from Histogram to Counter in an edited workflow while the old one is still running in the same controller process; name collisions across workflows relying on a shared controller-level registry.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/b1f796ab74def8b6. Report an issue: GitHub.