argoproj/argo-workflows · error

found unwanted type %s for custom metric %s of type %s

Error message

found unwanted type %s for custom metric %s of type %s

What it means

matchExistingMetric's default branch: an instrument with the metric's name exists in the registry but its underlying OpenTelemetry type is neither Float64ObservableGauge, Float64ObservableCounter, nor Float64Histogram, so the existing-type check cannot be performed and the upsert is rejected. It reports the actual Go type via reflect for diagnosis. In practice this means a base/built-in instrument (or an unexpected wrapper) already owns that name.

Source

Thrown at workflow/metrics/metrics_custom.go:161

		if inst.GetDescription() != metricSpec.Help {
			return nil, fmt.Errorf("help for metric %s is already set to %s, it cannot be changed", metricSpec.Name, inst.GetDescription())
		}
		wantedType := metricSpec.GetMetricType()
		switch inst.GetOtel().(type) {
		case *metric.Float64ObservableGauge:
			if wantedType != wfv1.MetricTypeGauge && !metricSpec.IsRealtime() {
				return nil, fmt.Errorf("found existing gauge for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64ObservableCounter:
			if wantedType != wfv1.MetricTypeCounter {
				return nil, fmt.Errorf("found existing counter for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		case *metric.Float64Histogram:
			if wantedType != wfv1.MetricTypeHistogram {
				return nil, fmt.Errorf("found existing histogram for custom metric %s of type %s", metricSpec.Name, wantedType)
			}
		default:
			return nil, fmt.Errorf("found unwanted type %s for custom metric %s of type %s", reflect.TypeOf(inst.GetOtel()), metricSpec.Name, wantedType)
		}
		return inst, nil
	}
	return nil, nil
}

func (m *Metrics) ensureBaseMetric(metricSpec *wfv1.Prometheus, ownerKey string) (*telemetry.Instrument, error) {
	// Fast path: check if metric already exists and is fully initialized (double-checked locking).
	metric, err := m.matchExistingMetric(metricSpec)
	if err != nil {
		return nil, err
	}
	if metric != nil {
		if customUserData(metric, false) != nil {
			m.attachCustomMetricToWorkflow(metricSpec, ownerKey)
			return metric, nil
		}
	}

View on GitHub (pinned to 35bff19146)

Solutions

  1. Rename the custom metric to a unique name that does not collide with existing instruments
  2. Check the logged Go type in the error to identify what already registered the name and align usage accordingly
  3. Avoid using names of Argo built-in/base metrics for custom workflow metrics
Defensive patterns

Strategy: validation

Validate before calling

// reject custom metric names that shadow Argo built-in metric names
var builtinMetrics = map[string]bool{"operation_count": true, "k8s_request_duration": true}
func isReservedName(name string) bool { return builtinMetrics[name] }

Prevention

When it happens

Trigger: UpsertCustomMetric is called with a metricSpec.Name that collides with a registered instrument of an unexpected OTel kind (e.g. a non-observable/base instrument created elsewhere in the controller) — the switch in matchExistingMetric falls through to default and returns this error immediately.

Common situations: A custom metric name shadowing a built-in Argo metric name (e.g. same name as a system instrument); upgrades where instrument kinds changed; copying an existing name from another metrics scope.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/15b928b346f86a29. Report an issue: GitHub.