What causes "failed to create new metric %s" in argoproj/argo-workflows?

Verify the metric name is valid (IsValidMetricName) — invalid names may fail registration paths inconsistently

failed to create new metric %s

Error message

failed to create new metric %s

What it means

After createCustomMetric succeeds, ensureBaseMetric looks the instrument back up via GetInstrument(metricSpec.Name); if the registry still has no instrument under that name, creation silently failed somewhere and this error is returned. It is an internal consistency guard: the metric spec was considered valid, but the instrument was not actually registered.

Source

Thrown at workflow/metrics/metrics_custom.go:203

	metric, err = m.matchExistingMetric(metricSpec)
	if err != nil {
		return nil, err
	}
	if metric != nil {
		if customUserData(metric, false) == nil {
			metric.SetUserdata(newUserData())
		}
		m.attachCustomMetricToWorkflow(metricSpec, ownerKey)
		return metric, nil
	}
	err = m.createCustomMetric(metricSpec)
	if err != nil {
		return nil, err
	}
	m.attachCustomMetricToWorkflow(metricSpec, ownerKey)
	inst := m.GetInstrument(metricSpec.Name)
	if inst == nil {
		return nil, fmt.Errorf("failed to create new metric %s", metricSpec.Name)
	}
	if customUserData(inst, false) == nil {
		inst.SetUserdata(newUserData())
	}
	return inst, nil
}

func (m *Metrics) UpsertCustomMetric(ctx context.Context, metricSpec *wfv1.Prometheus, ownerKey string, valueFunc RealTimeValueFunc) error {
	if !IsValidMetricName(metricSpec.Name) {
		return fmt.Errorf("%s", invalidMetricNameError)
	}
	baseMetric, err := m.ensureBaseMetric(metricSpec, ownerKey)
	if err != nil {
		return err
	}
	metricValue := getOrCreateValue(baseMetric, metricSpec.GetKey(), metricSpec.Labels)
	metricValue.mutex.Lock()
	defer metricValue.mutex.Unlock()

View on GitHub (pinned to 35bff19146)

Solutions

Verify the metric name is valid (IsValidMetricName) — invalid names may fail registration paths inconsistently
Check controller logs for the preceding error from createCustomMetric or the OTel meter
Retry the workflow/controller operation; if reproducible, file an issue with the metric spec and controller version
Ensure controller telemetry (metrics) configuration is enabled and the OTel meter provider is initialized

Defensive patterns

Strategy: retry

Validate before calling

if !metrics.IsValidMetricName(name) { return fmt.Errorf("invalid metric name %q", name) }

Try / catch

err := w.UpsertCustomMetric(ctx, spec, key, valFn)
if err != nil && strings.Contains(err.Error(), "failed to create new metric") {
	// log and retry once after controller re-init; else surface
}

Prevention

Ensure controller metrics/OTel provider is enabled and initialized
Validate metric names before upsert
If reproducible, capture the full metric spec and controller version for a bug report

When it happens

Trigger: ensureBaseMetric (called from UpsertCustomMetric) creates a metric, attaches it to the workflow, then GetInstrument returns nil — i.e. createCustomMetric did not register an instrument under the exact metricSpec.Name (name normalization/case mismatch, or registration path skipped for that spec shape).

Common situations: Unexpected internal states rather than user error; can surface after partial controller initialization, or if a plugin/custom instrumentation path created the metric under a different key than requested.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/1deb259e0321303b. Report an issue: GitHub.