{"record":{"id":"1deb259e0321303b","repo":"argoproj/argo-workflows","slug":"failed-to-create-new-metric-s","errorCode":null,"errorMessage":"failed to create new metric %s","messagePattern":"failed to create new metric (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/metrics/metrics_custom.go","lineNumber":203,"sourceCode":"\tmetric, err = m.matchExistingMetric(metricSpec)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tif metric != nil {\n\t\tif customUserData(metric, false) == nil {\n\t\t\tmetric.SetUserdata(newUserData())\n\t\t}\n\t\tm.attachCustomMetricToWorkflow(metricSpec, ownerKey)\n\t\treturn metric, nil\n\t}\n\terr = m.createCustomMetric(metricSpec)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tm.attachCustomMetricToWorkflow(metricSpec, ownerKey)\n\tinst := m.GetInstrument(metricSpec.Name)\n\tif inst == nil {\n\t\treturn nil, fmt.Errorf(\"failed to create new metric %s\", metricSpec.Name)\n\t}\n\tif customUserData(inst, false) == nil {\n\t\tinst.SetUserdata(newUserData())\n\t}\n\treturn inst, nil\n}\n\nfunc (m *Metrics) UpsertCustomMetric(ctx context.Context, metricSpec *wfv1.Prometheus, ownerKey string, valueFunc RealTimeValueFunc) error {\n\tif !IsValidMetricName(metricSpec.Name) {\n\t\treturn fmt.Errorf(\"%s\", invalidMetricNameError)\n\t}\n\tbaseMetric, err := m.ensureBaseMetric(metricSpec, ownerKey)\n\tif err != nil {\n\t\treturn err\n\t}\n\tmetricValue := getOrCreateValue(baseMetric, metricSpec.GetKey(), metricSpec.Labels)\n\tmetricValue.mutex.Lock()\n\tdefer metricValue.mutex.Unlock()","sourceCodeStart":185,"sourceCodeEnd":221,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/metrics/metrics_custom.go#L185-L221","documentation":"After createCustomMetric succeeds, ensureBaseMetric looks the instrument back up via GetInstrument(metricSpec.Name); if the registry still has no instrument under that name, creation silently failed somewhere and this error is returned. It is an internal consistency guard: the metric spec was considered valid, but the instrument was not actually registered.","triggerScenarios":"ensureBaseMetric (called from UpsertCustomMetric) creates a metric, attaches it to the workflow, then GetInstrument returns nil — i.e. createCustomMetric did not register an instrument under the exact metricSpec.Name (name normalization/case mismatch, or registration path skipped for that spec shape).","commonSituations":"Unexpected internal states rather than user error; can surface after partial controller initialization, or if a plugin/custom instrumentation path created the metric under a different key than requested.","solutions":["Verify the metric name is valid (IsValidMetricName) — invalid names may fail registration paths inconsistently","Check controller logs for the preceding error from createCustomMetric or the OTel meter","Retry the workflow/controller operation; if reproducible, file an issue with the metric spec and controller version","Ensure controller telemetry (metrics) configuration is enabled and the OTel meter provider is initialized"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if !metrics.IsValidMetricName(name) { return fmt.Errorf(\"invalid metric name %q\", name) }","typeGuard":null,"tryCatchPattern":"err := w.UpsertCustomMetric(ctx, spec, key, valFn)\nif err != nil && strings.Contains(err.Error(), \"failed to create new metric\") {\n\t// log and retry once after controller re-init; else surface\n}","preventionTips":["Ensure controller metrics/OTel provider is enabled and initialized","Validate metric names before upsert","If reproducible, capture the full metric spec and controller version for a bug report"],"tags":["metrics","otel","internal-error","workflow-controller"],"backgroundTag":"metric-registration-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}