{"record":{"id":"7e2a3cf02d7a460f","repo":"argoproj/argo-workflows","slug":"found-existing-counter-for-custom-metric-s-of-typ","errorCode":null,"errorMessage":"found existing counter for custom metric %s of type %s","messagePattern":"found existing counter for custom metric (.+?) of type (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/metrics/metrics_custom.go","lineNumber":154,"sourceCode":"\treturn m.GetCustomMetric(key) != nil\n}\n\n// TODO labels on custom metrics\nfunc (m *Metrics) matchExistingMetric(metricSpec *wfv1.Prometheus) (*telemetry.Instrument, error) {\n\tkey := metricSpec.Name\n\tif inst := m.GetInstrument(key); inst != nil {\n\t\tif inst.GetDescription() != metricSpec.Help {\n\t\t\treturn nil, fmt.Errorf(\"help for metric %s is already set to %s, it cannot be changed\", metricSpec.Name, inst.GetDescription())\n\t\t}\n\t\twantedType := metricSpec.GetMetricType()\n\t\tswitch inst.GetOtel().(type) {\n\t\tcase *metric.Float64ObservableGauge:\n\t\t\tif wantedType != wfv1.MetricTypeGauge && !metricSpec.IsRealtime() {\n\t\t\t\treturn nil, fmt.Errorf(\"found existing gauge for custom metric %s of type %s\", metricSpec.Name, wantedType)\n\t\t\t}\n\t\tcase *metric.Float64ObservableCounter:\n\t\t\tif wantedType != wfv1.MetricTypeCounter {\n\t\t\t\treturn nil, fmt.Errorf(\"found existing counter for custom metric %s of type %s\", metricSpec.Name, wantedType)\n\t\t\t}\n\t\tcase *metric.Float64Histogram:\n\t\t\tif wantedType != wfv1.MetricTypeHistogram {\n\t\t\t\treturn nil, fmt.Errorf(\"found existing histogram for custom metric %s of type %s\", metricSpec.Name, wantedType)\n\t\t\t}\n\t\tdefault:\n\t\t\treturn nil, fmt.Errorf(\"found unwanted type %s for custom metric %s of type %s\", reflect.TypeOf(inst.GetOtel()), metricSpec.Name, wantedType)\n\t\t}\n\t\treturn inst, nil\n\t}\n\treturn nil, nil\n}\n\nfunc (m *Metrics) ensureBaseMetric(metricSpec *wfv1.Prometheus, ownerKey string) (*telemetry.Instrument, error) {\n\t// Fast path: check if metric already exists and is fully initialized (double-checked locking).\n\tmetric, err := m.matchExistingMetric(metricSpec)\n\tif err != nil {\n\t\treturn nil, err","sourceCodeStart":136,"sourceCodeEnd":172,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/metrics/metrics_custom.go#L136-L172","documentation":"A custom Prometheus metric name already exists as a Float64ObservableCounter and the new spec requests a different metric type. Since an OpenTelemetry counter instrument is immutable in type for a given name, the conflicting declaration is rejected and the metric fails.","triggerScenarios":"Workflow spec declares the same metric name as `Counter` in one place and `Gauge`, `Histogram`/`Distribution` in another (counter case triggers when wantedType != MetricTypeCounter); typically across steps, retries, or a changed template.","commonSituations":"Retrying a workflow whose spec changed a metric from Counter to Histogram; copy-pasted metric blocks with edited kind but the same name; parameterized help/name collisions between parallel steps.","solutions":["Ensure every declaration sharing a metric name uses kind: Counter, or rename the differently-typed metric","Align the spec across retries — don't change metric kinds for an already-emitted metric name","Run `argo lint` to catch duplicate-name/different-type conflicts before submit","If migrating metric types intentionally, adopt a new metric name (e.g. append a version suffix) rather than reusing the old one"],"exampleFix":"// before\n- name: hits\n  kind: Counter\n- name: hits\n  kind: Distribution  # counter clash\n// after\n- name: hits\n  kind: Counter\n- name: hits_hist\n  kind: Distribution\n","handlingStrategy":"validation","validationCode":"kindByName := map[string]wfv1.MetricType{}\nfor _, m := range spec.Prometheus {\n    if t, seen := kindByName[m.Name]; seen && t != m.GetMetricType() {\n        return fmt.Errorf(\"metric %s declared as %v and %v\", m.Name, t, m.GetMetricType())\n    }\n    kindByName[m.Name] = m.GetMetricType()\n}","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"found existing counter\") {\n    // use a distinct metric name for the differently-typed metric\n}","preventionTips":["Align metric kind (Counter) everywhere the same metric name appears","On intentional type migration, adopt a new metric name instead of reusing the old one","Parameterized templates must not emit same-named metrics with different kinds","Run `argo lint` before submit"],"tags":["metrics","prometheus","otel","workflow"],"backgroundTag":"metric-type-conflict","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}