{"record":{"id":"b1f796ab74def8b6","repo":"argoproj/argo-workflows","slug":"found-existing-histogram-for-custom-metric-s-of-t","errorCode":null,"errorMessage":"found existing histogram for custom metric %s of type %s","messagePattern":"found existing histogram for custom metric (.+?) of type (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/metrics/metrics_custom.go","lineNumber":158,"sourceCode":"func (m *Metrics) matchExistingMetric(metricSpec *wfv1.Prometheus) (*telemetry.Instrument, error) {\n\tkey := metricSpec.Name\n\tif inst := m.GetInstrument(key); inst != nil {\n\t\tif inst.GetDescription() != metricSpec.Help {\n\t\t\treturn nil, fmt.Errorf(\"help for metric %s is already set to %s, it cannot be changed\", metricSpec.Name, inst.GetDescription())\n\t\t}\n\t\twantedType := metricSpec.GetMetricType()\n\t\tswitch inst.GetOtel().(type) {\n\t\tcase *metric.Float64ObservableGauge:\n\t\t\tif wantedType != wfv1.MetricTypeGauge && !metricSpec.IsRealtime() {\n\t\t\t\treturn nil, fmt.Errorf(\"found existing gauge for custom metric %s of type %s\", metricSpec.Name, wantedType)\n\t\t\t}\n\t\tcase *metric.Float64ObservableCounter:\n\t\t\tif wantedType != wfv1.MetricTypeCounter {\n\t\t\t\treturn nil, fmt.Errorf(\"found existing counter for custom metric %s of type %s\", metricSpec.Name, wantedType)\n\t\t\t}\n\t\tcase *metric.Float64Histogram:\n\t\t\tif wantedType != wfv1.MetricTypeHistogram {\n\t\t\t\treturn nil, fmt.Errorf(\"found existing histogram for custom metric %s of type %s\", metricSpec.Name, wantedType)\n\t\t\t}\n\t\tdefault:\n\t\t\treturn nil, fmt.Errorf(\"found unwanted type %s for custom metric %s of type %s\", reflect.TypeOf(inst.GetOtel()), metricSpec.Name, wantedType)\n\t\t}\n\t\treturn inst, nil\n\t}\n\treturn nil, nil\n}\n\nfunc (m *Metrics) ensureBaseMetric(metricSpec *wfv1.Prometheus, ownerKey string) (*telemetry.Instrument, error) {\n\t// Fast path: check if metric already exists and is fully initialized (double-checked locking).\n\tmetric, err := m.matchExistingMetric(metricSpec)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tif metric != nil {\n\t\tif customUserData(metric, false) != nil {\n\t\t\tm.attachCustomMetricToWorkflow(metricSpec, ownerKey)","sourceCodeStart":140,"sourceCodeEnd":176,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/metrics/metrics_custom.go#L140-L176","documentation":"Argo Workflows reuses a single OpenTelemetry instrument per custom metric name within the controller's Metrics registry. When a metric with the same name already exists as a Float64Histogram but the new metricSpec asks for a different type (gauge or counter), matchExistingMetric refuses to upsert, because OTel instruments cannot change type after creation. This protects the OTel SDK from duplicate-registration errors and keeps Prometheus series consistent.","triggerScenarios":"Calling UpsertCustomMetric (via controller UpsertCustomMetric) with a metricSpec whose Name collides with an already-registered histogram instrument while metricSpec.GetMetricType() is not MetricTypeHistogram — e.g. two workflow templates emit a custom metric with the same name but one declares it as Histogram and the other as Gauge/Counter, or a template was edited to change the metric type without renaming it.","commonSituations":"Two teams copying a template and changing the metric type but keeping the name; a user switching a metric from Histogram to Counter in an edited workflow while the old one is still running in the same controller process; name collisions across workflows relying on a shared controller-level registry.","solutions":["Rename one of the metrics so each name has exactly one type across all workflow templates","Change the new metricSpec to declare the same type as the existing histogram (use histogram: instead of gauge:/counter:)","Wait for/terminate the old workflow so the histogram instrument is no longer live, then resubmit with the new type","If running multiple controllers, scope metric names per controller to avoid cross-controller collisions"],"exampleFix":"// before\nmetrics:\n  - name: job_duration\n    histogram:\n      value: \"1\"\n// after (renamed to avoid type collision with existing counter of same name)\nmetrics:\n  - name: job_duration_hist\n    histogram:\n      value: \"1\"","handlingStrategy":"validation","validationCode":"// before submitting a workflow, ensure no running/sibling workflow declares the same metric name with a different type\nfunc metricTypesConsistent(specs []wfv1.Prometheus) error {\n\tseen := map[string]wfv1.MetricType{}\n\tfor _, m := range specs {\n\t\tt := m.GetMetricType()\n\t\tif prev, ok := seen[m.Name]; ok && prev != t {\n\t\t\treturn fmt.Errorf(\"metric %s declared as %v and %v\", m.Name, prev, t)\n\t\t}\n\t\tseen[m.Name] = t\n\t}\n\treturn nil\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use one consistent type per metric name across all workflow templates","Keep a registry/naming convention for shared custom metrics","Run argo lint on all templates in CI to catch type drift","Prefix metric names per team/workflow to avoid collisions"],"tags":["metrics","otel","workflow-controller","type-conflict"],"backgroundTag":"metric-type-conflict","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}