{"record":{"id":"bdcd35fe93a48db0","repo":"argoproj/argo-workflows","slug":"found-existing-gauge-for-custom-metric-s-of-type","errorCode":null,"errorMessage":"found existing gauge for custom metric %s of type %s","messagePattern":"found existing gauge for custom metric (.+?) of type (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/metrics/metrics_custom.go","lineNumber":150,"sourceCode":"\n// CustomMetricExists returns if metric exists from its key\n// This is exported for testing only\nfunc (m *Metrics) CustomMetricExists(key string) bool {\n\treturn m.GetCustomMetric(key) != nil\n}\n\n// TODO labels on custom metrics\nfunc (m *Metrics) matchExistingMetric(metricSpec *wfv1.Prometheus) (*telemetry.Instrument, error) {\n\tkey := metricSpec.Name\n\tif inst := m.GetInstrument(key); inst != nil {\n\t\tif inst.GetDescription() != metricSpec.Help {\n\t\t\treturn nil, fmt.Errorf(\"help for metric %s is already set to %s, it cannot be changed\", metricSpec.Name, inst.GetDescription())\n\t\t}\n\t\twantedType := metricSpec.GetMetricType()\n\t\tswitch inst.GetOtel().(type) {\n\t\tcase *metric.Float64ObservableGauge:\n\t\t\tif wantedType != wfv1.MetricTypeGauge && !metricSpec.IsRealtime() {\n\t\t\t\treturn nil, fmt.Errorf(\"found existing gauge for custom metric %s of type %s\", metricSpec.Name, wantedType)\n\t\t\t}\n\t\tcase *metric.Float64ObservableCounter:\n\t\t\tif wantedType != wfv1.MetricTypeCounter {\n\t\t\t\treturn nil, fmt.Errorf(\"found existing counter for custom metric %s of type %s\", metricSpec.Name, wantedType)\n\t\t\t}\n\t\tcase *metric.Float64Histogram:\n\t\t\tif wantedType != wfv1.MetricTypeHistogram {\n\t\t\t\treturn nil, fmt.Errorf(\"found existing histogram for custom metric %s of type %s\", metricSpec.Name, wantedType)\n\t\t\t}\n\t\tdefault:\n\t\t\treturn nil, fmt.Errorf(\"found unwanted type %s for custom metric %s of type %s\", reflect.TypeOf(inst.GetOtel()), metricSpec.Name, wantedType)\n\t\t}\n\t\treturn inst, nil\n\t}\n\treturn nil, nil\n}\n\nfunc (m *Metrics) ensureBaseMetric(metricSpec *wfv1.Prometheus, ownerKey string) (*telemetry.Instrument, error) {","sourceCodeStart":132,"sourceCodeEnd":168,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/metrics/metrics_custom.go#L132-L168","documentation":"A custom Prometheus metric name already exists but is registered as an OpenTelemetry Float64ObservableGauge, while the new spec requests a different metric type (and is not realtime, which is gauge-compatible). OpenTelemetry instruments cannot change type under the same name, so the mismatch is rejected.","triggerScenarios":"Workflow spec declares the same metric name twice with different `kind`/type (e.g. first `Gauge`, later `Histogram`, `Counter`, or `Distribution`), or a retry redeclares a gauge metric as a counter.","commonSituations":"Editing a metric's type between a step and its retry; two templates emitting a same-named metric with different types; changing `realtime: true` off so the gauge exemption no longer applies.","solutions":["Use a unique metric name per type, or align all declarations to the same type","Keep `realtime: true` if you must reuse the gauge name with realtime semantics, or drop realtime and use consistent Gauge specs","Make base-metric upgrades (e.g. changing built-in metric types) explicit by using a new metric name","Validate the spec with `argo lint` before submitting"],"exampleFix":"// before\n- name: duration\n  kind: Gauge\n- name: duration\n  kind: Histogram  # type clash\n// after\n- name: duration\n  kind: Gauge\n- name: duration_hist\n  kind: Histogram\n","handlingStrategy":"validation","validationCode":"kindByName := map[string]wfv1.MetricType{}\nfor _, m := range spec.Prometheus {\n    if t, seen := kindByName[m.Name]; seen && t != m.GetMetricType() && !m.IsRealtime() {\n        return fmt.Errorf(\"metric %s declared as %v and %v\", m.Name, t, m.GetMetricType())\n    }\n    kindByName[m.Name] = m.GetMetricType()\n}","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"found existing gauge\") {\n    // rename one of the metrics or unify kinds, then resubmit\n}","preventionTips":["Keep one metric type per metric name for the whole workflow lifetime","Don't change metric kinds between a step and its retry","Remember realtime metrics are gauges — dropping `realtime: true` can surface this conflict","Lint specs before submitting"],"tags":["metrics","prometheus","otel","workflow"],"backgroundTag":"metric-type-conflict","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}