{"record":{"id":"4333515378679d5b","repo":"apache/pulsar","slug":"label-name-s-is-invalid-prometheus-reserves-al","errorCode":null,"errorMessage":"Label name '%s' is invalid: Prometheus reserves all labels starting with '__' for internal use","messagePattern":"Label name '(.+?)' is invalid: Prometheus reserves all labels starting with '__' for internal use","errorType":"http","errorClass":"org.apache.pulsar.broker.admin.RestException","httpStatus":400,"severity":"error","filePath":"pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java","lineNumber":2557,"sourceCode":"        if (allowedCustomMetricLabelKeys == null) {\n            return;\n        }\n        boolean exposeCustomTopicMetricLabelsEnabled = config.isExposeCustomTopicMetricLabelsEnabled();\n        if (exposeCustomTopicMetricLabelsEnabled) {\n            for (String labelKey : allowedCustomMetricLabelKeys) {\n                isValidMetricsName(labelKey);\n            }\n        }\n    }\n\n    private static void isValidMetricsName(String labelName) {\n        if (labelName == null || labelName.isEmpty()) {\n            throw new RestException(Response.Status.BAD_REQUEST, \"Label name cannot be null or empty\");\n        }\n\n        // Prometheus reserves all labels starting with \"__\" for internal use.\n        if (labelName.startsWith(\"__\")) {\n            throw new RestException(Response.Status.BAD_REQUEST,\n                    String.format(\"Label name '%s' is invalid: Prometheus reserves all labels starting with '__' \"\n                            + \"for internal use\", labelName));\n        }\n\n        // Pulsar reserves all labels starting with \"pulsar_\" or \"pulsar.\" for internal use.\n        if (labelName.endsWith(\"pulsar.\") || labelName.endsWith(\"pulsar_\")) {\n            throw new RestException(Response.Status.BAD_REQUEST,\n                    String.format(\"Label name '%s' is invalid: Pulsar reserves all labels starting with 'pulsar_' \"\n                            + \"or 'pulsar.' for internal use\", labelName));\n        }\n\n        // OpenTelemetry reserves all labels starting with \"otel_\" or \"otel.\" for internal use.\n        if (labelName.startsWith(\"otel.\") || labelName.startsWith(\"otel_\")) {\n            throw new RestException(Response.Status.BAD_REQUEST,\n                    String.format(\"Label name '%s' is invalid: OpenTelemetry reserves all labels starting with \"\n                            + \"'otel_' or 'otel.' for internal use\", labelName));\n        }\n","sourceCodeStart":2539,"sourceCodeEnd":2575,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java#L2539-L2575","documentation":"Prometheus reserves label names beginning with double underscore (__) for its internal use, so Pulsar refuses to register user metrics labels with that prefix, returning HTTP 400. This prevents user labels from shadowing or colliding with internal Prometheus-generated labels.","triggerScenarios":"Adding a metrics label whose name starts with \"__\" (e.g. \"__env\", \"__my_label\") through the broker metrics API validated by PulsarService.isValidMetricsName.","commonSituations":"Copying Prometheus internal label conventions (like __name__) into custom Pulsar labels; tooling that programmatically prefixes labels with __; importing dashboards/rules that assume reserved-style names.","solutions":["Rename the label so it does not start with \"__\" (e.g. \"env\" or \"my_env\").","If the label must be distinguishable, use a custom non-reserved prefix such as \"custom_\".","Update automation/tooling that generates label names to strip or reject the \"__\" prefix before submitting."],"exampleFix":"// before\nmetrics.registerLabel(\"__environment\");\n\n// after\nmetrics.registerLabel(\"environment\");","handlingStrategy":"validation","validationCode":"if (labelName.startsWith(\"__\")) {\n    throw new IllegalArgumentException(\"label name uses reserved Prometheus '__' prefix: \" + labelName);\n}","typeGuard":"boolean isPrometheusSafe(String name) {\n    return name != null && !name.startsWith(\"__\") && name.matches(\"[a-zA-Z_][a-zA-Z0-9_]*\");\n}","tryCatchPattern":"try {\n    metricsApi.registerLabel(labelName);\n} catch (RestException e) {\n    if (e.getResponse().getStatus() == 400 && labelName.startsWith(\"__\")) {\n        labelName = labelName.replaceFirst(\"^__+\", \"\");\n    }\n}","preventionTips":["Never prefix custom labels with __ (reserved by Prometheus)","Strip reserved prefixes in label-generation code","Review imported dashboards/configs for reserved label names"],"tags":["metrics","prometheus","validation","reserved-prefix"],"backgroundTag":"metrics-label-name-invalid","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}