apache/pulsar · error · org.apache.pulsar.broker.admin.RestException

Label name '%s' is invalid: Prometheus reserves all labels s

Error message

Label name '%s' is invalid: Prometheus reserves all labels starting with '__' for internal use

What it means

Prometheus reserves label names beginning with double underscore (__) for its internal use, so Pulsar refuses to register user metrics labels with that prefix, returning HTTP 400. This prevents user labels from shadowing or colliding with internal Prometheus-generated labels.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java:2557

        if (allowedCustomMetricLabelKeys == null) {
            return;
        }
        boolean exposeCustomTopicMetricLabelsEnabled = config.isExposeCustomTopicMetricLabelsEnabled();
        if (exposeCustomTopicMetricLabelsEnabled) {
            for (String labelKey : allowedCustomMetricLabelKeys) {
                isValidMetricsName(labelKey);
            }
        }
    }

    private static void isValidMetricsName(String labelName) {
        if (labelName == null || labelName.isEmpty()) {
            throw new RestException(Response.Status.BAD_REQUEST, "Label name cannot be null or empty");
        }

        // Prometheus reserves all labels starting with "__" for internal use.
        if (labelName.startsWith("__")) {
            throw new RestException(Response.Status.BAD_REQUEST,
                    String.format("Label name '%s' is invalid: Prometheus reserves all labels starting with '__' "
                            + "for internal use", labelName));
        }

        // Pulsar reserves all labels starting with "pulsar_" or "pulsar." for internal use.
        if (labelName.endsWith("pulsar.") || labelName.endsWith("pulsar_")) {
            throw new RestException(Response.Status.BAD_REQUEST,
                    String.format("Label name '%s' is invalid: Pulsar reserves all labels starting with 'pulsar_' "
                            + "or 'pulsar.' for internal use", labelName));
        }

        // OpenTelemetry reserves all labels starting with "otel_" or "otel." for internal use.
        if (labelName.startsWith("otel.") || labelName.startsWith("otel_")) {
            throw new RestException(Response.Status.BAD_REQUEST,
                    String.format("Label name '%s' is invalid: OpenTelemetry reserves all labels starting with "
                            + "'otel_' or 'otel.' for internal use", labelName));
        }

View on GitHub (pinned to 820761864e)

Solutions

1. Rename the label so it does not start with "__" (e.g. "env" or "my_env").
2. If the label must be distinguishable, use a custom non-reserved prefix such as "custom_".
3. Update automation/tooling that generates label names to strip or reject the "__" prefix before submitting.

Example fix

// before
metrics.registerLabel("__environment");

// after
metrics.registerLabel("environment");

if (labelName.startsWith("__")) {
    throw new IllegalArgumentException("label name uses reserved Prometheus '__' prefix: " + labelName);
}

boolean isPrometheusSafe(String name) {
    return name != null && !name.startsWith("__") && name.matches("[a-zA-Z_][a-zA-Z0-9_]*");
}

try {
    metricsApi.registerLabel(labelName);
} catch (RestException e) {
    if (e.getResponse().getStatus() == 400 && labelName.startsWith("__")) {
        labelName = labelName.replaceFirst("^__+", "");
    }
}

When it happens

Trigger: Adding a metrics label whose name starts with "__" (e.g. "__env", "__my_label") through the broker metrics API validated by PulsarService.isValidMetricsName.

Common situations: Copying Prometheus internal label conventions (like __name__) into custom Pulsar labels; tooling that programmatically prefixes labels with __; importing dashboards/rules that assume reserved-style names.

Related errors

• Label name '%s' is invalid: Pulsar reserves all labels start
• Label name cannot be null or empty
• Label name '%s' is invalid: OpenTelemetry reserves all label
• Label name '%s' is invalid: must match the regex [a-zA-Z_][a
• Timeout during delete operation

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/4333515378679d5b. Report an issue: GitHub.