{"record":{"id":"05bd8d3ca0ddd173","repo":"BerriAI/litellm","slug":"prometheus-exclude-configuration-validation-failed","errorCode":null,"errorMessage":"Prometheus exclude configuration validation failed:\n{joined_errors}","messagePattern":"Prometheus exclude configuration validation failed:\n(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/integrations/prometheus.py","lineNumber":738,"sourceCode":"        from typing import get_args\n\n        import litellm\n\n        exclude_metrics: Final = frozenset(litellm.prometheus_exclude_metrics or ())\n        exclude_labels: Final = frozenset(litellm.prometheus_exclude_labels or ())\n\n        valid_metrics: Final = frozenset(get_args(DEFINED_PROMETHEUS_METRICS))\n        invalid_metrics: Final = sorted(exclude_metrics - valid_metrics)\n\n        valid_labels: Final = self._all_defined_labels()\n        invalid_labels: Final = sorted(exclude_labels - valid_labels)\n\n        errors: Final = (\n            *(f\"Invalid metric name in prometheus_exclude_metrics: {metric}\" for metric in invalid_metrics),\n            *(f\"Invalid label name in prometheus_exclude_labels: {label}\" for label in invalid_labels),\n        )\n        if errors:\n            raise ValueError(\"Prometheus exclude configuration validation failed:\\n\" + \"\\n\".join(errors))\n\n        return exclude_metrics, exclude_labels\n\n    @staticmethod\n    def _all_defined_labels() -> frozenset[str]:\n        \"\"\"Every label a metric can emit: enum labels, hard-coded labels, and configured custom labels / tags.\"\"\"\n        import litellm\n\n        builtin_labels: Final = frozenset(label.value for label in UserAPIKeyLabelNames)\n        custom_metadata_labels: Final = frozenset(\n            _sanitize_prometheus_label_name(label) for label in litellm.custom_prometheus_metadata_labels\n        )\n        custom_tag_labels: Final = frozenset(\n            _sanitize_prometheus_label_name(f\"tag_{tag}\") for tag in litellm.custom_prometheus_tags\n        )\n        return builtin_labels | _NON_ENUM_METRIC_LABELS | custom_metadata_labels | custom_tag_labels\n\n    def _validate_all_configurations(self, parsed_configs: list) -> ValidationResults:","sourceCodeStart":720,"sourceCodeEnd":756,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/integrations/prometheus.py#L720-L756","documentation":"The global exclude settings — prometheus_exclude_metrics and prometheus_exclude_labels — are validated against closed sets: metric names must be among DEFINED_PROMETHEUS_METRICS (Literal args) and label names among every label a metric can emit (enum labels, hard-coded labels, sanitized custom labels). Any unknown name yields one line per offender, and all lines are joined into this ValueError.","triggerScenarios":"Setting prometheus_exclude_metrics=['litellm_proxy_requst_total'] (typo) or excluding a label that no metric defines; excludes written for an older litellm version whose metric/label inventory changed; excluding a custom label while custom_prometheus_metadata_labels is empty.","commonSituations":"Version drift after upgrading litellm (renamed metrics), copy-pasted exclude lists between projects, excludes for labels that only exist when certain callbacks or tag configs are active.","solutions":["Correct each name listed verbatim in the error text","Enumerate valid metric names from litellm.integrations.prometheus.DIFF... (DEFINED_PROMETHEUS_METRICS Literal args, e.g. via typing.get_args) and valid labels via the callback's _all_defined_labels()","Drop stale exclude entries after upgrading litellm","Keep exclude lists in version control next to the litellm version pin so they drift visibly"],"exampleFix":"# before\nos.environ['PROMETHEUS_EXCLUDE_METRICS'] = 'litellm_llm_api_latency_metric,typo_metric'\n\n# after\nfrom typing import get_args\nfrom litellm.integrations.prometheus import DEFINED_PROMETHEUS_METRICS\nvalid = set(get_args(DEFINED_PROMETHEUS_METRICS))\nexcludes = [m for m in ['litellm_llm_api_latency_metric', 'typo_metric'] if m in valid]\nos.environ['PROMETHEUS_EXCLUDE_METRICS'] = ','.join(excludes)","handlingStrategy":"validation","validationCode":"from typing import get_args\nfrom litellm.integrations.prometheus import DEFINED_PROMETHEUS_METRICS\n\nvalid_metrics = set(get_args(DEFINED_PROMETHEUS_METRICS))\ninvalid = [m for m in exclude_metrics_list if m not in valid_metrics]\nassert not invalid, f'unknown metrics in exclude list: {invalid}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive exclude lists from DEFINED_PROMETHEUS_METRICS at build time instead of hardcoding","Add a unit test that validates excludes whenever the litellm pin changes","Treat unknown-name errors as drift signals, not typos only"],"tags":["prometheus","metrics","exclusion","validation","config"],"backgroundTag":"invalid-config-value","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}