{"record":{"id":"40c0c0cb80665052","repo":"mlflow/mlflow","slug":"third-party-scorer-type-self-name-instance","errorCode":null,"errorMessage":"Third-party scorer {type(self).__name__}: instance `_metric_name='{self._metric_name}'` does not match class ClassVar `metric_name='{class_metric_name}'`.","messagePattern":"Third-party scorer (.+?): instance `_metric_name='(.+?)'` does not match class ClassVar `metric_name='(.+?)'`\\.","errorType":"validation","errorClass":"MlflowException","httpStatus":null,"severity":"error","filePath":"mlflow/genai/scorers/base.py","lineNumber":1188,"sourceCode":"            error_message=(\n                \"Scorer must be a builtin, decorator, or third-party scorer to be copied.\"\n            )\n        )\n\n        if self.kind == ScorerKind.THIRD_PARTY:\n            # Rebuild via __init__ — some third-party metrics (e.g. RAGAS) hold\n            # `instructor`-wrapped clients whose __getattr__ recurses infinitely\n            # on deepcopy.\n            init_kwargs = dict(self._metric_kwargs)\n            # Two shapes of third-party class: (a) base wrappers (`RagasScorer` etc.)\n            # have no `metric_name` ClassVar — pass it as a kwarg; (b) concrete\n            # subclasses (`ExactMatch`) pin it via ClassVar and forward to\n            # `super().__init__`, so re-passing raises \"multiple values\".\n            class_metric_name = getattr(type(self), \"metric_name\", None)\n            if class_metric_name is None:\n                init_kwargs[\"metric_name\"] = self._metric_name\n            elif class_metric_name != self._metric_name:\n                raise MlflowException.invalid_parameter_value(\n                    f\"Third-party scorer {type(self).__name__}: instance \"\n                    f\"`_metric_name='{self._metric_name}'` does not match class \"\n                    f\"ClassVar `metric_name='{class_metric_name}'`.\"\n                )\n            if self._model is not None:\n                init_kwargs[\"model\"] = self._model\n            copy = type(self)(**init_kwargs)\n            copy.name = self.name\n            if self.description is not None:\n                copy.description = self.description\n            if self.aggregations is not None:\n                copy.aggregations = self.aggregations\n        elif self.kind == ScorerKind.ENSEMBLE:\n            # Copy each sub-scorer through its own _create_copy so kind-specific handling\n            # still applies; a deepcopy of `_scorers` would recurse infinitely on a\n            # third-party sub-scorer holding an `instructor`-wrapped client.\n            copy = make_scorer_ensemble(\n                name=self.name,","sourceCodeStart":1170,"sourceCodeEnd":1206,"githubUrl":"https://github.com/mlflow/mlflow/blob/6a27f2decc0b76eb1b54af31849784addb357dbc/mlflow/genai/scorers/base.py#L1170-L1206","documentation":"For third-party scorers (RAGAS, DeepEval, TruLens, Phoenix), Scorer._create_copy() reconstructs init kwargs when copying/registering. If the subclass pins metric_name as a ClassVar, the instance's _metric_name must match it; a mismatch means inconsistent state that would break re-instantiation, so MlflowException.invalid_parameter_value is raised.","triggerScenarios":"Subclassing a third-party scorer wrapper where __init__ sets instance _metric_name different from the class-level ClassVar metric_name (e.g., setting self._metric_name = \"my_ragas_score\" while the class declares metric_name = \"ragas_score\"), then calling .register() or _create_copy().","commonSituations":"Customizing a RAGAS/DeepEval scorer by overriding attributes after init; copying scorer code from examples and renaming one of the two fields; monkeypatching _metric_name at runtime.","solutions":["Make the instance attribute match: self._metric_name = ClassVar value (or vice versa).","Don't override _metric_name on ClassVar-pinned subclasses; instead override the ClassVar metric_name on your subclass.","Pass the metric name via __init__ to the parent (super().__init__(metric_name=...)) rather than reassigning attributes afterward."],"exampleFix":"// before\nclass MyRagas(RagasScorer):\n    metric_name = \"ragas_faithfulness\"\n    def __init__(self):\n        super().__init__()\n        self._metric_name = \"faithfulness_v2\"  # mismatch\n\n// after\nclass MyRagas(RagasScorer):\n    metric_name = \"faithfulness_v2\"\n    def __init__(self):\n        super().__init__(metric_name=\"faithfulness_v2\")","handlingStrategy":"validation","validationCode":"cls = type(scorer)\nclass_metric = getattr(cls, \"metric_name\", None)\ninst_metric = getattr(scorer, \"_metric_name\", None)\nif class_metric is not None and inst_metric is not None and class_metric != inst_metric:\n    raise ValueError(f\"{cls.__name__}: _metric_name ({inst_metric!r}) != ClassVar metric_name ({class_metric!r})\")","typeGuard":"def metric_names_consistent(s) -> bool:\n    cm = getattr(type(s), \"metric_name\", None)\n    return cm is None or getattr(s, \"_metric_name\", cm) == cm","tryCatchPattern":"try:\n    scorer.register()\nexcept MlflowException as e:\n    if \"does not match class ClassVar\" in str(e):\n        scorer._metric_name = type(scorer).metric_name\n        scorer.register()\n    else:\n        raise","preventionTips":["Never assign self._metric_name in subclasses that pin metric_name via ClassVar","Pass metric_name through super().__init__(metric_name=...) instead of attribute reassignment","Add an invariant test: instance _metric_name equals type metric_name when ClassVar is present"],"tags":["python","mlflow-genai","third-party-scorers","state-inconsistency"],"backgroundTag":"inconsistent-configuration-state","analyzedSha":"6a27f2decc0b76eb1b54af31849784addb357dbc","analyzedAt":"2026-08-29T20:54:51.419Z","schemaVersion":2},"datasetVersion":"2026-08-29T22:17:34.462Z"}