mlflow/mlflow · error · MlflowException

Third-party scorer {type(self).__name__}: instance `_metric_

Error message

Third-party scorer {type(self).__name__}: instance `_metric_name='{self._metric_name}'` does not match class ClassVar `metric_name='{class_metric_name}'`.

What it means

For third-party scorers (RAGAS, DeepEval, TruLens, Phoenix), Scorer._create_copy() reconstructs init kwargs when copying/registering. If the subclass pins metric_name as a ClassVar, the instance's _metric_name must match it; a mismatch means inconsistent state that would break re-instantiation, so MlflowException.invalid_parameter_value is raised.

Source

Thrown at mlflow/genai/scorers/base.py:1188

            error_message=(
                "Scorer must be a builtin, decorator, or third-party scorer to be copied."
            )
        )

        if self.kind == ScorerKind.THIRD_PARTY:
            # Rebuild via __init__ — some third-party metrics (e.g. RAGAS) hold
            # `instructor`-wrapped clients whose __getattr__ recurses infinitely
            # on deepcopy.
            init_kwargs = dict(self._metric_kwargs)
            # Two shapes of third-party class: (a) base wrappers (`RagasScorer` etc.)
            # have no `metric_name` ClassVar — pass it as a kwarg; (b) concrete
            # subclasses (`ExactMatch`) pin it via ClassVar and forward to
            # `super().__init__`, so re-passing raises "multiple values".
            class_metric_name = getattr(type(self), "metric_name", None)
            if class_metric_name is None:
                init_kwargs["metric_name"] = self._metric_name
            elif class_metric_name != self._metric_name:
                raise MlflowException.invalid_parameter_value(
                    f"Third-party scorer {type(self).__name__}: instance "
                    f"`_metric_name='{self._metric_name}'` does not match class "
                    f"ClassVar `metric_name='{class_metric_name}'`."
                )
            if self._model is not None:
                init_kwargs["model"] = self._model
            copy = type(self)(**init_kwargs)
            copy.name = self.name
            if self.description is not None:
                copy.description = self.description
            if self.aggregations is not None:
                copy.aggregations = self.aggregations
        elif self.kind == ScorerKind.ENSEMBLE:
            # Copy each sub-scorer through its own _create_copy so kind-specific handling
            # still applies; a deepcopy of `_scorers` would recurse infinitely on a
            # third-party sub-scorer holding an `instructor`-wrapped client.
            copy = make_scorer_ensemble(
                name=self.name,

View on GitHub (pinned to 6a27f2decc)

Solutions

1. Make the instance attribute match: self._metric_name = ClassVar value (or vice versa).
2. Don't override _metric_name on ClassVar-pinned subclasses; instead override the ClassVar metric_name on your subclass.
3. Pass the metric name via __init__ to the parent (super().__init__(metric_name=...)) rather than reassigning attributes afterward.

Example fix

// before
class MyRagas(RagasScorer):
    metric_name = "ragas_faithfulness"
    def __init__(self):
        super().__init__()
        self._metric_name = "faithfulness_v2"  # mismatch

// after
class MyRagas(RagasScorer):
    metric_name = "faithfulness_v2"
    def __init__(self):
        super().__init__(metric_name="faithfulness_v2")

cls = type(scorer)
class_metric = getattr(cls, "metric_name", None)
inst_metric = getattr(scorer, "_metric_name", None)
if class_metric is not None and inst_metric is not None and class_metric != inst_metric:
    raise ValueError(f"{cls.__name__}: _metric_name ({inst_metric!r}) != ClassVar metric_name ({class_metric!r})")

def metric_names_consistent(s) -> bool:
    cm = getattr(type(s), "metric_name", None)
    return cm is None or getattr(s, "_metric_name", cm) == cm

try:
    scorer.register()
except MlflowException as e:
    if "does not match class ClassVar" in str(e):
        scorer._metric_name = type(scorer).metric_name
        scorer.register()
    else:
        raise

When it happens

Trigger: Subclassing a third-party scorer wrapper where __init__ sets instance _metric_name different from the class-level ClassVar metric_name (e.g., setting self._metric_name = "my_ragas_score" while the class declares metric_name = "ragas_score"), then calling .register() or _create_copy().

Common situations: Customizing a RAGAS/DeepEval scorer by overriding attributes after init; copying scorer code from examples and renaming one of the two fields; monkeypatching _metric_name at runtime.

Related errors

• Session-level scorers require traces with session IDs. The f
• Every item in the list must be a dictionary.
• Invalid type for parameter `data`. Expected a list of dictio
• The `pyspark` package is required to use mlflow.genai.evalua
• The dataset is empty. Please provide a non-empty dataset.

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/40c0c0cb80665052. Report an issue: GitHub.