mlflow/mlflow · warning · UserWarning

The custom metric definitions were serialized using MLflow {

Error message

The custom metric definitions were serialized using MLflow {}. Deserializing them with the current version {} might cause mismatches. Please ensure compatibility or consider regenerating the metrics using the current version.

What it means

A UserWarning raised in _deserialize_genai_metric_args (invoked by retrieve_custom_metrics) when the MLflow version recorded in the serialized custom metric file differs from the currently running MLflow VERSION. The metric definitions were written by a different MLflow version, and deserializing them may produce mismatches in evaluation behavior. MLflow warns rather than failing, since the format is usually compatible.

Source

Thrown at mlflow/metrics/genai/genai_metric.py:700


def _filter_by_field(df, field_name, value):
    return df[df[field_name] == value]


def _deserialize_genai_metric_args(args_dict):
    mlflow_version_at_ser = args_dict.pop("mlflow_version", None)
    fn_name = args_dict.pop("fn_name", None)
    if fn_name is None or mlflow_version_at_ser is None:
        raise MlflowException(
            message="The artifact JSON file appears to be corrupted and cannot be deserialized. "
            "Please regenerate the custom metrics and rerun the evaluation. "
            "Ensure that the file is correctly formatted and not tampered with.",
            error_code=INTERNAL_ERROR,
        )

    if mlflow_version_at_ser != VERSION:
        warnings.warn(
            f"The custom metric definitions were serialized using MLflow {mlflow_version_at_ser}. "
            f"Deserializing them with the current version {VERSION} might cause mismatches. "
            "Please ensure compatibility or consider regenerating the metrics "
            "using the current version.",
            UserWarning,
            stacklevel=2,
        )

    if fn_name == make_genai_metric_from_prompt.__name__:
        return make_genai_metric_from_prompt(**args_dict)

    examples = args_dict["examples"]
    if examples is not None:
        args_dict["examples"] = [EvaluationExample(**example) for example in examples]

    return make_genai_metric(**args_dict)

View on GitHub (pinned to 6a27f2decc)

Solutions

Regenerate/re-serialize the custom metrics using the current MLflow version
Pin the environment to the MLflow version used at serialization time if regeneration is not possible
Review the metric definitions for compatibility after the version change and re-run evaluation to confirm results

Example fix

// before
# metric file serialized with mlflow 2.9.0, env runs mlflow 2.17.0
metrics = retrieve_custom_metrics(...)
// after
$ pip install mlflow==2.9.0  # match serialization version, recreate metrics
# then regenerate metrics with the target version
metrics = retrieve_custom_metrics(...)

Defensive patterns

Strategy: validation

Validate before calling

import json, mlflow
from mlflow import VERSION
with open("custom_metrics.json") as f:
    data = json.load(f)
if data.get("mlflow_version") != VERSION:
    print(f"Metric serialized with MLflow {data.get('mlflow_version')}, current is {VERSION} — regenerate metrics or pin mlflow=={data.get('mlflow_version')}")

Try / catch

import warnings
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    metrics = retrieve_custom_metrics(file_path)
    if any("serialized using MLflow" in str(x.message) for x in w):
        regenerate_metrics_with_current_version()

Prevention

Regenerate custom metric definitions after every MLflow upgrade
Pin the MLflow version used for serialization and deserialization in CI and prod
Store metric definitions in version control so they can be regenerated deterministically

When it happens

Trigger: Calling mlflow.metrics.genai retrieve_custom_metrics (e.g., when running evaluation with stored custom metric definitions) where the `mlflow_version` field in the serialized file != the installed mlflow version.

Common situations: Evaluation run in a different environment than the one that created the metrics (CI vs local, upgraded mlflow); custom metrics saved months earlier and replayed after an mlflow upgrade.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/2f34c3b6558c20f6. Report an issue: GitHub.