mlflow/mlflow · error · MlflowException

Scorers [{scorer_details}] return non-numerical values that

Error message

Scorers [{scorer_details}] return non-numerical values that cannot be automatically aggregated. Please provide an `objective` function to aggregate these values into a single score for optimization.

What it means

Raised when converting scorers to legacy metrics for optimization: the scorer returned a non-numerical value (e.g. string, dict, bool of non-numeric type) that MLflow cannot automatically aggregate into a single optimization objective. The optimizer needs one number per scorer to maximize/minimize.

Source

Thrown at mlflow/genai/optimize/util.py:210

            numeric_value = _convert_to_numeric(score)
            if numeric_value is not None:
                numeric_scores[name] = numeric_value

        if objective is not None:
            return objective(scores), rationales, numeric_scores

        # If all scores were convertible, use sum as default aggregation
        if len(numeric_scores) == len(scores):
            # We average the scores to get the score between 0 and 1.
            aggregated = sum(numeric_scores.values()) / len(numeric_scores)
            return aggregated, rationales, numeric_scores

        # Otherwise, report error with actual types
        non_convertible = {
            k: type(v).__name__ for k, v in scores.items() if k not in numeric_scores
        }
        scorer_details = ", ".join([f"{k} (type: {t})" for k, t in non_convertible.items()])
        raise MlflowException(
            f"Scorers [{scorer_details}] return non-numerical values that cannot be "
            "automatically aggregated. Please provide an `objective` function to aggregate "
            "these values into a single score for optimization."
        )

    return metric

View on GitHub (pinned to 6a27f2decc)

Solutions

Provide an `objective` function to optimize_prompts that maps the scorer outputs to a single numeric score
Change the scorer to return numeric values (e.g. 0.0/1.0 instead of 'yes'/'no')
If a Feedback wraps the value, ensure feedback.value is numeric rather than the raw string

Example fix

// before
optimize_prompts(..., scorers=[verdict_scorer])
// after
optimize_prompts(..., scorers=[verdict_scorer], objective=lambda scores: scores.get("verdict_scorer") == "yes")

Defensive patterns

Strategy: type-guard

Validate before calling

from mlflow.entities import Feedback

def scorer_values_numeric(scorers, sample_inputs, predict_fn):
    for s in scorers:
        fb = s(inputs=sample_inputs, outputs=predict_fn(sample_inputs), expectations={})
        val = fb.value if isinstance(fb, Feedback) else fb
        if val is not None and not isinstance(val, (int, float)):
            raise TypeError(f"{s.name} returns non-numeric {type(val).__name__}")

Type guard

def is_numeric_score(v) -> bool:
    import numbers
    return isinstance(v, numbers.Number) and not isinstance(v, bool) or isinstance(v, bool)

Try / catch

from mlflow.exceptions import MlflowException
try:
    mlflow.genai.optimize_prompts(..., scorers=scorers)
except MlflowException as e:
    if "non-numerical values" in str(e):
        # retry with an objective that coerces to numbers
        mlflow.genai.optimize_prompts(..., scorers=scorers, objective=lambda s: float(s))
    else:
        raise

Prevention

Make custom scorers return int/float values
Supply an objective function whenever scorers emit labels or objects
Smoke-test scorers on one sample before running optimization
Keep LLM judge output formats numeric (e.g. 1-5 scores)

When it happens

Trigger: Calling optimize_prompts with custom scorers (or objective-less setups) whose returned Feedback values are non-numeric strings/objects, so _convert_scorer_to_legacy_metric cannot coerce them.

Common situations: Custom scorers returning text verdicts like 'good'/'bad' or dicts; LLM judges configured to output strings; refactor changed a scorer from returning 0-1 floats to labels; forgetting to supply an `objective` function.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/b1df70b39b18bc7c. Report an issue: GitHub.