mlflow/mlflow · error · MlflowException

Failed to create InstructionsJudge scorer '{serialized.name}

Error message

Failed to create InstructionsJudge scorer '{serialized.name}': {e}

What it means

After type-validating instructions_judge_data, MLflow reconstructs the live InstructionsJudge scorer by calling its constructor with the serialized fields. If the constructor itself throws (invalid inference_params combination, bad aggregation spec, incompatible feedback_value_type), the exception is wrapped in MlflowException.invalid_parameter_value with this message.

Source

Thrown at mlflow/genai/scorers/base.py:539

            feedback_value_type = str  # default to str
            if "feedback_value_type" in data and data["feedback_value_type"] is not None:
                feedback_value_type = InstructionsJudge._deserialize_feedback_value_type(
                    data["feedback_value_type"]
                )

            try:
                return InstructionsJudge(
                    name=serialized.name,
                    description=serialized.description,
                    instructions=data["instructions"],
                    model=data["model"],
                    feedback_value_type=feedback_value_type,
                    generate_rationale_first=data.get("generate_rationale_first", False),
                    inference_params=data.get("inference_params"),
                    aggregations=serialized.aggregations,
                )
            except Exception as e:
                raise MlflowException.invalid_parameter_value(
                    f"Failed to create InstructionsJudge scorer '{serialized.name}': {e}"
                )

        # Handle MemoryAugmentedJudge scorers
        elif serialized.memory_augmented_judge_data is not None:
            from mlflow.genai.judges.optimizers.memalign.optimizer import MemoryAugmentedJudge

            return MemoryAugmentedJudge._from_serialized(serialized)

        elif serialized.third_party_scorer_data is not None:
            data = serialized.third_party_scorer_data
            module_path = data.get("module") or ""
            class_name = data.get("class")
            metric_name = data.get("metric_name")
            if not any(
                module_path == m or module_path.startswith(m + ".")
                for m in THIRD_PARTY_SCORER_ALLOWED_MODULES
            ):

View on GitHub (pinned to 6a27f2decc)

Solutions

Read the wrapped exception `{e}` for the concrete constructor failure and fix that field
Re-create the scorer in code (make_instructions_judge or equivalent) and re-serialize it with the current MLflow version
Validate inference_params against the model endpoint's accepted parameters
Pin/align MLflow versions between writer and reader of the serialized scorer

Example fix

// before
{"inference_params": {"temperature": "0.7"}}
// after
{"inference_params": {"temperature": 0.7}}

Defensive patterns

Strategy: try-catch

Validate before calling

ip = data.get("inference_params", {})
assert isinstance(ip, dict) and all(not isinstance(v, str) or v for v in ip.values())

Type guard

def has_buildable_judge_params(data: dict) -> bool:
    ip = data.get("inference_params")
    return ip is None or (isinstance(ip, dict) and bool(ip))

Try / catch

try:
    scorer = Scorer.model_validate(data)
except MlflowException as e:
    if "Failed to create InstructionsJudge" in str(e):
        scorer = rebuild_instructions_judge(data, fallback_params={"temperature": 0})
    else:
        raise

Prevention

Test deserialization in CI with the same MLflow version used in production
Keep inference_params minimal and validated against the judge model endpoint
Pin MLflow versions across environments that share serialized scorers

When it happens

Trigger: Scorer.model_validate on data that passed type checks but fails InstructionsJudge creation — e.g. unsupported aggregation names, inference_params missing required model/uri entries, feedback_value_type not accepted by the judge implementation.

Common situations: Serialized on one MLflow version, deserialized on another where the constructor signature or validation changed; hand-tuned inference_params (e.g. wrong temperature type or missing target); custom aggregations not recognized at build time.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/54436470891281ed. Report an issue: GitHub.