mlflow/mlflow · error · NotImplementedError

Alignment is not supported for session-level scorers.

Error message

Alignment is not supported for session-level scorers.

What it means

Aligning a scorer (optimizing its instructions/criteria against trace feedback) is intentionally unsupported for session-level scorers — scorers that evaluate an entire multi-span session rather than individual traces. The `align` method of Scorers (mlflow/genai/judges/base.py) raises NotImplementedError as a guard because the alignment optimizers operate on per-trace expectations.

Source

Thrown at mlflow/genai/judges/base.py:133

        Returns:
            A new Judge instance that is better aligned with the input traces.

        Raises:
            NotImplementedError: If called on a session-level scorer. Alignment is currently
                only supported for single-turn scorers.

        Note on Logging:
            By default, alignment optimization shows minimal progress information.
            To see detailed optimization output, set the optimizer's logger to DEBUG::

                import logging

                # For MemAlign optimizer (default)
                logging.getLogger("mlflow.genai.judges.optimizers.memalign").setLevel(logging.DEBUG)
        """
        if self.is_session_level_scorer:
            raise NotImplementedError("Alignment is not supported for session-level scorers.")

        if optimizer is None:
            optimizer = get_default_optimizer()
        return optimizer.align(self, traces)

View on GitHub (pinned to 6a27f2decc)

Solutions

1. Use a trace-level scorer instead if you need alignment — move the judge down to per-trace evaluation.
2. Manually tune the session-level scorer's instructions/criteria instead of calling align.
3. Check `scorer.is_session_level_scorer` before calling align and branch accordingly.
4. Upgrade MLflow: if you believe session-level alignment should exist, verify your version's docs; the API may have changed.

Example fix

// before
optimized = session_scorer.align(traces)  # NotImplementedError

// after
if session_scorer.is_session_level_scorer:
    # align only trace-level scorers
    optimized = trace_level_scorer.align(traces)
else:
    optimized = session_scorer.align(traces)

if getattr(scorer, "is_session_level_scorer", False):
    raise ValueError("scorer is session-level; align() is unsupported")
optimized = scorer.align(traces)

def is_alignable(scorer) -> bool:
    return not getattr(scorer, "is_session_level_scorer", False)

try:
    optimized = scorer.align(traces)
except NotImplementedError:
    logger.warning("%s is session-level; skipping alignment", scorer.name)
    optimized = scorer

When it happens

Trigger: Calling `scorer.align(traces)` (or `mlflow.genai.optimize(...)` on a session-level scorer) where the scorer was registered/constructed with session-level semantics (is_session_level_scorer is True), e.g. a judge created to evaluate a whole session via `expectations` spanning multiple traces.

Common situations: Users attach scorers to a session-level evaluation (e.g. multi-turn agent conversations registered with session-level aggregation) then attempt automated alignment; code reused across trace-level and session-level scorers calls align unconditionally.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Tags are not supported in Databricks environments. Tags are
• `version` is only supported for Databricks datasets.
• Dataset tag operations are not available in Databricks yet.
• Dataset association operations are not available in Databric
• Dataset association operations are not supported with FileSt

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/264b403c6c8072ac. Report an issue: GitHub.