mlflow/mlflow · error · RuntimeError

Failed to read scala version.

Error message

Failed to read scala version.

What it means

MLflow determines the Scala version of the installed PySpark by launching a child process and reading the result from a queue. If the child process exits with a non-zero exit code (crashed, JVM failed to start, fork unsupported), MLflow raises this RuntimeError because it cannot read the Scala version needed for Spark model loading.

Source

Thrown at mlflow/utils/_spark_utils.py:76

        return os.environ["SPARK_SCALA_VERSION"]

    if spark := _get_active_spark_session():
        return _get_spark_scala_version_from_spark_session(spark)

    result_queue = multiprocessing.Queue()

    # If we need to create a new spark local session for reading scala version,
    # we have to create the temporal spark session in a child process,
    # if we create the temporal spark session in current process,
    # after terminating the temporal spark session, creating another spark session
    # with "spark.jars.packages" configuration doesn't work.
    proc = multiprocessing.Process(
        target=_get_spark_scala_version_child_proc_target, args=(result_queue,)
    )
    proc.start()
    proc.join()
    if proc.exitcode != 0:
        raise RuntimeError("Failed to read scala version.")

    return result_queue.get()


def _create_local_spark_session_for_loading_spark_model():
    from pyspark.sql import SparkSession

    return (
        SparkSession.builder
        .config("spark.python.worker.reuse", "true")
        # The config is a workaround for avoiding databricks delta cache issue when loading
        # some specific model such as ALSModel.
        .config("spark.databricks.io.cache.enabled", "false")
        # In Spark 3.1 and above, we need to set this conf explicitly to enable creating
        # a SparkSession on the workers
        .config("spark.executor.allowSparkContext", "true")
        # Binding "spark.driver.host" to 127.0.0.1 helps avoiding some local hostname
        # related issues (e.g. https://github.com/mlflow/mlflow/issues/5733).

View on GitHub (pinned to 6a27f2decc)

Solutions

1. Verify Java is installed and JAVA_HOME points to a valid JDK 8/11/17 compatible with the installed PySpark
2. Run `python -c "import pyspark; pyspark.SparkContext.getOrCreate()"` to confirm Spark itself starts; fix any JVM errors reported
3. Try setting the multiprocessing start method, e.g. multiprocessing.set_start_method('spawn') before importing mlflow, or run in an environment where fork works
4. As a workaround, set the Scala version explicitly if supported by your MLflow version, or load Spark models inside a real Spark cluster job instead of locally

Example fix

// before
import mlflow
model = mlflow.spark.load_model("runs:/abc/model")  # RuntimeError: Failed to read scala version.
// after
import multiprocessing
multiprocessing.set_start_method("spawn", force=True)
import mlflow
model = mlflow.spark.load_model("runs:/abc/model")

import shutil
if not (shutil.which("java") or __import__("os").environ.get("JAVA_HOME")):
    raise SystemExit("Install a JDK and set JAVA_HOME before loading Spark models")

try:
    model = mlflow.spark.load_model(uri)
except RuntimeError as e:
    if "Failed to read scala version" in str(e):
        model = load_spark_model_in_external_session(uri)
    else:
        raise

When it happens

Trigger: Calling mlflow.spark.load_model (or any code path invoking _get_spark_scala_version, e.g. creating a local Spark session for loading a Spark model) when the spawned multiprocessing child fails before it can put the Scala version into result_queue.

Common situations: Environments where multiprocessing fork of a JVM is unreliable (macOS spawn default, Windows, containers without /proc, restricted sandboxes), broken or mismatched JAVA_HOME, PySpark installed without a working JVM, or memory limits killing the forked JVM.

Related errors

• Invalid model type: '{model_type}'. Must be one of {list(mod
• INTERNAL_ERROR
• Failed to load base model '{effective_base_model}'. If the m
• Failed to load base model '{base_model}'. If the model has m
• The `pyspark` package is required to use mlflow.genai.evalua

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/52d8cf5779576921. Report an issue: GitHub.