mlflow/mlflow · error · MlflowException

Exception while attempting to initialize JVM-side state for

Error message

Exception while attempting to initialize JVM-side state for Spark datasource autologging. Note that Spark datasource autologging only works with Spark 3.0 and above. Please create a new Spark session with required Spark version and ensure you have the mlflow-spark JAR attached to your Spark session as described in https://mlflow.org/docs/latest/tracking/autolog.html#spark Exception:
{e}

What it means

MLflow wraps any exception raised while initializing JVM-side state for Spark datasource autologging (attaching the table-info listener, starting the Py4j callback server) into this MlflowException. The most common underlying cause is the mlflow-spark JAR not being attached to the Spark session, or running a Spark version below 3.0. The original exception is appended after 'Exception:'.

Source

Thrown at mlflow/spark/autologging.py:179

        auth_token=params.auth_token,
    )
    callback_server_started = gw.start_callback_server(callback_server_params)

    try:
        event_publisher = _get_jvm_event_publisher(spark_context)
        event_publisher.init(1)
        _spark_table_info_listener = PythonSubscriber()
        event_publisher.register(_spark_table_info_listener)
    except Exception as e:
        if callback_server_started:
            try:
                gw.shutdown_callback_server()
            except Exception as e:
                _logger.warning(
                    "Failed to shut down Spark callback server for autologging: %s", str(e)
                )
        _spark_table_info_listener = None
        raise MlflowException(
            "Exception while attempting to initialize JVM-side state for Spark datasource "
            "autologging. Note that Spark datasource autologging only works with Spark 3.0 "
            "and above. Please create a new Spark session with required Spark version and "
            "ensure you have the mlflow-spark JAR attached to your Spark session as described "
            f"in https://mlflow.org/docs/latest/tracking/autolog.html#spark Exception:\n{e}"
        )

    # Register context provider for Spark autologging
    from mlflow.tracking.context.registry import _run_context_provider_registry

    _run_context_provider_registry.register(SparkAutologgingContext)

    _logger.info("Autologging successfully enabled for spark.")


def _get_repl_id():
    """
    Get a unique REPL ID for a PythonSubscriber instance. This is used to distinguish between

View on GitHub (pinned to 6a27f2decc)

Solutions

  1. Attach the mlflow-spark JAR to the Spark session (Databricks: install the mlflow-spark PyPI package on the cluster; other clusters: --jars or spark.jars.packages with mlflow-spark)
  2. Verify the Spark session is 3.0+ and restart the session after installing the JAR so it is on the classpath
  3. Read the embedded 'Exception: ...' text for the root cause (e.g. ClassNotFoundException indicates the JAR is missing)

Example fix

// before
spark = SparkSession.builder.getOrCreate()
mlflow.spark.autolog()
// after
spark = (SparkSession.builder
    .config("spark.jars.packages", "org.mlflow:mlflow-spark:2.x.x")
    .getOrCreate())
mlflow.spark.autolog()
Defensive patterns

Strategy: try-catch

Try / catch

try:
    mlflow.spark.autolog()
except MlflowException as e:
    root = str(e).split("Exception:")[-1].strip()
    if "ClassNotFoundException" in root or "mlflow-spark" in str(e):
        raise RuntimeError("Attach the mlflow-spark JAR/package to the Spark cluster and restart the session") from e
    raise

Prevention

When it happens

Trigger: mlflow.spark.autolog() invoked in a Spark session where the mlflow-spark package/JAR is not attached; JVM registration of _spark_table_info_listener throws; Py4j gateway or callback server setup fails.

Common situations: Forgetting --jars or spark.jars.packages config for mlflow-spark in cluster setup; cluster JVM classpath missing the listener class; Spark session created before mlflow-spark was installed; incompatible Spark/JVM state.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/2f65a54e9559b107. Report an issue: GitHub.