{"record":{"id":"2f65a54e9559b107","repo":"mlflow/mlflow","slug":"exception-while-attempting-to-initialize-jvm-side","errorCode":null,"errorMessage":"Exception while attempting to initialize JVM-side state for Spark datasource autologging. Note that Spark datasource autologging only works with Spark 3.0 and above. Please create a new Spark session with required Spark version and ensure you have the mlflow-spark JAR attached to your Spark session as described in https://mlflow.org/docs/latest/tracking/autolog.html#spark Exception:\n{e}","messagePattern":"Exception while attempting to initialize JVM-side state for Spark datasource autologging\\. Note that Spark datasource autologging only works with Spark 3\\.0 and above\\. Please create a new Spark session with required Spark version and ensure you have the mlflow-spark JAR attached to your Spark session as described in https://mlflow\\.org/docs/latest/tracking/autolog\\.html#spark Exception:\n(.+?)","errorType":"exception","errorClass":"MlflowException","httpStatus":null,"severity":"error","filePath":"mlflow/spark/autologging.py","lineNumber":179,"sourceCode":"        auth_token=params.auth_token,\n    )\n    callback_server_started = gw.start_callback_server(callback_server_params)\n\n    try:\n        event_publisher = _get_jvm_event_publisher(spark_context)\n        event_publisher.init(1)\n        _spark_table_info_listener = PythonSubscriber()\n        event_publisher.register(_spark_table_info_listener)\n    except Exception as e:\n        if callback_server_started:\n            try:\n                gw.shutdown_callback_server()\n            except Exception as e:\n                _logger.warning(\n                    \"Failed to shut down Spark callback server for autologging: %s\", str(e)\n                )\n        _spark_table_info_listener = None\n        raise MlflowException(\n            \"Exception while attempting to initialize JVM-side state for Spark datasource \"\n            \"autologging. Note that Spark datasource autologging only works with Spark 3.0 \"\n            \"and above. Please create a new Spark session with required Spark version and \"\n            \"ensure you have the mlflow-spark JAR attached to your Spark session as described \"\n            f\"in https://mlflow.org/docs/latest/tracking/autolog.html#spark Exception:\\n{e}\"\n        )\n\n    # Register context provider for Spark autologging\n    from mlflow.tracking.context.registry import _run_context_provider_registry\n\n    _run_context_provider_registry.register(SparkAutologgingContext)\n\n    _logger.info(\"Autologging successfully enabled for spark.\")\n\n\ndef _get_repl_id():\n    \"\"\"\n    Get a unique REPL ID for a PythonSubscriber instance. This is used to distinguish between","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/mlflow/mlflow/blob/6a27f2decc0b76eb1b54af31849784addb357dbc/mlflow/spark/autologging.py#L161-L197","documentation":"MLflow wraps any exception raised while initializing JVM-side state for Spark datasource autologging (attaching the table-info listener, starting the Py4j callback server) into this MlflowException. The most common underlying cause is the mlflow-spark JAR not being attached to the Spark session, or running a Spark version below 3.0. The original exception is appended after 'Exception:'.","triggerScenarios":"mlflow.spark.autolog() invoked in a Spark session where the mlflow-spark package/JAR is not attached; JVM registration of _spark_table_info_listener throws; Py4j gateway or callback server setup fails.","commonSituations":"Forgetting --jars or spark.jars.packages config for mlflow-spark in cluster setup; cluster JVM classpath missing the listener class; Spark session created before mlflow-spark was installed; incompatible Spark/JVM state.","solutions":["Attach the mlflow-spark JAR to the Spark session (Databricks: install the mlflow-spark PyPI package on the cluster; other clusters: --jars or spark.jars.packages with mlflow-spark)","Verify the Spark session is 3.0+ and restart the session after installing the JAR so it is on the classpath","Read the embedded 'Exception: ...' text for the root cause (e.g. ClassNotFoundException indicates the JAR is missing)"],"exampleFix":"// before\nspark = SparkSession.builder.getOrCreate()\nmlflow.spark.autolog()\n// after\nspark = (SparkSession.builder\n    .config(\"spark.jars.packages\", \"org.mlflow:mlflow-spark:2.x.x\")\n    .getOrCreate())\nmlflow.spark.autolog()","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    mlflow.spark.autolog()\nexcept MlflowException as e:\n    root = str(e).split(\"Exception:\")[-1].strip()\n    if \"ClassNotFoundException\" in root or \"mlflow-spark\" in str(e):\n        raise RuntimeError(\"Attach the mlflow-spark JAR/package to the Spark cluster and restart the session\") from e\n    raise","preventionTips":["Always install the mlflow-spark package/JAR on the cluster before starting the session","Restart the Spark session after changing jars/packages config","Log spark.version and jar config during environment bootstrap"],"tags":["spark","jvm","autologging","jar-missing"],"backgroundTag":"jvm-initialization-failed","analyzedSha":"6a27f2decc0b76eb1b54af31849784addb357dbc","analyzedAt":"2026-08-29T20:54:51.419Z","schemaVersion":2},"datasetVersion":"2026-08-29T22:17:34.462Z"}