prestodb/presto · error · IllegalArgumentException

SparkConf must be initialized with PrestoSparkConfInitialize

Error message

SparkConf must be initialized with PrestoSparkConfInitializer.initialize before creating SparkContext

What it means

Presto requires SparkConf objects to have gone through PrestoSparkConfInitializer.initialize (which registers Kryo classes and sets an INITIALIZED_MARKER). checkInitialized validates a SparkContext's conf after creation and throws IllegalArgumentException if the marker is absent, meaning SparkContext was created with an uninitialized conf.

Source

Thrown at presto-spark-classloader-interface/src/main/java/com/facebook/presto/spark/classloader_interface/PrestoSparkConfInitializer.java:52

    private static void registerKryoClasses(SparkConf sparkConf)
    {
        sparkConf.registerKryoClasses(new Class[] {
                MutablePartitionId.class,
                PrestoSparkMutableRow.class,
                PrestoSparkSerializedPage.class,
                SerializedPrestoSparkTaskDescriptor.class,
                SerializedTaskInfo.class,
                PrestoSparkShuffleStats.class,
                PrestoSparkStorageHandle.class,
                SerializedPrestoSparkTaskSource.class
        });
    }

    public static void checkInitialized(SparkContext sparkContext)
    {
        SparkConf sparkConf = sparkContext.getConf();
        if (sparkConf.get(INITIALIZED_MARKER, null) == null) {
            throw new IllegalArgumentException("SparkConf must be initialized with PrestoSparkConfInitializer.initialize before creating SparkContext");
        }
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

1. Call PrestoSparkConfInitializer.initialize(sparkConf) before constructing the SparkContext
2. Ensure the same conf object that was initialized is passed to the SparkContext constructor
3. In tests/tools, replicate the Presto launch path instead of new SparkConf()

Example fix

// before
SparkConf conf = new SparkConf().setAppName("presto");
SparkContext ctx = new SparkContext(conf);
// after
SparkConf conf = new SparkConf().setAppName("presto");
PrestoSparkConfInitializer.initialize(conf);
SparkContext ctx = new SparkContext(conf);

Preconditions.checkState(sparkConf.get("__presto_conf_initialized__", null) != null, "call PrestoSparkConfInitializer.initialize first");

try { PrestoSparkConfInitializer.checkInitialized(sparkContext); } catch (IllegalArgumentException e) { log.error("SparkContext created without Presto init; recreate with initialized conf"); throw e; }

When it happens

Trigger: Creating a SparkContext/SparkSession with a raw new SparkConf() (or a conf loaded from a system properties default) that never passed PrestoSparkConfInitializer.initialize.

Common situations: Standard Spark application bootstrap not adapted for Presto on Spark; rebuilding the conf in tests or tools bypassing the initializer; Spark creating a default conf internally.

Related errors

• SparkConf is already initialized
• UNSUPPORTED_STORAGE_TYPE
• File doesn't exist %s
• INVALID_RETRY_EXECUTION_STRATEGY
• ACCUMULO_TABLE_EXISTS

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/2c691abd4f440a83. Report an issue: GitHub.