{"record":{"id":"7cbd59a612883eea","repo":"prestodb/presto","slug":"not-supported-7cbd59","errorCode":"NOT_SUPPORTED","errorMessage":"Automatic writers scaling is not supported by Presto on Spark","messagePattern":"Automatic writers scaling is not supported by Presto on Spark","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-spark-base/src/main/java/com/facebook/presto/spark/planner/PrestoSparkRddFactory.java","lineNumber":145,"sourceCode":"    public <T extends PrestoSparkTaskOutput> JavaPairRDD<MutablePartitionId, T> createSparkRdd(\n            JavaSparkContext sparkContext,\n            Session session,\n            PlanFragment fragment,\n            Map<PlanFragmentId, JavaPairRDD<MutablePartitionId, PrestoSparkMutableRow>> rddInputs,\n            Map<PlanFragmentId, Broadcast<?>> broadcastInputs,\n            PrestoSparkTaskExecutorFactoryProvider executorFactoryProvider,\n            CollectionAccumulator<SerializedTaskInfo> taskInfoCollector,\n            CollectionAccumulator<PrestoSparkShuffleStats> shuffleStatsCollector,\n            TableWriteInfo tableWriteInfo,\n            Class<T> outputType,\n            TempStorage nativeTempStorage)\n    {\n        checkArgument(!fragment.getStageExecutionDescriptor().isStageGroupedExecution(), \"unexpected grouped execution fragment: %s\", fragment.getId());\n\n        PartitioningHandle partitioning = fragment.getPartitioning();\n\n        if (partitioning.equals(SCALED_WRITER_DISTRIBUTION)) {\n            throw new PrestoException(NOT_SUPPORTED, \"Automatic writers scaling is not supported by Presto on Spark\");\n        }\n\n        checkArgument(!partitioning.equals(COORDINATOR_DISTRIBUTION), \"COORDINATOR_DISTRIBUTION fragment must be run on the driver\");\n        checkArgument(!partitioning.equals(FIXED_BROADCAST_DISTRIBUTION), \"FIXED_BROADCAST_DISTRIBUTION can only be set as an output partitioning scheme, and not as a fragment distribution\");\n        checkArgument(!partitioning.equals(FIXED_PASSTHROUGH_DISTRIBUTION), \"FIXED_PASSTHROUGH_DISTRIBUTION can only be set as local exchange partitioning\");\n\n        // TODO: ARBITRARY_DISTRIBUTION is something very weird.\n        // TODO: It doesn't have partitioning function, and it is never set as a fragment partitioning.\n        // TODO: We should consider removing ARBITRARY_DISTRIBUTION.\n        checkArgument(!partitioning.equals(ARBITRARY_DISTRIBUTION), \"ARBITRARY_DISTRIBUTION is not expected to be set as a fragment distribution\");\n\n        if (partitioning.equals(SINGLE_DISTRIBUTION) ||\n                partitioning.equals(FIXED_HASH_DISTRIBUTION) ||\n                partitioning.equals(FIXED_ARBITRARY_DISTRIBUTION) ||\n                partitioning.equals(SOURCE_DISTRIBUTION) ||\n                partitioning.getConnectorId().isPresent()) {\n            return createRdd(\n                    sparkContext,","sourceCodeStart":127,"sourceCodeEnd":163,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-spark-base/src/main/java/com/facebook/presto/spark/planner/PrestoSparkRddFactory.java#L127-L163","documentation":"createSparkRdd validates the fragment's partitioning handle before building the RDD. If the partitioning is SCALED_WRITER_DISTRIBUTION (automatic writer scaling), it throws PrestoException with code NOT_SUPPORTED because scaling writers dynamically is unsupported in Presto on Spark.","triggerScenarios":"A query fragment's output partitioning resolves to SCALED_WRITER_DISTRIBUTION while the RDD is being created in createSparkRdd — typically when scale-writers session property is enabled for writing queries.","commonSituations":"Users enabling SET SESSION scale_writers = true (or scale_writers-supported connector settings) on Presto on Spark during heavy write queries; porting tuned write queries from Presto-on-Hive to Spark.","solutions":["Disable the scale_writers session property (SET SESSION scale_writers = false) for Spark deployments","Remove connector/table properties that request scaled writer distribution","Tune writer parallelism statically instead (e.g. repartition output) since dynamic scaling is unavailable"],"exampleFix":"// before\nSET SESSION scale_writers = true;\n// after\nSET SESSION scale_writers = false; -- automatic writers scaling unsupported on Presto on Spark","handlingStrategy":"validation","validationCode":"if (\"true\".equals(session.getSystemProperty(\"scale_writers\"))) {\n    throw new PrestoException(NOT_SUPPORTED, \"scale_writers must be false on Presto on Spark\");\n}","typeGuard":null,"tryCatchPattern":"try { rddFactory.createSparkRdd(...) } catch (PrestoException e) { if (e.getErrorCode().getCode() == NOT_SUPPORTED.toErrorCode().getCode()) { log.error(\"scaled writers unsupported\", e); } throw e; }","preventionTips":["Never enable scale_writers in Spark deployment session catalogs","Use static repartitioning for write parallelism instead","Validate write-heavy query sessions in CI before production rollout"],"tags":["presto-on-spark","not-supported","scaled-writers","presto-exception"],"backgroundTag":"scaled-writers-unsupported","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-12T02:17:10.037Z"}