{"record":{"id":"784ce52b9acf6590","repo":"apache/hadoop","slug":"multipart-uploads-are-disabled-for-the-filesystem","errorCode":null,"errorMessage":"Multipart uploads are disabled for the FileSystem, the committer can't proceed.","messagePattern":"Multipart uploads are disabled for the FileSystem, the committer can't proceed\\.","errorType":"exception","errorClass":"PathCommitException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java","lineNumber":222,"sourceCode":"  protected AbstractS3ACommitter(\n      Path outputPath,\n      TaskAttemptContext context) throws IOException {\n    super(outputPath, context);\n    setOutputPath(outputPath);\n    this.jobContext = requireNonNull(context, \"null job context\");\n    this.role = \"Task committer \" + context.getTaskAttemptID();\n    setConf(context.getConfiguration());\n    Pair<String, JobUUIDSource> id = buildJobUUID(\n        conf, context.getJobID());\n    this.uuid = id.getLeft();\n    this.uuidSource = id.getRight();\n    LOG.info(\"Job UUID {} source {}\", getUUID(), getUUIDSource().getText());\n    initOutput(outputPath);\n    LOG.debug(\"{} instantiated for job \\\"{}\\\" ID {} with destination {}\",\n        role, jobName(context), jobIdString(context), outputPath);\n    S3AFileSystem fs = getDestS3AFS();\n    if (!fs.isMultipartUploadEnabled()) {\n      throw new PathCommitException(outputPath, \"Multipart uploads are disabled for the FileSystem,\"\n          + \" the committer can't proceed.\");\n    }\n    // set this thread's context with the job ID.\n    // audit spans created in this thread will pick\n    // up this value., including the commit operations instance\n    // soon to be created.\n    new AuditContextUpdater(jobContext)\n        .updateCurrentAuditContext();\n\n    // the filesystem is the span source, always.\n    this.auditSpanSource = fs.getAuditSpanSource();\n    this.createJobMarker = context.getConfiguration().getBoolean(\n        CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER,\n        DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER);\n    // the statistics are shared between this committer and its operations.\n    this.committerStatistics = fs.newCommitterStatistics();\n    this.commitOperations = new CommitOperations(fs, committerStatistics,\n        outputPath.toString());","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L204-L240","documentation":"All S3A committers (staging, magic, directory) stage their work as S3 multipart uploads, so at committer construction AbstractS3ACommitter checks S3AFileSystem.isMultipartUploadEnabled() and aborts with PathCommitException if multipart uploads are disabled. The check is deliberately early: without multipart, pending commits could never be completed.","triggerScenarios":"Task or job committer initialization runs against a destination whose configuration sets fs.s3a.multipart.uploads.enabled=false (default is true). Magic committer integration is also disabled by the same flag, so magic-committed writes fail identically.","commonSituations":"An operator disabled multipart uploads cluster-wide (e.g. to work around an S3 or proxy issue) while Spark/Hive jobs still select an S3A committer via mapreduce.outputcommitter.factory.scheme.s3a; a job-level override sets the flag false for one path; copy-pasted 'tuning' configs carrying the false value.","solutions":["Remove the override or set fs.s3a.multipart.uploads.enabled=true (default) for jobs using S3A committers","If multipart must stay disabled on that filesystem, do not route its jobs to S3A committers: unset mapreduce.outputcommitter.factory.scheme.s3a so the classic FileOutputCommitter is used","Check both cluster core-site.xml and job/spark.hadoop.* overrides; the task-side config is the one that counts"],"exampleFix":"<!-- before -->\n<property><name>fs.s3a.multipart.uploads.enabled</name><value>false</value></property>\n\n<!-- after (or just remove the property; default is true) -->\n<property><name>fs.s3a.multipart.uploads.enabled</name><value>true</value></property>","handlingStrategy":"validation","validationCode":"Configuration conf = job.getConfiguration();\nif (!conf.getBoolean(\"fs.s3a.multipart.uploads.enabled\", true)) {\n  throw new IOException(\"S3A committers require fs.s3a.multipart.uploads.enabled=true\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate the multipart flag at job submission, not at task time","Config-lint clusters for fs.s3a.multipart.uploads.enabled=false when committers are in use","When multipart must be disabled, also unbind the S3A committer factory for that path"],"tags":["aws","s3a","committer","multipart-upload","configuration"],"backgroundTag":"committer-misconfiguration","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}