{"record":{"id":"f0d328f942b7fab3","repo":"apache/hadoop","slug":"e-wrong-fs","errorCode":"E_WRONG_FS","errorMessage":"Output path is not on an S3A Filesystem","messagePattern":"Output path is not on an S3A Filesystem","errorType":"exception","errorClass":"PathCommitException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitUtils.java","lineNumber":88,"sourceCode":"      // dump out details to console for support diagnostics\n      String fsUri = fs.getUri().toString();\n      LOG.error(\"{}: {}:\\n{}\", E_NORMAL_FS, fsUri, fs);\n      // then fail\n      throw new PathCommitException(fsUri, E_NORMAL_FS);\n    }\n  }\n\n  /**\n   * Verify that an FS is an S3A FS.\n   * @param fs filesystem\n   * @param path path to to use in exception\n   * @return the typecast FS.\n   * @throws PathCommitException if the FS is not an S3A FS.\n   */\n  public static S3AFileSystem verifyIsS3AFS(FileSystem fs, Path path)\n      throws PathCommitException {\n    if (!(fs instanceof S3AFileSystem)) {\n      throw new PathCommitException(path, E_WRONG_FS);\n    }\n    return (S3AFileSystem) fs;\n  }\n\n  /**\n   * Get the S3A FS of a path.\n   * @param path path to examine\n   * @param conf config\n   * @param magicCommitRequired is magic complete required in the FS?\n   * @return the filesystem\n   * @throws PathCommitException output path isn't to an S3A FS instance, or\n   * if {@code magicCommitRequired} is set, if doesn't support these commits.\n   * @throws IOException failure to instantiate the FS\n   */\n  public static S3AFileSystem getS3AFileSystem(Path path,\n      Configuration conf,\n      boolean magicCommitRequired)\n      throws PathCommitException, IOException {","sourceCodeStart":70,"sourceCodeEnd":106,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitUtils.java#L70-L106","documentation":"Thrown by CommitUtils.verifyIsS3AFS when the filesystem bound to an output path is not an S3AFileSystem instance. S3A committers can only work against s3a:// URLs because they rely on S3A-specific commit machinery; any other filesystem implementation (s3n, s3b, hdfs, local) fails this instanceof check. The method exists to fail fast before committer setup touches S3A-only APIs.","triggerScenarios":"Calling verifyIsS3AFS or getOutputFS (which delegates to it) when mapreduce.output.fileoutputformat.outputdir (or Spark's spark.sql.sources.output.pathCommitterClass path) uses a non-s3a scheme, or when FileSystem.get returned a cached instance for an alias like s3://.","commonSituations":"The job's output path is written as s3n://bucket/out or s3b://bucket/out (legacy connectors); output goes to hdfs:// or file:// while the code assumes S3; the output path has no scheme at all and resolves against the default filesystem (fs.defaultFS).","solutions":["Change the output path to use the s3a:// scheme, e.g. s3a://bucket/output","If the path is relative or scheme-less, qualify it against an explicit s3a:// base path before passing it to the committer","Confirm no stray fs.s3a.impl aliasing maps s3a:// to another FileSystem class in core-site.xml","If the output is genuinely not on S3, use the standard FileOutputCommitter instead of an S3A committer"],"exampleFix":"// before\nJobConf conf = new JobConf();\nFileOutputFormat.setOutputPath(job, new Path(\"s3n://bucket/output\"));\n\n// after\nFileOutputFormat.setOutputPath(job, new Path(\"s3a://bucket/output\"));","handlingStrategy":"type-guard","validationCode":"Path qualified = path.makeQualified(fs.getUri(), fs.getWorkingDirectory());\nif (!\"s3a\".equals(qualified.toUri().getScheme())) {\n  throw new IOException(\"Output path must use s3a://, got: \" + qualified);\n}","typeGuard":"public static boolean isS3AFileSystem(FileSystem fs) {\n  return fs instanceof S3AFileSystem;\n}","tryCatchPattern":"try {\n  S3AFileSystem s3a = CommitUtils.verifyIsS3AFS(fs, outputPath);\n} catch (PathCommitException e) {\n  // scheme problem: rewrite the output path with the s3a:// scheme and retry once\n  LOG.error(\"Output path not on S3A: {}\", outputPath, e);\n}","preventionTips":["Standardize all bucket URLs in pipelines on the s3a:// scheme; grep configs for s3n:// and s3b:// leftovers","Always makeQualified() user-supplied output paths so the effective scheme is visible before job submission","Fail fast at job-submission time by checking the output scheme in a driver preflight"],"tags":["s3a","filesystem-scheme","committer","hadoop"],"backgroundTag":"wrong-filesystem-scheme","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}