{"record":{"id":"b9655e76263edb54","repo":"apache/hadoop","slug":"e-normal-fs","errorCode":"E_NORMAL_FS","errorMessage":"Filesystem does not have support for 'magic' committer enabled in configuration option fs.s3a.committer.magic.enabled","messagePattern":"Filesystem does not have support for 'magic' committer enabled in configuration option fs\\.s3a\\.committer\\.magic\\.enabled","errorType":"exception","errorClass":"PathCommitException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitUtils.java","lineNumber":74,"sourceCode":"    verifyIsMagicCommitFS(fs);\n    if (!fs.isMagicCommitPath(path)) {\n      throw new PathCommitException(path, E_BAD_PATH);\n    }\n  }\n\n  /**\n   * Verify that an S3A FS instance is a magic commit FS.\n   * @param fs filesystem\n   * @throws PathCommitException if the FS isn't a magic commit FS.\n   */\n  public static void verifyIsMagicCommitFS(S3AFileSystem fs)\n      throws PathCommitException {\n    if (!fs.isMagicCommitEnabled()) {\n      // dump out details to console for support diagnostics\n      String fsUri = fs.getUri().toString();\n      LOG.error(\"{}: {}:\\n{}\", E_NORMAL_FS, fsUri, fs);\n      // then fail\n      throw new PathCommitException(fsUri, E_NORMAL_FS);\n    }\n  }\n\n  /**\n   * Verify that an FS is an S3A FS.\n   * @param fs filesystem\n   * @param path path to to use in exception\n   * @return the typecast FS.\n   * @throws PathCommitException if the FS is not an S3A FS.\n   */\n  public static S3AFileSystem verifyIsS3AFS(FileSystem fs, Path path)\n      throws PathCommitException {\n    if (!(fs instanceof S3AFileSystem)) {\n      throw new PathCommitException(path, E_WRONG_FS);\n    }\n    return (S3AFileSystem) fs;\n  }\n","sourceCodeStart":56,"sourceCodeEnd":92,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitUtils.java#L56-L92","documentation":"Thrown by CommitUtils.verifyIsMagicCommitFS when the S3AFileSystem instance was created without magic-commit support (fs.isMagicCommitEnabled() is false). Before throwing, the code LOG.error's the error code, the filesystem URI and the filesystem's own toString() diagnostics, so the log already contains the support details. It means the 'fs.s3a.committer.magic.enabled' option was not true in the configuration used when the filesystem instance was constructed.","triggerScenarios":"Any code path that calls verifyIsMagicCommitFS or verifyIsMagicCommitPath against an S3AFileSystem whose configuration lacks fs.s3a.committer.magic.enabled=true, typically when initializing or using the magic committer.","commonSituations":"fs.s3a.committer.name=magic is set in the job config but fs.s3a.committer.magic.enabled=true is missing; the option is set only in the client's core-site.xml while the FileSystem is instantiated on nodes with a different core-site.xml; a typo in the option name; a per-bucket override (fs.s3a.bucket.<bucket>.committer.magic.enabled) is set to false and wins over the base key.","solutions":["Set fs.s3a.committer.magic.enabled=true together with fs.s3a.committer.name=magic in core-site.xml on every node that will create the S3AFileSystem (or pass both in the job Configuration)","Check the LOG.error line emitted just before the exception: it prints the FS URI and full filesystem state showing which config was actually in effect","Verify no per-bucket override (fs.s3a.bucket.<bucket>.*) disables the feature","If magic commit is not intended, stop calling the magic commit APIs and use another committer"],"exampleFix":"<!-- before -->\n<property><name>fs.s3a.committer.name</name><value>magic</value></property>\n\n<!-- after -->\n<property><name>fs.s3a.committer.name</name><value>magic</value></property>\n<property><name>fs.s3a.committer.magic.enabled</name><value>true</value></property>","handlingStrategy":"validation","validationCode":"FileSystem fs = path.getFileSystem(conf);\nif (!(fs instanceof S3AFileSystem)\n    || !((S3AFileSystem) fs).isMagicCommitEnabled()) {\n  throw new IOException(\"Magic commits not enabled for \" + path\n      + \" -- set fs.s3a.committer.magic.enabled=true\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  CommitUtils.verifyIsMagicCommitFS(s3aFs);\n} catch (PathCommitException e) {\n  // read the LOG.error dump of the FS state emitted just before the throw\n  throw new IOException(\"FS lacks magic commit support; fix core-site.xml\", e);\n}","preventionTips":["Ship fs.s3a.committer.magic.enabled=true and fs.s3a.committer.name=magic together in the same core-site.xml on all nodes","Smoke-test with a tiny job after cluster config changes instead of discovering this on a production job","Prefer setting the pair in the job Configuration so it cannot be missed by node-level config drift"],"tags":["s3a","magic-committer","configuration","hadoop"],"backgroundTag":"missing-configuration-value","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}