apache/hadoop · error · PathIOException

Filesystem is configured to use unknown S3Guard store " + cl

Error message

Filesystem is configured to use unknown S3Guard store " + classname + " origin " + origin

What it means

Companion to the DynamoDB rejection in S3Guard.checkNoS3Guard(): when fs.s3a.metadatastore.impl is set to any value that is neither empty, the NullMetadataStore, the LocalMetadataStore, nor DynamoDBMetadataStore, S3A initialization throws PathIOException('unknown S3Guard store <classname> origin <source>'). Post-3.4 S3A recognizes no metadata store implementations at all, so a custom/typo'd class name is fatal.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java:115

      LOG.warn("Ignoring S3Guard store option of {} -no longer needed or supported. "
              + "Origin {}",
          S3GUARD_METASTORE_LOCAL, origin);
      break;
    case S3GUARD_METASTORE_DYNAMO:
      // this is the dangerous one, as it is a sign that a config is in use where
      // older releases will use DDB for listing metadata, yet this
      // client will not update it.
      final String message = String.format("S3Guard is no longer needed/supported,"
              + " yet %s is configured to use DynamoDB as the S3Guard metadata store."
              + " This is no longer needed or supported. " +
              "Origin of setting is %s",
          fsPath, origin);
      LOG.error(message);
      throw new PathIOException(fsPath, message);

    default:
      // an unknown store entirely.
      throw new PathIOException(fsPath,
          "Filesystem is configured to use unknown S3Guard store " + classname
              + " origin " + origin);
    }

    // an option was set, but it was harmless
    return true;
  }


  /**
   * Get the authoritative paths of a filesystem.
   *
   * @param uri FS URI
   * @param conf configuration
   * @param qualifyToDir a qualification operation
   * @return list of URIs valid for this FS.
   */
  @VisibleForTesting

View on GitHub (pinned to 2add963021)

Solutions

1. Unset fs.s3a.metadatastore.impl entirely (S3Guard is removed; no metadata store is used).
2. Use the 'origin' in the message to find and edit the exact config file (or programmatic source) supplying the bad class name.
3. Remove remaining fs.s3a.s3guard.* / fs.s3a.metadatastore.* options to avoid the DynamoDB sibling failure.
4. Sweep configs cluster-wide: grep -r 'metadatastore' across core-site.xml and job submission templates.

Example fix

<!-- before -->
<property>
  <name>fs.s3a.metadatastore.impl</name>
  <value>com.myco.s3.MyMetadataStore</value>
</property>

<!-- after: property removed; no metadata store on Hadoop 3.4+ -->

String store = conf.getTrimmed("fs.s3a.metadatastore.impl", "");
if (!store.isEmpty()) {
  // only Null/Local/DynamoDB names are even recognized (all S3Guard-era);
  // on 3.4+ just unset everything
  conf.unset("fs.s3a.metadatastore.impl");
}

try {
  FileSystem fs = path.getFileSystem(conf);
} catch (PathIOException e) {
  if (e.getMessage().contains("unknown S3Guard store")) {
    // remove fs.s3a.metadatastore.impl from the origin file, retry
  } else { throw e; }
}

When it happens

Trigger: fs.s3a.metadatastore.impl set to a typo, a custom third-party MetadataStore class (e.g. from an old internal fork), or a class name copied from S3Guard-era docs; initialization of any s3a:// filesystem then fails before the bucket is touched.

Common situations: Leftover or hand-edited S3Guard config after upgrading to Hadoop 3.4+; forks that once shipped custom metadata stores; property inheritance from a parent cluster's core-site.xml (origin string in the message points at the file).

Related errors

• S3Guard is no longer needed/supported, yet %s is configured
• Invalid stream type: \"" + s + "\"
• Cannot find password option {key}
• Unable to create OutputStream with the given multipart uploa
• SSE-C is enabled but no encryption key was declared in fs.s3

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0106de73a6063e46. Report an issue: GitHub.