apache/hadoop · error · PathIOException

S3Guard is no longer needed/supported, yet %s is configured

Error message

S3Guard is no longer needed/supported, yet %s is configured to use DynamoDB as the S3Guard metadata store. This is no longer needed or supported. Origin of setting is %s

What it means

S3Guard (the DynamoDB metadata store for S3A consistency) was removed in Hadoop 3.4.0. S3Guard.checkNoS3Guard() runs at filesystem initialization and, when fs.s3a.metadatastore.impl is set to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore, throws PathIOException with this message (including the config origin that supplied the setting). It hard-fails deliberately: an old config would mean older clients maintain DynamoDB metadata this client silently ignores, diverging listings.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java:111

    case S3GUARD_METASTORE_LOCAL:
      // used in some libraries (e.g. hboss) to force a consistent s3 in a test
      // run.
      // print a message and continue
      LOG.warn("Ignoring S3Guard store option of {} -no longer needed or supported. "
              + "Origin {}",
          S3GUARD_METASTORE_LOCAL, origin);
      break;
    case S3GUARD_METASTORE_DYNAMO:
      // this is the dangerous one, as it is a sign that a config is in use where
      // older releases will use DDB for listing metadata, yet this
      // client will not update it.
      final String message = String.format("S3Guard is no longer needed/supported,"
              + " yet %s is configured to use DynamoDB as the S3Guard metadata store."
              + " This is no longer needed or supported. " +
              "Origin of setting is %s",
          fsPath, origin);
      LOG.error(message);
      throw new PathIOException(fsPath, message);

    default:
      // an unknown store entirely.
      throw new PathIOException(fsPath,
          "Filesystem is configured to use unknown S3Guard store " + classname
              + " origin " + origin);
    }

    // an option was set, but it was harmless
    return true;
  }


  /**
   * Get the authoritative paths of a filesystem.
   *
   * @param uri FS URI
   * @param conf configuration

View on GitHub (pinned to 2add963021)

Solutions

1. Remove fs.s3a.metadatastore.impl (or leave it empty/unset) - S3 list consistency is now provided by S3 itself.
2. Delete all fs.s3a.s3guard.* properties (ddb table name, region, capacity, etc.) from core-site.xml, job configs, and policies.
3. Use the origin printed in the message to locate which file (or 'programmatically') still supplies the setting, then fix it there.
4. If you still run mixed-version clusters, isolate configs per version so 3.3.x nodes keep their S3Guard settings while 3.4+ nodes drop them.

Example fix

<!-- before (Hadoop 3.2/3.3 era core-site.xml) -->
<property>
  <name>fs.s3a.metadatastore.impl</name>
  <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
</property>

<!-- after: delete the property entirely; S3 is now strongly consistent -->
<!-- also remove fs.s3a.s3guard.ddb.* and related options -->

// fail fast in job setup, with the exact origin
String store = conf.getTrimmed("fs.s3a.metadatastore.impl", "");
if (store.contains("DynamoDBMetadataStore")) {
  String[] src = conf.getPropertySources("fs.s3a.metadatastore.impl");
  throw new IOException("Remove S3Guard setting from " + (src == null ? "?" : src[0]));
}
conf.unset("fs.s3a.metadatastore.impl");

try {
  FileSystem fs = path.getFileSystem(conf);
} catch (PathIOException e) {
  if (e.getMessage().contains("S3Guard is no longer needed")) {
    // unset the property named by 'Origin of setting', then retry init once
  } else { throw e; }
}

When it happens

Trigger: Initializing any s3a:// filesystem on Hadoop 3.4+ while fs.s3a.metadatastore.impl=org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore in core-site.xml, a job's configuration, or a shared config layer; the message's 'Origin of setting' names the exact source file.

Common situations: Upgrading a cluster or job jars from Hadoop 3.2/3.3 to 3.4+ without cleansing S3Guard properties (fs.s3a.s3guard.*, fs.s3a.metadatastore.impl=dynamodb) that were required for consistent listings; vendor distributions carrying legacy core-site.xml; EMR-to-self-managed migrations carrying old configs.

Related errors

• Filesystem is configured to use unknown S3Guard store " + cl
• Invalid stream type: \"" + s + "\"
• Cannot find password option {key}
• Unable to create OutputStream with the given multipart uploa
• SSE-C is enabled but no encryption key was declared in fs.s3

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0a12e5851f0ae0d0. Report an issue: GitHub.