{"record":{"id":"0a12e5851f0ae0d0","repo":"apache/hadoop","slug":"s3guard-is-no-longer-needed-supported-yet-s-is-c","errorCode":null,"errorMessage":"S3Guard is no longer needed/supported, yet %s is configured to use DynamoDB as the S3Guard metadata store. This is no longer needed or supported. Origin of setting is %s","messagePattern":"S3Guard is no longer needed/supported, yet (.+?) is configured to use DynamoDB as the S3Guard metadata store\\. This is no longer needed or supported\\. Origin of setting is (.+?)","errorType":"exception","errorClass":"PathIOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java","lineNumber":111,"sourceCode":"    case S3GUARD_METASTORE_LOCAL:\n      // used in some libraries (e.g. hboss) to force a consistent s3 in a test\n      // run.\n      // print a message and continue\n      LOG.warn(\"Ignoring S3Guard store option of {} -no longer needed or supported. \"\n              + \"Origin {}\",\n          S3GUARD_METASTORE_LOCAL, origin);\n      break;\n    case S3GUARD_METASTORE_DYNAMO:\n      // this is the dangerous one, as it is a sign that a config is in use where\n      // older releases will use DDB for listing metadata, yet this\n      // client will not update it.\n      final String message = String.format(\"S3Guard is no longer needed/supported,\"\n              + \" yet %s is configured to use DynamoDB as the S3Guard metadata store.\"\n              + \" This is no longer needed or supported. \" +\n              \"Origin of setting is %s\",\n          fsPath, origin);\n      LOG.error(message);\n      throw new PathIOException(fsPath, message);\n\n    default:\n      // an unknown store entirely.\n      throw new PathIOException(fsPath,\n          \"Filesystem is configured to use unknown S3Guard store \" + classname\n              + \" origin \" + origin);\n    }\n\n    // an option was set, but it was harmless\n    return true;\n  }\n\n\n  /**\n   * Get the authoritative paths of a filesystem.\n   *\n   * @param uri FS URI\n   * @param conf configuration","sourceCodeStart":93,"sourceCodeEnd":129,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java#L93-L129","documentation":"S3Guard (the DynamoDB metadata store for S3A consistency) was removed in Hadoop 3.4.0. S3Guard.checkNoS3Guard() runs at filesystem initialization and, when fs.s3a.metadatastore.impl is set to org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore, throws PathIOException with this message (including the config origin that supplied the setting). It hard-fails deliberately: an old config would mean older clients maintain DynamoDB metadata this client silently ignores, diverging listings.","triggerScenarios":"Initializing any s3a:// filesystem on Hadoop 3.4+ while fs.s3a.metadatastore.impl=org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore in core-site.xml, a job's configuration, or a shared config layer; the message's 'Origin of setting' names the exact source file.","commonSituations":"Upgrading a cluster or job jars from Hadoop 3.2/3.3 to 3.4+ without cleansing S3Guard properties (fs.s3a.s3guard.*, fs.s3a.metadatastore.impl=dynamodb) that were required for consistent listings; vendor distributions carrying legacy core-site.xml; EMR-to-self-managed migrations carrying old configs.","solutions":["Remove fs.s3a.metadatastore.impl (or leave it empty/unset) - S3 list consistency is now provided by S3 itself.","Delete all fs.s3a.s3guard.* properties (ddb table name, region, capacity, etc.) from core-site.xml, job configs, and policies.","Use the origin printed in the message to locate which file (or 'programmatically') still supplies the setting, then fix it there.","If you still run mixed-version clusters, isolate configs per version so 3.3.x nodes keep their S3Guard settings while 3.4+ nodes drop them."],"exampleFix":"<!-- before (Hadoop 3.2/3.3 era core-site.xml) -->\n<property>\n  <name>fs.s3a.metadatastore.impl</name>\n  <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>\n</property>\n\n<!-- after: delete the property entirely; S3 is now strongly consistent -->\n<!-- also remove fs.s3a.s3guard.ddb.* and related options -->","handlingStrategy":"validation","validationCode":"// fail fast in job setup, with the exact origin\nString store = conf.getTrimmed(\"fs.s3a.metadatastore.impl\", \"\");\nif (store.contains(\"DynamoDBMetadataStore\")) {\n  String[] src = conf.getPropertySources(\"fs.s3a.metadatastore.impl\");\n  throw new IOException(\"Remove S3Guard setting from \" + (src == null ? \"?\" : src[0]));\n}\nconf.unset(\"fs.s3a.metadatastore.impl\");","typeGuard":null,"tryCatchPattern":"try {\n  FileSystem fs = path.getFileSystem(conf);\n} catch (PathIOException e) {\n  if (e.getMessage().contains(\"S3Guard is no longer needed\")) {\n    // unset the property named by 'Origin of setting', then retry init once\n  } else { throw e; }\n}","preventionTips":["Before a 3.4+ upgrade, grep all configs for 'metadatastore' and 's3guard' and remove them.","Use the message's origin field to fix the offending file, not just the local copy.","Keep per-version config templates so legacy S3Guard settings never reach new clusters."],"tags":["hadoop-aws","s3guard","dynamodb","hadoop-3-4-upgrade","configuration"],"backgroundTag":"removed-feature-config-error","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}