apache/hadoop · error · RuntimeException

Repetitive schemas in EC policy configuration file: {schemaI

Error message

Repetitive schemas in EC policy configuration file: {schemaId}

What it means

ECPolicyLoader.loadSchemas() rejects duplicate schemas: before inserting each <schema id="..."> it checks schemas.containsValue(schema), i.e. equality of the parsed ECSchema (codec, k/m, options), not the id. Two <schema> entries that produce equal ECSchema objects — even under different ids — make the second one throw this RuntimeException. The message reports the id of the offending (second) entry.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java:173

   * Load schemas from root element in the XML configuration file.
   * @param root root element
   * @return EC schema map
   */
  private Map<String, ECSchema> loadSchemas(Element root) {
    NodeList elements = root.getElementsByTagName("schemas")
        .item(0).getChildNodes();
    Map<String, ECSchema> schemas = new HashMap<String, ECSchema>();
    for (int i = 0; i < elements.getLength(); i++) {
      Node node = elements.item(i);
      if (node instanceof Element) {
        Element element = (Element) node;
        if ("schema".equals(element.getTagName())) {
          String schemaId = element.getAttribute("id");
          ECSchema schema = loadSchema(element);
          if (!schemas.containsValue(schema)) {
            schemas.put(schemaId, schema);
          } else {
            throw new RuntimeException("Repetitive schemas in EC policy"
                + " configuration file: " + schemaId);
          }
        } else {
          throw new RuntimeException("Bad element in EC policy"
              + " configuration file: " + element.getTagName());
        }
      }
    }

    return schemas;
  }

  /**
   * Load EC policies from root element in the XML configuration file.
   * @param root root element
   * @param schemas schema map
   * @return EC policy list
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Delete the duplicate <schema> entry and keep one canonical definition per distinct codec/k/m combination
2. Give each remaining schema a unique id and reference that id from every <policy><schema> element
3. If two policies need the same schema, share one <schema> definition — ids are referenced, not redefined

Example fix

<!-- before -->
<schema id="rs-6-3">
  <k>6</k><m>3</m><codec>rs</codec>
</schema>
<schema id="rs-6-3-alt">
  <k>6</k><m>3</m><codec>rs</codec>
</schema>

<!-- after -->
<schema id="rs-6-3">
  <k>6</k><m>3</m><codec>rs</codec>
</schema>
<!-- policies both reference <schema>rs-6-3</schema> -->

Set<String> seen = new HashSet<>();
for (Element s : schemaElements(doc)) {
  String key = s.getAttribute("codec") + ":" + textOf(s, "k") + ":" + textOf(s, "m");
  if (!seen.add(key)) {
    throw new IllegalArgumentException("duplicate schema params (codec:k:m) at id " + s.getAttribute("id"));
  }
}

try { new ECPolicyLoader().loadPolicy(path); }
catch (RuntimeException e) {
  LOG.error("EC policy file {} rejected: {}", path, e.getMessage());
}

When it happens

Trigger: `hdfs ec -loadPolicy` where the <schemas> block defines the same erasure schema twice: identical id and parameters, or different ids (e.g. 'rs-6-3' and 'rs-6-3-copy') with the same codec and k/m values.

Common situations: Copy-pasting a schema block to tweak a name but leaving k/m/codec unchanged, then referencing both ids from policies; merging policy files that each define RS(6,3).

Related errors

• Failed to load EC policy file: {policyFilePath}
• Bad EC policy configuration file: top-level element not <con
• Bad EC policy configuration file: no <policies> element
• Bad EC policy configuration file: no <schemas> element
• The parse failed because of bad layoutversion value

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9d5af49b79c3d9a3. Report an issue: GitHub.