apache/hadoop · error · IOException

Invalid define for {defineName}

Error message

Invalid define for {defineName}

What it means

Error "Invalid define for {defineName}" thrown in apache/hadoop.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/CompositeInputFormat.java:114

      Parser.WNode.addIdentifier("tbl", WrappedRecordReader.class);
    } catch (NoSuchMethodException e) {
      throw new RuntimeException("FATAL: Failed to init defaults", e);
    }
  }

  /**
   * Inform the parser of user-defined types.
   */
  private void addUserIdentifiers(Configuration conf) throws IOException {
    Pattern x = Pattern.compile("^mapreduce\\.join\\.define\\.(\\w+)$");
    for (Map.Entry<String,String> kv : conf) {
      Matcher m = x.matcher(kv.getKey());
      if (m.matches()) {
        try {
          Parser.CNode.addIdentifier(m.group(1),
              conf.getClass(m.group(0), null, ComposableRecordReader.class));
        } catch (NoSuchMethodException e) {
          throw new IOException("Invalid define for " + m.group(1), e);
        }
      }
    }
  }

  /**
   * Build a CompositeInputSplit from the child InputFormats by assigning the
   * ith split from each child to the ith composite split.
   */
  @SuppressWarnings("unchecked")
  public List<InputSplit> getSplits(JobContext job) 
      throws IOException, InterruptedException {
    setFormat(job.getConfiguration());
    job.getConfiguration().setLong("mapreduce.input.fileinputformat.split.minsize", Long.MAX_VALUE);
    return root.getSplits(job);
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Verify that every identifier used in the join expression was declared before use in the composite input format string.
2. Fix the mapred.join.expr / join expression passed to CompositeInputFormat so each {defineName} token refers to a previously defined stream or map.

Example fix

CompositeInputFormat.setFormat(job, "outer(KeyValueTextInputFormat.class,\"in/a\",inner(SequenceFileInputFormat.class,\"in/b\"))");

When it happens

Trigger: Thrown by CompositeInputFormat.setFormat()/addDefaults() when a join expression identifier is defined twice with conflicting types (e.g. the same name used once as a stream and once as a record reader class). Give each stream in the join expression a unique identifier.

Common situations: Typo in a stream name in the join expression; referencing a table alias that was never defined in the expression.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/51073c52c7481ae2. Report an issue: GitHub.