apache/hadoop · error · IllegalArgumentException

key class or comparator option must be set

Error message

key class or comparator option must be set

What it means

Thrown by the MapFile.Writer(Configuration, Path, Option...) constructor when KeyClassOption and ComparatorOption are either both absent or both present — the condition ((keyClassOption == null) == (comparatorOption == null)) enforces exactly one. MapFile needs to know the key type either directly (keyClass option) or indirectly (a comparator whose getKeyClass() supplies it); giving neither leaves the sort order undefined, and giving both is contradictory.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java:322

        CompressionCodec codec) {
      return SequenceFile.Writer.compression(type, codec);
    }

    public static SequenceFile.Writer.Option progressable(Progressable value) {
      return SequenceFile.Writer.progressable(value);
    }

    @SuppressWarnings("unchecked")
    public Writer(Configuration conf, 
                  Path dirName,
                  SequenceFile.Writer.Option... opts
                  ) throws IOException {
      KeyClassOption keyClassOption = 
        Options.getOption(KeyClassOption.class, opts);
      ComparatorOption comparatorOption =
        Options.getOption(ComparatorOption.class, opts);
      if ((keyClassOption == null) == (comparatorOption == null)) {
        throw new IllegalArgumentException("key class or comparator option "
                                           + "must be set");
      }
      this.indexInterval = conf.getInt(INDEX_INTERVAL, this.indexInterval);

      Class<? extends WritableComparable> keyClass;
      if (keyClassOption == null) {
        this.comparator = comparatorOption.getValue();
        keyClass = comparator.getKeyClass();
      } else {
        keyClass= 
          (Class<? extends WritableComparable>) keyClassOption.getValue();
        this.comparator = WritableComparator.get(keyClass, conf);
      }
      this.lastKey = comparator.newKey();
      FileSystem fs = dirName.getFileSystem(conf);

      if (!fs.mkdirs(dirName)) {
        throw new IOException("Mkdirs failed to create directory " + dirName);

View on GitHub (pinned to 2add963021)

Solutions

1. Pass exactly one of the two: the common fix is adding Writer.keyClass(YourKey.class) alongside Writer.valueClass(...).
2. If you need a custom ordering, keep only Writer.comparator(cmp) — the key class is then taken from cmp.getKeyClass().
3. Remove whichever of the two options you duplicated before retrying construction.

Example fix

// before: no key class or comparator
new MapFile.Writer(conf, new Path("out"),
    MapFile.Writer.valueClass(Text.class)); // throws

// after: exactly one way to determine the key class
new MapFile.Writer(conf, new Path("out"),
    MapFile.Writer.keyClass(Text.class),
    MapFile.Writer.valueClass(Text.class));

boolean hasKey = Arrays.stream(opts).anyMatch(o -> o instanceof MapFile.Writer.KeyClassOption);
boolean hasCmp = Arrays.stream(opts).anyMatch(o -> o instanceof MapFile.Writer.ComparatorOption);
if (hasKey == hasCmp) {
  throw new IllegalArgumentException("Pass exactly one of keyClass or comparator");
}
new MapFile.Writer(conf, dir, opts);

try {
  writer = new MapFile.Writer(conf, dir, opts);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("key class or comparator")) {
    throw new IllegalStateException("MapFile.Writer misconfigured", e);
  }
  throw e;
}

When it happens

Trigger: new MapFile.Writer(conf, dir) with only Writer.valueClass(...) or file(...) options; passing both MapFile.Writer.keyClass(Text.class) and MapFile.Writer.comparator(comparator) at once. Note SetFile.Writer and subclass constructors route through this check too.

Common situations: Porting old MapFile.Writer code that used the deprecated String keyClass/valueClass constructor to the Option-based API and dropping the key class argument; copy-pasting an example that included a comparator while your code already sets the key class.

Related errors

• Mkdirs failed to create directory {dirName}
• key out of order: {key} after {lastKey}
• key out of order: {k} after {lastKey}
• Could not rename {oldDir} to {newDir}
• Input files cannot be merged as they have different Key and

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/dfc705276bdaa7f5. Report an issue: GitHub.