apache/hadoop · error · InvalidJobConfException

Native output collector doesn't support compression codec {}

Error message

Native output collector doesn't support compression codec {}

What it means

When map output compression is enabled (mapreduce.map.output.compress), the native collector can only spill through codecs its C++ side implements (checked via NativeRuntime.supportsCompressionCodec over the codec's class-name bytes - practically DefaultCodec/zlib and GzipCodec). Any other codec (Snappy, LZ4, Bzip2, Zstd, custom) yields InvalidJobConfException naming the unsupported codec.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java:137

        String message = "Native output collector doesn't support this key, " +
          "this key is not comparable in native: " + keyCls.getName();
        LOG.error(message);
        throw new InvalidJobConfException(message);
      }
    } catch (final IOException e) {
      String message = "Cannot find serializer for " + keyCls.getName();
      LOG.error(message);
      throw new IOException(message);
    }

    final boolean ret = NativeRuntime.isNativeLibraryLoaded();
    if (ret) {
      if (job.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false)) {
        String codec = job.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);
        if (!NativeRuntime.supportsCompressionCodec(codec.getBytes(StandardCharsets.UTF_8))) {
          String message = "Native output collector doesn't support compression codec " + codec;
          LOG.error(message);
          throw new InvalidJobConfException(message);
        }
      }
      NativeRuntime.configure(job);

      final long updateInterval = job.getLong(Constants.NATIVE_STATUS_UPDATE_INTERVAL,
          Constants.NATIVE_STATUS_UPDATE_INTERVAL_DEFVAL);
      updater = new StatusReportChecker(context.getReporter(), updateInterval);
      updater.start();

    } else {
      String message = "NativeRuntime cannot be loaded, please check that " +
        "libnativetask.so is in hadoop library dir";
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    this.handler = null;
    try {

View on GitHub (pinned to 2add963021)

Solutions

1. Set mapreduce.map.output.compress.codec to org.apache.hadoop.io.compress.DefaultCodec or GzipCodec, which the native runtime supports
2. Or turn off map-output compression (mapreduce.map.output.compress=false) for this job
3. Or disable the native collector (unset mapreduce.job.map.output.collector.class) if the faster codec matters more than the native collector

Example fix

# before
<property><name>mapreduce.map.output.compress</name><value>true</value></property>
<property><name>mapreduce.map.output.compress.codec</name><value>org.apache.hadoop.io.compress.SnappyCodec</value></property>

# after
<property><name>mapreduce.map.output.compress.codec</name><value>org.apache.hadoop.io.compress.DefaultCodec</value></property>

if (NativeRuntime.isNativeLibraryLoaded()
    && jobConf.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false)) {
  String codec = jobConf.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);
  if (codec != null && !NativeRuntime.supportsCompressionCodec(codec.getBytes(StandardCharsets.UTF_8))) {
    jobConf.set(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC,
        org.apache.hadoop.io.compress.DefaultCodec.class.getName());
  }
}

try {
  collector.init(context);
} catch (InvalidJobConfException e) {
  if (e.getMessage() != null && e.getMessage().contains("compression codec")) {
    // set codec to DefaultCodec/GzipCodec or disable map-output compression
  }
  throw e;
}

When it happens

Trigger: mapreduce.map.output.compress=true with mapreduce.map.output.compress.codec set to e.g. SnappyCodec/Lz4Codec/BZip2Codec while mapreduce.job.map.output.collector.class = NativeMapOutputCollectorDelegator; init checks the codec against the native registry and fails.

Common situations: Throughput-tuning presets that enable Snappy/LZ4 map-output compression combined with native-collector enabling for shuffle-heavy jobs; upgrading pipelines where the codec default changed site-wide.

Related errors

• Compression codec {} was not found.
• SequenceFileAsBinaryOutputFormat doesn't support Record Comp
• There is no reducer, no need to use native output collector
• Native output collector doesn't support customized java comp
• Native-Task doesn't support sort class {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6b2706746ec8a0d1. Report an issue: GitHub.