apache/hadoop · error · InvalidJobConfException

Native output collector doesn't support compression codec

Error message

Native output collector doesn't support compression codec {}

What it means

When map output compression is enabled (mapreduce.map.output.compress), the native collector can only spill through codecs its C++ side implements (checked via NativeRuntime.supportsCompressionCodec over the codec's class-name bytes - practically DefaultCodec/zlib and GzipCodec). Any other codec (Snappy, LZ4, Bzip2, Zstd, custom) yields InvalidJobConfException naming the unsupported codec.

Solutions

1. Set mapreduce.map.output.compress.codec to org.apache.hadoop.io.compress.DefaultCodec or GzipCodec, which the native runtime supports
2. Or turn off map-output compression (mapreduce.map.output.compress=false) for this job
3. Or disable the native collector (unset mapreduce.job.map.output.collector.class) if the faster codec matters more than the native collector

Example fix

# before
<property><name>mapreduce.map.output.compress</name><value>true</value></property>
<property><name>mapreduce.map.output.compress.codec</name><value>org.apache.hadoop.io.compress.SnappyCodec</value></property>

# after
<property><name>mapreduce.map.output.compress.codec</name><value>org.apache.hadoop.io.compress.DefaultCodec</value></property>

if (NativeRuntime.isNativeLibraryLoaded()
    && jobConf.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false)) {
  String codec = jobConf.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);
  if (codec != null && !NativeRuntime.supportsCompressionCodec(codec.getBytes(StandardCharsets.UTF_8))) {
    jobConf.set(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC,
        org.apache.hadoop.io.compress.DefaultCodec.class.getName());
  }
}

try {
  collector.init(context);
} catch (InvalidJobConfException e) {
  if (e.getMessage() != null && e.getMessage().contains("compression codec")) {
    // set codec to DefaultCodec/GzipCodec or disable map-output compression
  }
  throw e;
}

When it happens

Trigger: mapreduce.map.output.compress=true with mapreduce.map.output.compress.codec set to e.g. SnappyCodec/Lz4Codec/BZip2Codec while mapreduce.job.map.output.collector.class = NativeMapOutputCollectorDelegator; init checks the codec against the native registry and fails.

Common situations: Throughput-tuning presets that enable Snappy/LZ4 map-output compression combined with native-collector enabling for shuffle-heavy jobs; upgrading pipelines where the codec default changed site-wide.

Related errors

• Cannot find serializer for
• Compression codec was not found.
• Key type not supported. Cannot find serializer for
• Native output collector doesn't support customized java…
• Native output collector doesn't support this key, this key…

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6b2706746ec8a0d1. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java:137

        String message = "Native output collector doesn't support this key, " +
          "this key is not comparable in native: " + keyCls.getName();
        LOG.error(message);
        throw new InvalidJobConfException(message);
      }
    } catch (final IOException e) {
      String message = "Cannot find serializer for " + keyCls.getName();
      LOG.error(message);
      throw new IOException(message);
    }

    final boolean ret = NativeRuntime.isNativeLibraryLoaded();
    if (ret) {
      if (job.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false)) {
        String codec = job.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);
        if (!NativeRuntime.supportsCompressionCodec(codec.getBytes(StandardCharsets.UTF_8))) {
          String message = "Native output collector doesn't support compression codec " + codec;
          LOG.error(message);
          throw new InvalidJobConfException(message);
        }
      }
      NativeRuntime.configure(job);

      final long updateInterval = job.getLong(Constants.NATIVE_STATUS_UPDATE_INTERVAL,
          Constants.NATIVE_STATUS_UPDATE_INTERVAL_DEFVAL);
      updater = new StatusReportChecker(context.getReporter(), updateInterval);
      updater.start();

    } else {
      String message = "NativeRuntime cannot be loaded, please check that " +
        "libnativetask.so is in hadoop library dir";
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    this.handler = null;
    try {

View on GitHub (pinned to 2add963021)