{"record":{"id":"6b2706746ec8a0d1","repo":"apache/hadoop","slug":"native-output-collector-doesn-t-support-compressio","errorCode":null,"errorMessage":"Native output collector doesn't support compression codec {}","messagePattern":"Native output collector doesn't support compression codec (.+?)","errorType":"exception","errorClass":"InvalidJobConfException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java","lineNumber":137,"sourceCode":"        String message = \"Native output collector doesn't support this key, \" +\n          \"this key is not comparable in native: \" + keyCls.getName();\n        LOG.error(message);\n        throw new InvalidJobConfException(message);\n      }\n    } catch (final IOException e) {\n      String message = \"Cannot find serializer for \" + keyCls.getName();\n      LOG.error(message);\n      throw new IOException(message);\n    }\n\n    final boolean ret = NativeRuntime.isNativeLibraryLoaded();\n    if (ret) {\n      if (job.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false)) {\n        String codec = job.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);\n        if (!NativeRuntime.supportsCompressionCodec(codec.getBytes(StandardCharsets.UTF_8))) {\n          String message = \"Native output collector doesn't support compression codec \" + codec;\n          LOG.error(message);\n          throw new InvalidJobConfException(message);\n        }\n      }\n      NativeRuntime.configure(job);\n\n      final long updateInterval = job.getLong(Constants.NATIVE_STATUS_UPDATE_INTERVAL,\n          Constants.NATIVE_STATUS_UPDATE_INTERVAL_DEFVAL);\n      updater = new StatusReportChecker(context.getReporter(), updateInterval);\n      updater.start();\n\n    } else {\n      String message = \"NativeRuntime cannot be loaded, please check that \" +\n        \"libnativetask.so is in hadoop library dir\";\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n    this.handler = null;\n    try {","sourceCodeStart":119,"sourceCodeEnd":155,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java#L119-L155","documentation":"When map output compression is enabled (mapreduce.map.output.compress), the native collector can only spill through codecs its C++ side implements (checked via NativeRuntime.supportsCompressionCodec over the codec's class-name bytes - practically DefaultCodec/zlib and GzipCodec). Any other codec (Snappy, LZ4, Bzip2, Zstd, custom) yields InvalidJobConfException naming the unsupported codec.","triggerScenarios":"mapreduce.map.output.compress=true with mapreduce.map.output.compress.codec set to e.g. SnappyCodec/Lz4Codec/BZip2Codec while mapreduce.job.map.output.collector.class = NativeMapOutputCollectorDelegator; init checks the codec against the native registry and fails.","commonSituations":"Throughput-tuning presets that enable Snappy/LZ4 map-output compression combined with native-collector enabling for shuffle-heavy jobs; upgrading pipelines where the codec default changed site-wide.","solutions":["Set mapreduce.map.output.compress.codec to org.apache.hadoop.io.compress.DefaultCodec or GzipCodec, which the native runtime supports","Or turn off map-output compression (mapreduce.map.output.compress=false) for this job","Or disable the native collector (unset mapreduce.job.map.output.collector.class) if the faster codec matters more than the native collector"],"exampleFix":"# before\n<property><name>mapreduce.map.output.compress</name><value>true</value></property>\n<property><name>mapreduce.map.output.compress.codec</name><value>org.apache.hadoop.io.compress.SnappyCodec</value></property>\n\n# after\n<property><name>mapreduce.map.output.compress.codec</name><value>org.apache.hadoop.io.compress.DefaultCodec</value></property>","handlingStrategy":"validation","validationCode":"if (NativeRuntime.isNativeLibraryLoaded()\n    && jobConf.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false)) {\n  String codec = jobConf.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);\n  if (codec != null && !NativeRuntime.supportsCompressionCodec(codec.getBytes(StandardCharsets.UTF_8))) {\n    jobConf.set(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC,\n        org.apache.hadoop.io.compress.DefaultCodec.class.getName());\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  collector.init(context);\n} catch (InvalidJobConfException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"compression codec\")) {\n    // set codec to DefaultCodec/GzipCodec or disable map-output compression\n  }\n  throw e;\n}","preventionTips":["Pair nativetask jobs with DefaultCodec or GzipCodec for map-output compression","Block Snappy/LZ4/Bzip2 map-output codecs in native-enabled job templates","Prefer job-level codec overrides over site-wide defaults when clusters mix native and Java collectors"],"tags":["nativetask","compression","codec","job-configuration"],"backgroundTag":"unsupported-compression-codec","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}