apache/hadoop · error · IllegalArgumentException

Compression codec {} was not found.

Error message

Compression codec {} was not found.

What it means

Thrown as IllegalArgumentException from FileOutputFormat.getOutputCompressorClass (FileOutputFormat.java:140). It reads mapreduce.output.fileoutputformat.compress.codec from the job configuration, loads the class via conf.getClassByName(name), and wraps any ClassNotFoundException in this IllegalArgumentException. The configured codec class name is not resolvable on the classpath of the JVM that calls the method (job client at submission or the task JVM at runtime).

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java:140

   * Get the {@link CompressionCodec} for compressing the job outputs.
   * @param job the {@link Job} to look in
   * @param defaultValue the {@link CompressionCodec} to return if not set
   * @return the {@link CompressionCodec} to be used to compress the 
   *         job outputs
   * @throws IllegalArgumentException if the class was specified, but not found
   */
  public static Class<? extends CompressionCodec> 
  getOutputCompressorClass(JobContext job, 
                       Class<? extends CompressionCodec> defaultValue) {
    Class<? extends CompressionCodec> codecClass = defaultValue;
    Configuration conf = job.getConfiguration();
    String name = conf.get(FileOutputFormat.COMPRESS_CODEC);
    if (name != null) {
      try {
        codecClass =
            conf.getClassByName(name).asSubclass(CompressionCodec.class);
      } catch (ClassNotFoundException e) {
        throw new IllegalArgumentException("Compression codec " + name + 
                                           " was not found.", e);
      }
    }
    return codecClass;
  }
  
  public abstract RecordWriter<K, V> 
     getRecordWriter(TaskAttemptContext job
                     ) throws IOException, InterruptedException;

  public void checkOutputSpecs(JobContext job
                               ) throws FileAlreadyExistsException, IOException{
    // Ensure that the output directory is set and not already there
    Path outDir = getOutputPath(job);
    if (outDir == null) {
      throw new InvalidJobConfException("Output directory not set.");
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the exact FQCN: for built-in codecs use org.apache.hadoop.io.compress.GzipCodec / DefaultCodec / BZip2Codec / SnappyCodec
2. Ship the codec jar to the cluster: hadoop jar app.jar -libjars codec.jar Driver, or place it in the cluster's share/lib directory; for MR jobs ensure tasks also get it via mapreduce.job.classpath.files / DistributedCache
3. Sanity-check resolution before submit: Class.forName(name) with the job's classloader in a try/catch
4. If a dependency brought a relocated codec, set the config to the relocated FQCN or stop shading that package

Example fix

// before: codec class not on the job classpath
conf.setBoolean("mapreduce.output.fileoutputformat.compress", true);
conf.set("mapreduce.output.fileoutputformat.compress.codec", "com.hadoop.compression.lzo.LzoCodec");

// after: built-in codec, always resolvable
conf.setBoolean("mapreduce.output.fileoutputformat.compress", true);
conf.setClass(FileOutputFormat.COMPRESS_CODEC,
    org.apache.hadoop.io.compress.GzipCodec.class,
    org.apache.hadoop.io.compress.CompressionCodec.class);

// before submit: resolve the codec class with the job classloader
String codec = conf.get("mapreduce.output.fileoutputformat.compress.codec");
if (codec != null) {
  try { Class.forName(codec, false, conf.getClassLoader()); }
  catch (ClassNotFoundException e) { throw new IllegalArgumentException("Codec not on classpath: " + codec, e); }
}

// Java 'type guard': is the configured name a loadable CompressionCodec?
static boolean isResolvableCompressionCodec(String name, ClassLoader cl) {
  try {
    return org.apache.hadoop.io.compress.CompressionCodec.class
        .isAssignableFrom(Class.forName(name, false, cl));
  } catch (ClassNotFoundException e) { return false; }
}

catch IllegalArgumentException around getOutputCompressorClass(job, DefaultCodec.class); fall back to the default codec or abort submission with a clear classpath message

When it happens

Trigger: FileOutputFormat.setCompressOutput(job, true) plus a codec class name that cannot be loaded: typo in the FQCN, a third-party codec (e.g. LZO/LZopCodec, Snappy-native, Brotli) whose jar is not shipped to the cluster, or a shaded/relocated class name that does not exist in the deployed jar. Triggered at getOutputCompressorClass() call sites — checkOutputSpecs-adjacent client code, RecordWriter creation, or manual calls.

Common situations: Enabling compressed map/reduce output with a custom codec but forgetting -libjars or the share-DIR deployment; upgrading Hadoop or a shading plugin that moved codec packages; copy-pasting a codec FQCN from another project version.

Related errors

• Could not locate MapReduce framework name '{}' in mapreduce.
• Couldn't open queue configuration at " + xmlInUrl
• Error in instantiating YarnClient
• No KeyProviderFactory for ${uri} in ${KEY_PROVIDER_PATH}
• Could not find configured fencing method {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fb44a4bd69b0342d. Report an issue: GitHub.