{"record":{"id":"fb44a4bd69b0342d","repo":"apache/hadoop","slug":"compression-codec-was-not-found-fb44a4","errorCode":null,"errorMessage":"Compression codec {} was not found.","messagePattern":"Compression codec (.+?) was not found\\.","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java","lineNumber":140,"sourceCode":"   * Get the {@link CompressionCodec} for compressing the job outputs.\n   * @param job the {@link Job} to look in\n   * @param defaultValue the {@link CompressionCodec} to return if not set\n   * @return the {@link CompressionCodec} to be used to compress the \n   *         job outputs\n   * @throws IllegalArgumentException if the class was specified, but not found\n   */\n  public static Class<? extends CompressionCodec> \n  getOutputCompressorClass(JobContext job, \n                       Class<? extends CompressionCodec> defaultValue) {\n    Class<? extends CompressionCodec> codecClass = defaultValue;\n    Configuration conf = job.getConfiguration();\n    String name = conf.get(FileOutputFormat.COMPRESS_CODEC);\n    if (name != null) {\n      try {\n        codecClass =\n            conf.getClassByName(name).asSubclass(CompressionCodec.class);\n      } catch (ClassNotFoundException e) {\n        throw new IllegalArgumentException(\"Compression codec \" + name + \n                                           \" was not found.\", e);\n      }\n    }\n    return codecClass;\n  }\n  \n  public abstract RecordWriter<K, V> \n     getRecordWriter(TaskAttemptContext job\n                     ) throws IOException, InterruptedException;\n\n  public void checkOutputSpecs(JobContext job\n                               ) throws FileAlreadyExistsException, IOException{\n    // Ensure that the output directory is set and not already there\n    Path outDir = getOutputPath(job);\n    if (outDir == null) {\n      throw new InvalidJobConfException(\"Output directory not set.\");\n    }\n","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java#L122-L158","documentation":"Thrown as IllegalArgumentException from FileOutputFormat.getOutputCompressorClass (FileOutputFormat.java:140). It reads mapreduce.output.fileoutputformat.compress.codec from the job configuration, loads the class via conf.getClassByName(name), and wraps any ClassNotFoundException in this IllegalArgumentException. The configured codec class name is not resolvable on the classpath of the JVM that calls the method (job client at submission or the task JVM at runtime).","triggerScenarios":"FileOutputFormat.setCompressOutput(job, true) plus a codec class name that cannot be loaded: typo in the FQCN, a third-party codec (e.g. LZO/LZopCodec, Snappy-native, Brotli) whose jar is not shipped to the cluster, or a shaded/relocated class name that does not exist in the deployed jar. Triggered at getOutputCompressorClass() call sites — checkOutputSpecs-adjacent client code, RecordWriter creation, or manual calls.","commonSituations":"Enabling compressed map/reduce output with a custom codec but forgetting -libjars or the share-DIR deployment; upgrading Hadoop or a shading plugin that moved codec packages; copy-pasting a codec FQCN from another project version.","solutions":["Verify the exact FQCN: for built-in codecs use org.apache.hadoop.io.compress.GzipCodec / DefaultCodec / BZip2Codec / SnappyCodec","Ship the codec jar to the cluster: hadoop jar app.jar -libjars codec.jar Driver, or place it in the cluster's share/lib directory; for MR jobs ensure tasks also get it via mapreduce.job.classpath.files / DistributedCache","Sanity-check resolution before submit: Class.forName(name) with the job's classloader in a try/catch","If a dependency brought a relocated codec, set the config to the relocated FQCN or stop shading that package"],"exampleFix":"// before: codec class not on the job classpath\nconf.setBoolean(\"mapreduce.output.fileoutputformat.compress\", true);\nconf.set(\"mapreduce.output.fileoutputformat.compress.codec\", \"com.hadoop.compression.lzo.LzoCodec\");\n\n// after: built-in codec, always resolvable\nconf.setBoolean(\"mapreduce.output.fileoutputformat.compress\", true);\nconf.setClass(FileOutputFormat.COMPRESS_CODEC,\n    org.apache.hadoop.io.compress.GzipCodec.class,\n    org.apache.hadoop.io.compress.CompressionCodec.class);","handlingStrategy":"validation","validationCode":"// before submit: resolve the codec class with the job classloader\nString codec = conf.get(\"mapreduce.output.fileoutputformat.compress.codec\");\nif (codec != null) {\n  try { Class.forName(codec, false, conf.getClassLoader()); }\n  catch (ClassNotFoundException e) { throw new IllegalArgumentException(\"Codec not on classpath: \" + codec, e); }\n}","typeGuard":"// Java 'type guard': is the configured name a loadable CompressionCodec?\nstatic boolean isResolvableCompressionCodec(String name, ClassLoader cl) {\n  try {\n    return org.apache.hadoop.io.compress.CompressionCodec.class\n        .isAssignableFrom(Class.forName(name, false, cl));\n  } catch (ClassNotFoundException e) { return false; }\n}","tryCatchPattern":"catch IllegalArgumentException around getOutputCompressorClass(job, DefaultCodec.class); fall back to the default codec or abort submission with a clear classpath message","preventionTips":["Ship third-party codec jars via -libjars or cluster share dir","Prefer conf.setClass(..., GzipCodec.class, CompressionCodec.class) over string config so compile-time types are used","Test job submission in CI with the same classpath as the cluster"],"tags":["hadoop","mapreduce","compression","codec","classpath","classnotfound","configuration"],"backgroundTag":"class-not-found-on-classpath","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}