{"record":{"id":"26aca6606dca1f40","repo":"apache/flink","slug":"unable-to-load-the-provided-hadoop-codec-s","errorCode":null,"errorMessage":"Unable to load the provided Hadoop codec [%s]","messagePattern":"Unable to load the provided Hadoop codec \\[(.+?)\\]","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-formats/flink-compress/src/main/java/org/apache/flink/formats/compress/CompressWriterFactory.java","lineNumber":132,"sourceCode":"\n    private void initializeCompressionCodec() {\n        if (hadoopCodec == null) {\n            Configuration conf = new Configuration();\n\n            for (Map.Entry<String, String> entry : hadoopConfigMap.entrySet()) {\n                conf.set(entry.getKey(), entry.getValue());\n            }\n\n            hadoopCodec = new CompressionCodecFactory(conf).getCodecByName(this.hadoopCodecName);\n        }\n    }\n\n    private String getHadoopCodecExtension(String hadoopCodecName, Configuration conf)\n            throws IOException {\n        CompressionCodec codec = new CompressionCodecFactory(conf).getCodecByName(hadoopCodecName);\n\n        if (codec == null) {\n            throw new IOException(\n                    \"Unable to load the provided Hadoop codec [\" + hadoopCodecName + \"]\");\n        }\n\n        return codec.getDefaultExtension();\n    }\n}\n","sourceCodeStart":114,"sourceCodeEnd":139,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-formats/flink-compress/src/main/java/org/apache/flink/formats/compress/CompressWriterFactory.java#L114-L139","documentation":"Thrown by CompressWriterFactory.getHadoopCodecExtension() when Hadoop's CompressionCodecFactory.getCodecByName() returns null for the configured compression codec name — i.e. no codec registered in the supplied Hadoop Configuration matches that name. This is a factory-construction-time failure: the writer cannot determine the file extension without a resolvable codec.","triggerScenarios":"Setting 'compression' for the compress format to a codec Hadoop does not know: short names of codecs whose classes are not on the classpath (e.g. 'lz4', 'lzo', 'snappy' without the corresponding hadoop-* dependency), typos like 'gizp', or fully-qualified class names of absent classes. Also triggered when a custom Configuration lacks io.compression.codecs entries for bundled codecs.","commonSituations":"Using file sinks with compression on clusters where snappy/lzo native libs or jars are missing; passing 'zstd' on old Hadoop versions; relying on the default Configuration instead of injecting one that lists the codecs.","solutions":["Use the fully-qualified codec class name Hadoop can load: org.apache.hadoop.io.compress.GzipCodec, ...BZip2Codec, ...DefaultCodec, or org.apache.hadoop.io.compress.SnappyCodec.","Add the missing codec dependency (e.g. hadoop-lzo, aircompressor for lz4/lzo) to the Flink lib/ directory or the job jar and restart.","Verify with a small job: new CompressionCodecFactory(new Configuration(true)).getCodecByName(\"<name>\") should be non-null.","Check spelling/case of the compression option value."],"exampleFix":"-- before\n'sink.properties.compression' = 'gizp'  -- typo, resolves to null\n\n-- after\n'sink.properties.compression' = 'org.apache.hadoop.io.compress.GzipCodec'","handlingStrategy":"validation","validationCode":"// Pre-flight in a test or job bootstrap:\norg.apache.hadoop.conf.Configuration conf = new Configuration(true);\nif (new org.apache.hadoop.io.compress.CompressionCodecFactory(conf)\n        .getCodecByName(codecName) == null) {\n    throw new IllegalArgumentException(\"Codec not on classpath: \" + codecName\n        + \" — add the hadoop codec jar or use a fully-qualified class name\");\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) { log codec name and hadoop classpath; fail fast at factory time — retrying will not load a missing class }","preventionTips":["Prefer fully-qualified codec class names over short names","Ship codec jars (snappy/lzo/zstd) in the job jar or Flink lib/","Test compression config in a one-off local job before production"],"tags":["hadoop","compression","codec","filesystem","configuration"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}