apache/flink · error · IOException

Unable to load the provided Hadoop codec [%s]

Error message

Unable to load the provided Hadoop codec [%s]

What it means

Thrown by CompressWriterFactory.getHadoopCodecExtension() when Hadoop's CompressionCodecFactory.getCodecByName() returns null for the configured compression codec name — i.e. no codec registered in the supplied Hadoop Configuration matches that name. This is a factory-construction-time failure: the writer cannot determine the file extension without a resolvable codec.

Source

Thrown at flink-formats/flink-compress/src/main/java/org/apache/flink/formats/compress/CompressWriterFactory.java:132

    private void initializeCompressionCodec() {
        if (hadoopCodec == null) {
            Configuration conf = new Configuration();

            for (Map.Entry<String, String> entry : hadoopConfigMap.entrySet()) {
                conf.set(entry.getKey(), entry.getValue());
            }

            hadoopCodec = new CompressionCodecFactory(conf).getCodecByName(this.hadoopCodecName);
        }
    }

    private String getHadoopCodecExtension(String hadoopCodecName, Configuration conf)
            throws IOException {
        CompressionCodec codec = new CompressionCodecFactory(conf).getCodecByName(hadoopCodecName);

        if (codec == null) {
            throw new IOException(
                    "Unable to load the provided Hadoop codec [" + hadoopCodecName + "]");
        }

        return codec.getDefaultExtension();
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Use the fully-qualified codec class name Hadoop can load: org.apache.hadoop.io.compress.GzipCodec, ...BZip2Codec, ...DefaultCodec, or org.apache.hadoop.io.compress.SnappyCodec.
2. Add the missing codec dependency (e.g. hadoop-lzo, aircompressor for lz4/lzo) to the Flink lib/ directory or the job jar and restart.
3. Verify with a small job: new CompressionCodecFactory(new Configuration(true)).getCodecByName("<name>") should be non-null.
4. Check spelling/case of the compression option value.

Example fix

-- before
'sink.properties.compression' = 'gizp'  -- typo, resolves to null

-- after
'sink.properties.compression' = 'org.apache.hadoop.io.compress.GzipCodec'

// Pre-flight in a test or job bootstrap:
org.apache.hadoop.conf.Configuration conf = new Configuration(true);
if (new org.apache.hadoop.io.compress.CompressionCodecFactory(conf)
        .getCodecByName(codecName) == null) {
    throw new IllegalArgumentException("Codec not on classpath: " + codecName
        + " — add the hadoop codec jar or use a fully-qualified class name");
}

catch (IOException e) { log codec name and hadoop classpath; fail fast at factory time — retrying will not load a missing class }

When it happens

Trigger: Setting 'compression' for the compress format to a codec Hadoop does not know: short names of codecs whose classes are not on the classpath (e.g. 'lz4', 'lzo', 'snappy' without the corresponding hadoop-* dependency), typos like 'gizp', or fully-qualified class names of absent classes. Also triggered when a custom Configuration lacks io.compression.codecs entries for bundled codecs.

Common situations: Using file sinks with compression on clusters where snappy/lzo native libs or jars are missing; passing 'zstd' on old Hadoop versions; relying on the default Configuration instead of injecting one that lists the codecs.

Related errors

• Could not create writer state serializer.
• Could not create committable serializer.
• Error opening the Input Split {} [{},{}]: {}
• The given configuration directory name '{}' ({}) does not de
• The Flink config file '{}' ({}) does not exist.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/26aca6606dca1f40. Report an issue: GitHub.