What causes "LZO codec %s=%s could not be loaded" in apache/hadoop?

Install a third-party LZO implementation (e.g. the kevinweil/hadoop-lzo jar plus its native library) on every node and client, and add it to the classpath

apache/hadoop · error · IOException

LZO codec %s=%s could not be loaded

Error message

LZO codec %s=%s could not be loaded

What it means

Thrown by Compression.Algorithm.LZO.getCodec() when the LZO codec class cannot be loaded. The class name comes from the configuration key io.compression.codec.lzo.class (or a JVM system property of the same name), defaulting to org.apache.hadoop.io.compress.LzoCodec. Apache Hadoop cannot ship an LZO implementation because LZO is GPL licensed, so the class is only present when a third-party hadoop-lzo jar is installed. The original ClassNotFoundException is chained as the cause and names the exact missing class.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Compression.java:116

          checked = true;
          reinitCodecInTests = conf.getBoolean("test.reload.lzo.codec", false);
          clazz = getLzoCodecClass();
          try {
            LOG.info("Trying to load Lzo codec class: " + clazz);
            codec =
                (CompressionCodec) ReflectionUtils.newInstance(Class
                    .forName(clazz), conf);
          } catch (ClassNotFoundException e) {
            cnf = e;
          }
        }
        return codec != null;
      }

      @Override
      CompressionCodec getCodec() throws IOException {
        if (!isSupported()) {
          throw new IOException(String.format(
              "LZO codec %s=%s could not be loaded", CONF_LZO_CLASS, clazz),
                  cnf);
        }

        return codec;
      }

      @Override
      public synchronized InputStream createDecompressionStream(
          InputStream downStream, Decompressor decompressor,
          int downStreamBufferSize) throws IOException {
        if (!isSupported()) {
          throw new IOException(
              "LZO codec class not specified. Did you forget to set property "
                  + CONF_LZO_CLASS + "?");
        }
        InputStream bis1 = null;
        if (downStreamBufferSize > 0) {

View on GitHub (pinned to 2add963021)

Solutions

Install a third-party LZO implementation (e.g. the kevinweil/hadoop-lzo jar plus its native library) on every node and client, and add it to the classpath
If you use a different LZO codec class, set io.compression.codec.lzo.class in the Configuration (or as a JVM system property) to its fully qualified name, then verify with Compression.Algorithm.LZO.isSupported()
If LZO is not required, write and read the TFile with "gz", "snappy", or "none" instead
If the file must remain LZO, read it on a host that has the codec and re-compress it to a supported algorithm

Example fix

// before
TFile.Writer writer = new TFile.Writer(fout, TFile.MIN_BLOCK_SIZE, TFile.COMPRESSION_LZO, null);

// after
String algo = Compression.Algorithm.LZO.isSupported()
    ? TFile.COMPRESSION_LZO : TFile.COMPRESSION_GZ;
TFile.Writer writer = new TFile.Writer(fout, TFile.MIN_BLOCK_SIZE, algo, null);

Defensive patterns

Strategy: validation

Validate before calling

// Probe once at startup before any LZO TFile work
if (!Compression.Algorithm.LZO.isSupported()) {
  throw new IllegalStateException(
      "LZO codec class (io.compression.codec.lzo.class="
      + ". Defaults to org.apache.hadoop.io.compress.LzoCodec) is not on the classpath. "
      + "Install hadoop-lzo or choose another compression algorithm.");
}

Try / catch

catch (IOException e) {
  if (e.getCause() instanceof ClassNotFoundException) {
    // codec class missing: fail fast with a deployment message, do not retry
  } else { throw e; }
}

Prevention

Run Algorithm.LZO.isSupported() as a startup self-check on every node role that reads or writes LZO TFiles
Ship the hadoop-lzo jar and native library via the same deployment mechanism as Hadoop itself so versions stay in sync
Centralize the compression-algorithm choice in one config key validated at job submission

When it happens

Trigger: Creating a TFile.Writer or opening a TFile.Reader with compression name "lzo"; calling Compression.Algorithm.LZO.getCodec(), createCompressionStream(), or createDecompressionStream() when Class.forName() on the configured codec class throws ClassNotFoundException because io.compression.codec.lzo.class (or its default) is not on the classpath.

Common situations: Using a vanilla Apache Hadoop distribution that has no hadoop-lzo jar; io.compression.codec.lzo.class pointing at a class from an incompatible hadoop-lzo version; the codec jar present on the cluster but missing from the client or task classpath; reading data files produced on another cluster that had LZO installed.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/dbc73f6f9fe9b1de. Report an issue: GitHub.