apache/hadoop · error · IOException

LZO codec class not specified. Did you forget to set propert

Error message

LZO codec class not specified. Did you forget to set property io.compression.codec.lzo.class?

What it means

Thrown by Compression.Algorithm.LZO.createDecompressionStream() when a TFile data block compressed with LZO must be decompressed but Algorithm.LZO.isSupported() returns false, i.e. no usable LZO codec was loaded. Despite the wording, the property io.compression.codec.lzo.class always has a default value, so in practice this means the configured codec class failed to load (usually ClassNotFoundException) rather than that it was never specified. Reading an LZO-compressed TFile is impossible without the codec.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Compression.java:129

      }

      @Override
      CompressionCodec getCodec() throws IOException {
        if (!isSupported()) {
          throw new IOException(String.format(
              "LZO codec %s=%s could not be loaded", CONF_LZO_CLASS, clazz),
                  cnf);
        }

        return codec;
      }

      @Override
      public synchronized InputStream createDecompressionStream(
          InputStream downStream, Decompressor decompressor,
          int downStreamBufferSize) throws IOException {
        if (!isSupported()) {
          throw new IOException(
              "LZO codec class not specified. Did you forget to set property "
                  + CONF_LZO_CLASS + "?");
        }
        InputStream bis1 = null;
        if (downStreamBufferSize > 0) {
          bis1 = new BufferedInputStream(downStream, downStreamBufferSize);
        } else {
          bis1 = downStream;
        }
        conf.setInt(IO_COMPRESSION_CODEC_LZO_BUFFERSIZE_KEY,
            IO_COMPRESSION_CODEC_LZO_BUFFERSIZE_DEFAULT);
        CompressionInputStream cis =
            codec.createInputStream(bis1, decompressor);
        BufferedInputStream bis2 = new BufferedInputStream(cis, DATA_IBUF_SIZE);
        return bis2;
      }

      @Override

View on GitHub (pinned to 2add963021)

Solutions

1. Install the matching hadoop-lzo jar (and native library) on the node doing the read and retry
2. Check the effective value of io.compression.codec.lzo.class (Configuration and system property) and correct it to the installed codec class
3. Copy the file to a host that has the codec and rewrite it with "gz" or "none" via TFile before reading cluster-wide

Example fix

// before
TFile.Reader reader = new TFile.Reader(hfileIn, TFile.RFS_LENGTH, conf); // file written with "lzo"

// after
if (!Compression.Algorithm.LZO.isSupported()) {
  throw new IOException("Cannot read LZO TFile: hadoop-lzo codec not installed");
}
TFile.Reader reader = new TFile.Reader(hfileIn, TFile.RFS_LENGTH, conf);

// Before opening a reader over a possibly-LZO file
String algo = TFile.getCompressionAlgorithmByName(reader.getCompressionName());
if (algo.equals(TFile.COMPRESSION_LZO) && !Compression.Algorithm.LZO.isSupported()) {
  throw new IOException("File is LZO-compressed but no LZO codec is installed on this host");
}

catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("LZO codec")) {
    // decompression impossible without deployment fix: surface to operator
  }
}

When it happens

Trigger: Opening a TFile.Reader over a file written with the "lzo" algorithm (the reader builds a decompression stream per data block) while io.compression.codec.lzo.class resolves to a class that is not loadable on the current JVM.

Common situations: Reading LZO-compressed TFiles or old HFile/sequence data on a cluster that lacks the hadoop-lzo jar; a task or client node whose classpath excludes the codec jar; a codec class name set for a different hadoop-lago/hadoop-lzo version; moving files from an LZO-enabled cluster to one without it.

Related errors

• LZO codec %s=%s could not be loaded
• Unsupported compression algorithm name:
• Compression option provided does not match the file
• All merged files must be compressed or not.
• block overrun

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5a0db286ae8dff3d. Report an issue: GitHub.