apache/hadoop · error · RuntimeException

lz4-java library is not available: Lz4Decompressor has not b

Error message

lz4-java library is not available: Lz4Decompressor has not been loaded. You need to add lz4-java.jar to your CLASSPATH. {}

What it means

Lz4Decompressor's constructor calls LZ4Factory.fastestInstance().safeDecompressor(); if the lz4-java runtime cannot produce any decompressor implementation (jar missing or unusable), the factory throws AssertionError, which is rethrown as RuntimeException with the 'add lz4-java.jar to your CLASSPATH' guidance. Same environmental failure as the compressor side, hitting the decompression path at object construction.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java:63

  private byte[] userBuf = null;
  private int userBufOff = 0, userBufLen = 0;
  private boolean finished;

  private LZ4SafeDecompressor lz4Decompressor;

  /**
   * Creates a new compressor.
   *
   * @param directBufferSize size of the direct buffer to be used.
   */
  public Lz4Decompressor(int directBufferSize) {
    this.directBufferSize = directBufferSize;

    try {
      LZ4Factory lz4Factory = LZ4Factory.fastestInstance();
      lz4Decompressor = lz4Factory.safeDecompressor();
    } catch (AssertionError t) {
      throw new RuntimeException("lz4-java library is not available: " +
              "Lz4Decompressor has not been loaded. You need to add " +
              "lz4-java.jar to your CLASSPATH. " + t, t);
    }

    compressedDirectBuf = ByteBuffer.allocateDirect(directBufferSize);
    uncompressedDirectBuf = ByteBuffer.allocateDirect(directBufferSize);
    uncompressedDirectBuf.position(directBufferSize);

  }

  /**
   * Creates a new decompressor with the default buffer size.
   */
  public Lz4Decompressor() {
    this(DEFAULT_DIRECT_BUFFER_SIZE);
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

Ship/declare org.lz4:lz4-java (lz4-java.jar) on every node and client that constructs Lz4Decompressor, pinned to the version matching your Hadoop build.
Fix shading rules to include net.jpountz.lz4 classes and native libraries; verify with jar tf your-uber.jar | grep jpountz.
Remove duplicate/conflicting lz4 artifacts (mvn dependency:tree | grep lz4) so the factory finds one working implementation.
Validate codec availability at service startup by constructing the decompressor once in a health check.

Example fix

// before
Decompressor d = new Lz4Decompressor(64 * 1024); // RuntimeException, job dies mid-task

// after
static final boolean LZ4_PRESENT;
static {
  boolean ok;
  try {
    LZ4Factory.fastestInstance();
    ok = true;
  } catch (AssertionError | NoClassDefFoundError t) {
    ok = false;
  }
  LZ4_PRESENT = ok;
}
Decompressor d = LZ4_PRESENT
    ? new Lz4Decompressor(64 * 1024)
    : codecFactory.getCodec("org.apache.hadoop.io.compress.DefaultCodec").createDecompressor();

Defensive patterns

Strategy: validation

Validate before calling

static boolean lz4DecompressorAvailable() {
  try {
    LZ4Factory.fastestInstance().safeDecompressor();
    return true;
  } catch (AssertionError | NoClassDefFoundError t) {
    return false;
  }
}

Decompressor d = lz4DecompressorAvailable()
    ? new Lz4Decompressor(64 * 1024)
    : fallbackCodec.createDecompressor();

Try / catch

try {
  return new Lz4Decompressor(directBufferSize);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("lz4-java library is not available")) {
    return fallbackCodec.createDecompressor(); // e.g. DefaultCodec path
  }
  throw e;
}

Prevention

Ship lz4-java.jar on every node that reads, not just writes, lz4 data.
Pin lz4-java across writer and reader deployments to the same version.
Add a startup capability probe per node class (client/worker) so gaps surface at deploy time.

When it happens

Trigger: new Lz4Decompressor(bufferSize) (directly or via Lz4Codec.createDecompressor()) when LZ4Factory.fastestInstance() throws AssertionError: no net.jpountz.lz4 classes resolvable, shaded-out natives, or a corrupted lz4-java jar on the classpath.

Common situations: Readers/MapReduce tasks on nodes with a different (slimmer) classpath than writers; user jars bundling an incompatible lz4-java shadowing Hadoop's; 'minimized' shaded clients (sqoop/flume custom sinks) dropping lz4 classes; upgrading hadoop-common without updating its compression dependencies.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8bc8fcb2e4a6cb6b. Report an issue: GitHub.