apache/hadoop · error · RuntimeException

lz4-java library is not available: Lz4Compressor has not bee

Error message

lz4-java library is not available: Lz4Compressor has not been loaded. You need to add lz4-java.jar to your CLASSPATH. {}

What it means

Lz4Compressor's constructor delegates to net.jpountz.lz4's LZ4Factory.fastestInstance() to obtain a compressor. When the lz4-java runtime cannot instantiate any implementation (jar missing, unusable natives, incompatible build), the factory throws AssertionError, which the constructor catches and rethrows as a RuntimeException telling you to add lz4-java.jar to the CLASSPATH. It is an environment/classpath failure, not a code-logic bug, and it fails at construction time, before any compression happens.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java:74

  /**
   * Creates a new compressor.
   *
   * @param directBufferSize size of the direct buffer to be used.
   * @param useLz4HC use high compression ratio version of lz4, 
   *                 which trades CPU for compression ratio.
   */
  public Lz4Compressor(int directBufferSize, boolean useLz4HC) {
    this.directBufferSize = directBufferSize;

    try {
      LZ4Factory lz4Factory = LZ4Factory.fastestInstance();
      if (useLz4HC) {
        lz4Compressor = lz4Factory.highCompressor();
      } else {
        lz4Compressor = lz4Factory.fastCompressor();
      }
    } catch (AssertionError t) {
      throw new RuntimeException("lz4-java library is not available: " +
              "Lz4Compressor has not been loaded. You need to add " +
              "lz4-java.jar to your CLASSPATH. " + t, t);
    }

    uncompressedDirectBuf = ByteBuffer.allocateDirect(directBufferSize);

    // Compression is guaranteed to succeed if 'dstCapacity' >=
    // LZ4_compressBound(srcSize)
    // whereas LZ4_compressBound(isize) is (isize) + ((isize)/255) + 16)
    this.dstCapacity = (directBufferSize) + ((directBufferSize) / 255) + 16;

    compressedDirectBuf = ByteBuffer.allocateDirect(this.dstCapacity);
    compressedDirectBuf.position(this.dstCapacity);
  }

  /**
   * Creates a new compressor.
   *

View on GitHub (pinned to 2add963021)

Solutions

Add the lz4-java dependency explicitly (org.lz4:lz4-java) at the version matching your hadoop-common distribution, and confirm it appears on the final classpath (mvn dependency:tree / printed classpath).
If shading, keep net.jpountz.lz4 (and its native resources) in the uber-jar; exclude minimizeJar effects for that package.
Drop conflicting/older lz4 artifacts so LZ4Factory resolves exactly one implementation.
Smoke-test at startup: construct Lz4Compressor once so deployment fails fast with the clear message instead of mid-job.

Example fix

// before
Compressor c = new Lz4Compressor(64 * 1024); // RuntimeException if lz4-java absent

// after
try {
  Class.forName("net.jpountz.lz4.LZ4Factory");
} catch (ClassNotFoundException cnfe) {
  throw new IllegalStateException(
      "lz4-java.jar missing from classpath; add org.lz4:lz4-java", cnfe);
}
Compressor c = new Lz4Compressor(64 * 1024);

Defensive patterns

Strategy: validation

Validate before calling

static boolean lz4Available() {
  try {
    Class.forName("net.jpountz.lz4.LZ4Factory");
    LZ4Factory.fastestInstance(); // force binding resolution now
    return true;
  } catch (AssertionError | ClassNotFoundException | NoClassDefFoundError t) {
    return false;
  }
}

if (!lz4Available()) {
  throw new IllegalStateException("add org.lz4:lz4-java to the classpath");
}
new Lz4Compressor(64 * 1024);

Try / catch

try {
  new Lz4Compressor(64 * 1024);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("lz4-java library is not available")) {
    // classpath defect: fix deployment, do not retry
    throw new IllegalStateException("lz4-java missing on " + nodeName, e);
  }
  throw e;
}

Prevention

Declare org.lz4:lz4-java at the version matching your hadoop-common build.
When shading, keep net.jpountz.lz4 classes and native resources; verify with 'jar tf app.jar | grep jpountz'.
Resolve duplicate lz4 artifacts with dependency:tree so LZ4Factory binds one implementation.
Health-check codec construction at service startup to fail fast on slim nodes.

When it happens

Trigger: new Lz4Compressor(bufferSize) (or Lz4Codec creating its compressor) on a JVM where LZ4Factory.fastestInstance() cannot load an implementation: lz4-java.jar absent from the classpath, a shaded uber-jar that stripped net.jpountz classes or natives, or native artifacts for a wrong platform so even the JNI binding fails the factory's self-check.

Common situations: Slimmed-down deployments (hadoop-common without its optional lz4-java dependency); application shading (maven-shade minimizeJar) excluding net.jpountz.lz4; version conflicts between hadoop-common and a separately pinned lz4-java; running on an uncommon architecture where bundled natives do not load.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bf1ef0c7eb8aea39. Report an issue: GitHub.