apache/hadoop · error · UnsupportedOperationException

NativeIO is not available.

Error message

NativeIO is not available.

What it means

ShortCircuitShm needs mmap and raw memory access, which Hadoop performs through NativeIO (JNI into libhadoop). The constructor's first guard throws UnsupportedOperationException when NativeIO.isAvailable() is false - NativeCodeLoader never loaded a working native library. This fails before any shm logic runs, so it is an environment problem (missing/broken native library), not a data or config-semantics problem.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java:470

  /**
   * Create the ShortCircuitShm.
   *
   * @param shmId       The ID to use.
   * @param stream      The stream that we're going to use to create this
   *                    shared memory segment.
   *
   *                    Although this is a FileInputStream, we are going to
   *                    assume that the underlying file descriptor is writable
   *                    as well as readable. It would be more appropriate to use
   *                    a RandomAccessFile here, but that class does not have
   *                    any public accessor which returns a FileDescriptor,
   *                    unlike FileInputStream.
   */
  public ShortCircuitShm(ShmId shmId, FileInputStream stream)
        throws IOException {
    if (!NativeIO.isAvailable()) {
      throw new UnsupportedOperationException("NativeIO is not available.");
    }
    if (Shell.WINDOWS) {
      throw new UnsupportedOperationException(
          "DfsClientShm is not yet implemented for Windows.");
    }
    if (unsafe == null) {
      throw new UnsupportedOperationException(
          "can't use DfsClientShm because we failed to " +
          "load misc.Unsafe.");
    }
    this.shmId = shmId;
    this.mmappedLength = getUsableLength(stream);
    this.baseAddress = POSIX.mmap(stream.getFD(),
        POSIX.MMAP_PROT_READ | POSIX.MMAP_PROT_WRITE, true, mmappedLength);
    this.slots = new Slot[mmappedLength / BYTES_PER_SLOT];
    this.allocatedSlots = new BitSet(slots.length);
    LOG.trace("creating {}(shmId={}, mmappedLength={}, baseAddress={}, "
        + "slots.length={})", this.getClass().getSimpleName(), shmId,

View on GitHub (pinned to 2add963021)

Solutions

1. Run `hadoop checknative -a` on the client host to see whether libhadoop.so loads and why not
2. Install or build the matching hadoop-native library, set HADOOP_COMMON_LIB_NATIVE_DIR (and LD_LIBRARY_PATH), and re-run checknative to verify
3. If native code is unavailable by design, set dfs.client.read.shortcircuit=false so the shm path is never entered
4. On exotic platforms, build libhadoop from source for that toolchain, or run the client on a supported Linux host

if (!org.apache.hadoop.io.nativeio.NativeIO.isAvailable()) {
  conf.setBoolean("dfs.client.read.shortcircuit", false); // shm path requires NativeIO
}

try {
  // operations that may build a ShortCircuitShm
} catch (UnsupportedOperationException e) {
  if ("NativeIO is not available.".equals(e.getMessage())) {
    // disable short-circuit reads and retry over the normal path
  } else { throw e; }
}

When it happens

Trigger: new ShortCircuitShm(...) on a JVM where NativeIO.isAvailable() returns false - libhadoop.so absent from the library path, built for another architecture or glibc, or native loading disabled - when the client attempts the shm handshake for short-circuit reads.

Common situations: Edge or gateway machines and containers without the hadoop-native package; HADOOP_COMMON_LIB_NATIVE_DIR / LD_LIBRARY_PATH not pointing at the native directory; musl-based images (Alpine) where the stock libhadoop does not load.

Related errors

• Bailing out since native library couldn't be loaded
• Bailing out since native library couldn't be loaded
• Although a UNIX domain socket path is configured as {domainS
• Cannot start datanode because the configured max locked memo
• Replica generation stamp < block generation stamp, block={bl

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b632463d9673aaac. Report an issue: GitHub.