{"record":{"id":"b632463d9673aaac","repo":"apache/hadoop","slug":"nativeio-is-not-available","errorCode":null,"errorMessage":"NativeIO is not available.","messagePattern":"NativeIO is not available\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java","lineNumber":470,"sourceCode":"\n  /**\n   * Create the ShortCircuitShm.\n   *\n   * @param shmId       The ID to use.\n   * @param stream      The stream that we're going to use to create this\n   *                    shared memory segment.\n   *\n   *                    Although this is a FileInputStream, we are going to\n   *                    assume that the underlying file descriptor is writable\n   *                    as well as readable. It would be more appropriate to use\n   *                    a RandomAccessFile here, but that class does not have\n   *                    any public accessor which returns a FileDescriptor,\n   *                    unlike FileInputStream.\n   */\n  public ShortCircuitShm(ShmId shmId, FileInputStream stream)\n        throws IOException {\n    if (!NativeIO.isAvailable()) {\n      throw new UnsupportedOperationException(\"NativeIO is not available.\");\n    }\n    if (Shell.WINDOWS) {\n      throw new UnsupportedOperationException(\n          \"DfsClientShm is not yet implemented for Windows.\");\n    }\n    if (unsafe == null) {\n      throw new UnsupportedOperationException(\n          \"can't use DfsClientShm because we failed to \" +\n          \"load misc.Unsafe.\");\n    }\n    this.shmId = shmId;\n    this.mmappedLength = getUsableLength(stream);\n    this.baseAddress = POSIX.mmap(stream.getFD(),\n        POSIX.MMAP_PROT_READ | POSIX.MMAP_PROT_WRITE, true, mmappedLength);\n    this.slots = new Slot[mmappedLength / BYTES_PER_SLOT];\n    this.allocatedSlots = new BitSet(slots.length);\n    LOG.trace(\"creating {}(shmId={}, mmappedLength={}, baseAddress={}, \"\n        + \"slots.length={})\", this.getClass().getSimpleName(), shmId,","sourceCodeStart":452,"sourceCodeEnd":488,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java#L452-L488","documentation":"ShortCircuitShm needs mmap and raw memory access, which Hadoop performs through NativeIO (JNI into libhadoop). The constructor's first guard throws UnsupportedOperationException when NativeIO.isAvailable() is false - NativeCodeLoader never loaded a working native library. This fails before any shm logic runs, so it is an environment problem (missing/broken native library), not a data or config-semantics problem.","triggerScenarios":"new ShortCircuitShm(...) on a JVM where NativeIO.isAvailable() returns false - libhadoop.so absent from the library path, built for another architecture or glibc, or native loading disabled - when the client attempts the shm handshake for short-circuit reads.","commonSituations":"Edge or gateway machines and containers without the hadoop-native package; HADOOP_COMMON_LIB_NATIVE_DIR / LD_LIBRARY_PATH not pointing at the native directory; musl-based images (Alpine) where the stock libhadoop does not load.","solutions":["Run `hadoop checknative -a` on the client host to see whether libhadoop.so loads and why not","Install or build the matching hadoop-native library, set HADOOP_COMMON_LIB_NATIVE_DIR (and LD_LIBRARY_PATH), and re-run checknative to verify","If native code is unavailable by design, set dfs.client.read.shortcircuit=false so the shm path is never entered","On exotic platforms, build libhadoop from source for that toolchain, or run the client on a supported Linux host"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if (!org.apache.hadoop.io.nativeio.NativeIO.isAvailable()) {\n  conf.setBoolean(\"dfs.client.read.shortcircuit\", false); // shm path requires NativeIO\n}","typeGuard":null,"tryCatchPattern":"try {\n  // operations that may build a ShortCircuitShm\n} catch (UnsupportedOperationException e) {\n  if (\"NativeIO is not available.\".equals(e.getMessage())) {\n    // disable short-circuit reads and retry over the normal path\n  } else { throw e; }\n}","preventionTips":["Add `hadoop checknative -a` to deployment smoke tests on every host that will use short-circuit reads","Pin the hadoop-native artifact version to the exact client Hadoop release","Do not strip libhadoop from slim container images used as HDFS clients"],"tags":["hdfs","short-circuit-read","native-library","jni","nativeio","libhadoop"],"backgroundTag":"native-library-not-loaded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}