apache/hadoop · error · UnsupportedOperationException

can't use DfsClientShm because we failed to load misc.Unsafe

Error message

can't use DfsClientShm because we failed to load misc.Unsafe.

What it means

ShortCircuitShm accesses the mmapped slot array through sun.misc.Unsafe, captured reflectively at class initialization (safetyDance() grabs Unsafe.theUnsafe and swallows failures). If that lookup failed on this JVM, the static field stays null and the constructor throws UnsupportedOperationException. The cause is JVM-level: reflective access to theUnsafe was blocked (security manager, module restrictions) or the runtime lacks the class.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java:477

   *
   *                    Although this is a FileInputStream, we are going to
   *                    assume that the underlying file descriptor is writable
   *                    as well as readable. It would be more appropriate to use
   *                    a RandomAccessFile here, but that class does not have
   *                    any public accessor which returns a FileDescriptor,
   *                    unlike FileInputStream.
   */
  public ShortCircuitShm(ShmId shmId, FileInputStream stream)
        throws IOException {
    if (!NativeIO.isAvailable()) {
      throw new UnsupportedOperationException("NativeIO is not available.");
    }
    if (Shell.WINDOWS) {
      throw new UnsupportedOperationException(
          "DfsClientShm is not yet implemented for Windows.");
    }
    if (unsafe == null) {
      throw new UnsupportedOperationException(
          "can't use DfsClientShm because we failed to " +
          "load misc.Unsafe.");
    }
    this.shmId = shmId;
    this.mmappedLength = getUsableLength(stream);
    this.baseAddress = POSIX.mmap(stream.getFD(),
        POSIX.MMAP_PROT_READ | POSIX.MMAP_PROT_WRITE, true, mmappedLength);
    this.slots = new Slot[mmappedLength / BYTES_PER_SLOT];
    this.allocatedSlots = new BitSet(slots.length);
    LOG.trace("creating {}(shmId={}, mmappedLength={}, baseAddress={}, "
        + "slots.length={})", this.getClass().getSimpleName(), shmId,
        mmappedLength, String.format("%x", baseAddress), slots.length);
  }

  public final ShmId getShmId() {
    return shmId;
  }

View on GitHub (pinned to 2add963021)

Solutions

If shm-backed short-circuit reads are not a hard requirement, set dfs.client.read.shortcircuit=false and the error disappears
Run on a standard JDK distribution that includes jdk.unsupported and permits the theUnsafe reflection (no SecurityManager denying it)
For custom runtimes, include jdk.unsupported in the image and verify with a trivial snippet: Field f = sun.misc.Unsafe.class.getDeclaredField("theUnsafe"); f.setAccessible(true);
Upgrade Hadoop; newer releases removed the direct Unsafe dependency on this path

Example fix

// before: restricted runtime
java -jar app.jar

// after: allow reflective access the shm code needs
java --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \
     --add-opens java.base/sun.misc=ALL-UNNAMED -jar app.jar

Defensive patterns

Strategy: try-catch

Validate before calling

static boolean unsafeAvailable() {
  try {
    java.lang.reflect.Field f = sun.misc.Unsafe.class.getDeclaredField("theUnsafe");
    f.setAccessible(true);
    return f.get(null) != null;
  } catch (Throwable t) { return false; }
}
// if (!unsafeAvailable()) disable short-circuit shm usage

Try / catch

try {
  // path that constructs ShortCircuitShm
} catch (UnsupportedOperationException e) {
  if (e.getMessage() != null && e.getMessage().contains("misc.Unsafe")) {
    // fall back to non-short-circuit reads
  } else { throw e; }
}

Prevention

Verify Unsafe availability in the same JVM before enabling dfs.client.read.shortcircuit
Include jdk.unsupported in jlink runtime images for Hadoop clients
Avoid SecurityManager policies that deny setAccessible on JDK internals in HDFS client processes

When it happens

Trigger: new ShortCircuitShm(...) on a JVM where the reflective grab of sun.misc.Unsafe.theUnsafe failed at class-init - a SecurityManager denying setAccessible, a jlink runtime missing jdk.unsupported, or a restricted/exotic JVM - followed by an attempt to use shm-backed short-circuit reads.

Common situations: Hardened or custom Java runtimes (jlink images without jdk.unsupported); test JVMs with restrictive security managers; JVM upgrades that tighten reflective access to JDK internals.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7f077026f6877bdc. Report an issue: GitHub.