{"record":{"id":"7f077026f6877bdc","repo":"apache/hadoop","slug":"can-t-use-dfsclientshm-because-we-failed-to-load-m","errorCode":null,"errorMessage":"can't use DfsClientShm because we failed to load misc.Unsafe.","messagePattern":"can't use DfsClientShm because we failed to load misc\\.Unsafe\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java","lineNumber":477,"sourceCode":"   *\n   *                    Although this is a FileInputStream, we are going to\n   *                    assume that the underlying file descriptor is writable\n   *                    as well as readable. It would be more appropriate to use\n   *                    a RandomAccessFile here, but that class does not have\n   *                    any public accessor which returns a FileDescriptor,\n   *                    unlike FileInputStream.\n   */\n  public ShortCircuitShm(ShmId shmId, FileInputStream stream)\n        throws IOException {\n    if (!NativeIO.isAvailable()) {\n      throw new UnsupportedOperationException(\"NativeIO is not available.\");\n    }\n    if (Shell.WINDOWS) {\n      throw new UnsupportedOperationException(\n          \"DfsClientShm is not yet implemented for Windows.\");\n    }\n    if (unsafe == null) {\n      throw new UnsupportedOperationException(\n          \"can't use DfsClientShm because we failed to \" +\n          \"load misc.Unsafe.\");\n    }\n    this.shmId = shmId;\n    this.mmappedLength = getUsableLength(stream);\n    this.baseAddress = POSIX.mmap(stream.getFD(),\n        POSIX.MMAP_PROT_READ | POSIX.MMAP_PROT_WRITE, true, mmappedLength);\n    this.slots = new Slot[mmappedLength / BYTES_PER_SLOT];\n    this.allocatedSlots = new BitSet(slots.length);\n    LOG.trace(\"creating {}(shmId={}, mmappedLength={}, baseAddress={}, \"\n        + \"slots.length={})\", this.getClass().getSimpleName(), shmId,\n        mmappedLength, String.format(\"%x\", baseAddress), slots.length);\n  }\n\n  public final ShmId getShmId() {\n    return shmId;\n  }\n","sourceCodeStart":459,"sourceCodeEnd":495,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java#L459-L495","documentation":"ShortCircuitShm accesses the mmapped slot array through sun.misc.Unsafe, captured reflectively at class initialization (safetyDance() grabs Unsafe.theUnsafe and swallows failures). If that lookup failed on this JVM, the static field stays null and the constructor throws UnsupportedOperationException. The cause is JVM-level: reflective access to theUnsafe was blocked (security manager, module restrictions) or the runtime lacks the class.","triggerScenarios":"new ShortCircuitShm(...) on a JVM where the reflective grab of sun.misc.Unsafe.theUnsafe failed at class-init - a SecurityManager denying setAccessible, a jlink runtime missing jdk.unsupported, or a restricted/exotic JVM - followed by an attempt to use shm-backed short-circuit reads.","commonSituations":"Hardened or custom Java runtimes (jlink images without jdk.unsupported); test JVMs with restrictive security managers; JVM upgrades that tighten reflective access to JDK internals.","solutions":["If shm-backed short-circuit reads are not a hard requirement, set dfs.client.read.shortcircuit=false and the error disappears","Run on a standard JDK distribution that includes jdk.unsupported and permits the theUnsafe reflection (no SecurityManager denying it)","For custom runtimes, include jdk.unsupported in the image and verify with a trivial snippet: Field f = sun.misc.Unsafe.class.getDeclaredField(\"theUnsafe\"); f.setAccessible(true);","Upgrade Hadoop; newer releases removed the direct Unsafe dependency on this path"],"exampleFix":"// before: restricted runtime\njava -jar app.jar\n\n// after: allow reflective access the shm code needs\njava --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \\\n     --add-opens java.base/sun.misc=ALL-UNNAMED -jar app.jar","handlingStrategy":"try-catch","validationCode":"static boolean unsafeAvailable() {\n  try {\n    java.lang.reflect.Field f = sun.misc.Unsafe.class.getDeclaredField(\"theUnsafe\");\n    f.setAccessible(true);\n    return f.get(null) != null;\n  } catch (Throwable t) { return false; }\n}\n// if (!unsafeAvailable()) disable short-circuit shm usage","typeGuard":null,"tryCatchPattern":"try {\n  // path that constructs ShortCircuitShm\n} catch (UnsupportedOperationException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"misc.Unsafe\")) {\n    // fall back to non-short-circuit reads\n  } else { throw e; }\n}","preventionTips":["Verify Unsafe availability in the same JVM before enabling dfs.client.read.shortcircuit","Include jdk.unsupported in jlink runtime images for Hadoop clients","Avoid SecurityManager policies that deny setAccessible on JDK internals in HDFS client processes"],"tags":["hdfs","short-circuit-read","sun-misc-unsafe","jdk-compatibility","reflection","jlink"],"backgroundTag":"sun-misc-unsafe-unavailable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}