apache/hadoop · error · UnsupportedOperationException

DfsClientShm is not yet implemented for Windows.

Error message

DfsClientShm is not yet implemented for Windows.

What it means

ShortCircuitShm, the client-side mapping of the datanode's shared-memory slot registry, is implemented with POSIX mmap and is not ported to Windows. The constructor throws UnsupportedOperationException immediately when Shell.WINDOWS is true, before touching any resource. The condition means the feature is unsupported on this OS, not that anything is misconfigured.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java:473

   *
   * @param shmId       The ID to use.
   * @param stream      The stream that we're going to use to create this
   *                    shared memory segment.
   *
   *                    Although this is a FileInputStream, we are going to
   *                    assume that the underlying file descriptor is writable
   *                    as well as readable. It would be more appropriate to use
   *                    a RandomAccessFile here, but that class does not have
   *                    any public accessor which returns a FileDescriptor,
   *                    unlike FileInputStream.
   */
  public ShortCircuitShm(ShmId shmId, FileInputStream stream)
        throws IOException {
    if (!NativeIO.isAvailable()) {
      throw new UnsupportedOperationException("NativeIO is not available.");
    }
    if (Shell.WINDOWS) {
      throw new UnsupportedOperationException(
          "DfsClientShm is not yet implemented for Windows.");
    }
    if (unsafe == null) {
      throw new UnsupportedOperationException(
          "can't use DfsClientShm because we failed to " +
          "load misc.Unsafe.");
    }
    this.shmId = shmId;
    this.mmappedLength = getUsableLength(stream);
    this.baseAddress = POSIX.mmap(stream.getFD(),
        POSIX.MMAP_PROT_READ | POSIX.MMAP_PROT_WRITE, true, mmappedLength);
    this.slots = new Slot[mmappedLength / BYTES_PER_SLOT];
    this.allocatedSlots = new BitSet(slots.length);
    LOG.trace("creating {}(shmId={}, mmappedLength={}, baseAddress={}, "
        + "slots.length={})", this.getClass().getSimpleName(), shmId,
        mmappedLength, String.format("%x", baseAddress), slots.length);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. On Windows clients, set dfs.client.read.shortcircuit=false and use normal datanode-mediated reads
2. Run the short-circuit workload on a Linux client or gateway host
3. If you maintain the client code, gate shm usage on !Shell.WINDOWS and degrade to regular reads

if (org.apache.hadoop.util.Shell.WINDOWS) {
  // ShortCircuitShm is POSIX-only
  conf.setBoolean("dfs.client.read.shortcircuit", false);
}

When it happens

Trigger: Constructing ShortCircuitShm on a Windows JVM - transitively a Windows HDFS client with dfs.client.read.shortcircuit=true attempting the shm handshake, or unit tests exercising ShortCircuitCache / DfsClientShmManager on Windows.

Common situations: Windows-based HDFS clients enabling short-circuit reads; CI runners executing short-circuit tests on Windows; developers porting HDFS client tooling across platforms.

Related errors

• size of shared memory segment was {intSize}, but that is not
• {this}: no more slots are available.
• {this}: slot {slotIdx} does not exist.
• {this}: invalid negative slot index {slotIdx}
• {this}: invalid slot index {slotIdx}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9b2d1c690c50f30a. Report an issue: GitHub.