apache/hadoop · error · IOException

Error creating file descriptor in ${path}: ${e.getMessage()}

Error message

Error creating file descriptor in ${path}: ${e.getMessage()}

What it means

SharedFileDescriptorFactory.create probes each candidate directory by creating a test descriptor (createDescriptor0); on IOException it appends 'Error creating file descriptor in <path>: <cause>' and tries the next directory. If every candidate fails, the accumulated fragments are thrown as one IOException, each naming a directory and its underlying OS reason (permission denied, no such file, no space...).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.java:99

      throw new IOException("no SharedFileDescriptorFactory paths were " +
          "configured.");
    }
    StringBuilder errors = new StringBuilder();
    String strPrefix = "";
    for (String path : paths) {
      try {
        FileInputStream fis = 
            new FileInputStream(createDescriptor0(prefix + "test", path, 1));
        fis.close();
        deleteStaleTemporaryFiles0(prefix, path);
        return new SharedFileDescriptorFactory(prefix, path);
      } catch (IOException e) {
        errors.append(strPrefix).append("Error creating file descriptor in ").
               append(path).append(": ").append(e.getMessage());
        strPrefix = ", ";
      }
    }
    throw new IOException(errors.toString());
  }

  /**
   * Create a SharedFileDescriptorFactory.
   *
   * @param prefix    Prefix to add to all file names we use.
   * @param path      Path to use.
   */
  private SharedFileDescriptorFactory(String prefix, String path) {
    this.prefix = prefix;
    this.path = path;
  }

  public String getPath() {
    return path;
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Read each '<path>: <message>' fragment and fix the stated cause for the first candidate (mount missing, permission, space).
2. Ensure /dev/shm exists, is writable by the daemon user, and is large enough (df -h /dev/shm; raise --shm-size in containers).
3. Extend the candidate list (comma-separated) with a guaranteed-writable fallback such as /dev/shm,/tmp,/var/tmp.
4. If short-circuit reads are not required, disable them (dfs.client.read.shortcircuit=false) rather than fighting the environment.

Example fix

# container with missing/too-small /dev/shm
docker run --shm-size=1g ...

# or widen candidates in hdfs-site.xml
<property>
  <name>dfs.datanode.shared.file.descriptor.paths</name>
  <value>/dev/shm,/tmp</value>
</property>

for (String p : paths) {
  File d = new File(p);
  if (!d.isDirectory() || !d.canWrite()) {
    LOG.warn("shared file descriptor path unusable: {}", p);
  }
}

When it happens

Trigger: All configured dirs failing at once: /dev/shm missing or mounted tiny/full (common in containers), /tmp not writable by the daemon user, SELinux/AppArmor denials on the shm path, disk exhaustion on the fallback dir.

Common situations: Docker/Kubernetes containers without /dev/shm or with a small --shm-size; hardened hosts where the DataNode user cannot write the configured dir; /tmp filled by spill files.

Related errors

• size of shared memory segment was {intSize}, but that is not
• no SharedFileDescriptorFactory paths were configured.
• got EOF while trying to transfer the file descriptor for the
• Error accessing Bucket %s
• Error accessing %s

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a69234dd646cdfd6. Report an issue: GitHub.