apache/hadoop · error

EIO

EIO

Error message

fuseConnectAsThreadUid: failed to open a libhdfs connection!  error %d.

What it means

Emitted by the fuse-dfs readdir handler when fuseConnectAsThreadUid() cannot obtain a libhdfs connection for the UID making the request; fuse_dfs_readdir returns -EIO, so directory listings on the mount fail with 'Input/output error'. fuseConnectAsThreadUid (fuse_connect.c) resolves the caller UID to a username and creates or reuses a cached hdfsFS via hdfsBuilderConnect, so the real cause is printed on fuse-dfs stderr just above this line.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_readdir.c:40

#include "fuse_connect.h"

int dfs_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
                       off_t offset, struct fuse_file_info *fi)
{
  int ret;
  struct hdfsConn *conn = NULL;
  hdfsFS fs;
  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;

  TRACE1("readdir", path)

  assert(dfs);
  assert(path);
  assert(buf);

  ret = fuseConnectAsThreadUid(&conn);
  if (ret) {
    fprintf(stderr, "fuseConnectAsThreadUid: failed to open a libhdfs "
            "connection!  error %d.\n", ret);
    ret = -EIO;
    goto cleanup;
  }
  fs = hdfsConnGetFs(conn);

  // Read dirents. Calling a variant that just returns the final path
  // component (HDFS-975) would save us from parsing it out below.
  int numEntries = 0;
  hdfsFileInfo *info = hdfsListDirectory(fs, path, &numEntries);

  // NULL means either the directory doesn't exist or maybe IO error.
  if (NULL == info) {
    ret = (errno > 0) ? -errno : -ENOENT;
    goto cleanup;
  }

  int i ;

View on GitHub (pinned to 2add963021)

Solutions

1. Read the fuse-dfs stderr lines immediately above this message - fuseNewConnect/fuseConnect there name the real cause (missing kinit, hdfsBuilderConnect error code, OOM).
2. If Kerberos: run kinit as the accessing user and confirm the ticket cache file for that uid exists and is fresh (ls /tmp/krb5cc_<uid> or $KRB5CCNAME).
3. Prove connectivity with the same environment before blaming the mount: CLASSPATH=$(hdfs classpath --glob) hdfs dfs -ls hdfs://<nn>:<port>/
4. Ensure the accessing uid resolves locally: getent passwd <uid> (fuse-dfs maps uid to username itself).
5. Restart fuse_dfs with a correct CLASSPATH, sized LIBHDFS_OPTS (-Xmx) and correct fuse_dfs.conf uri/port, then remount.

Example fix

// before
$ ls /hdfs_mount/data
ls: reading directory /hdfs_mount/data: Input/output error

# fuse-dfs stderr shows the root cause:
# fuseNewConnect: failed to find Kerberos ticket cache file '/tmp/krb5cc_1001'.
#   Did you remember to kinit for UID 1001?

// after
$ kinit alice@EXAMPLE.COM
$ ls /hdfs_mount/data   # succeeds

# preflight before trusting the fuse-dfs mount (as each accessing user)
klist -s || kinit "$USER"@EXAMPLE.COM
getent passwd "$(id -u)" >/dev/null || echo "FATAL: uid has no local passwd entry"
CLASSPATH="$(hdfs classpath --glob)" hdfs dfs -ls "${HDFS_URI:-hdfs://nn:8020/}" || exit 1
# only then rely on: ls /hdfs_mount/...

// users of the mount only see EIO; correlate with fuse-dfs stderr
DIR *d = opendir("/hdfs_mount/data");
if (!d && errno == EIO) {
    /* backend connect failed for this uid: kinit / CLASSPATH / namenode */
    report_and_retry_after_creds_refresh();
}

When it happens

Trigger: Running ls/find/glob on the fuse-dfs mount when: (1) getUsername(ctx->uid) fails because the uid has no local passwd entry (returns EIO); (2) Kerberos is configured but the uid's ticket cache (/tmp/krb5cc_<uid> or KRB5CCNAME from the caller's /proc/<pid>/environ) is missing -> fuseNewConnect returns -EACCES; (3) hdfsBuilderConnect fails because the JVM cannot start (broken CLASSPATH), fuse_dfs.conf has a wrong namenode URI/port, or the NameNode is unreachable.

Common situations: Kerberized cluster where the accessing user never ran kinit or the ticket expired after mount; fuse_dfs launched without the Hadoop CLASSPATH or with a too-small LIBHDFS_OPTS -Xmx; fuse_dfs.conf pointing at the wrong namenode address; uids that exist in HDFS but have no entry in the local passwd file.

Related errors

• EIO
• EIO
• EIO
• EIO
• EIO

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9d46159c883fc298. Report an issue: GitHub.