EIO

EIO

Error message

fuseConnectAsThreadUid: failed to open a libhdfs connection!  error %d.

What it means

Emitted by the fuse-dfs statfs handler when fuseConnectAsThreadUid() cannot obtain a libhdfs connection for the calling UID; statvfs on the mount fails with EIO, so df and disk-space checks over the mount break. After a successful connect the handler queries hdfsGetCapacity/hdfsGetUsed/hdfsGetDefaultBlockSize to fill the statvfs struct, but none of that runs when the connection step fails.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_statfs.c:41

int dfs_statfs(const char *path, struct statvfs *st)
{
  struct hdfsConn *conn = NULL;
  hdfsFS fs;
  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
  int ret;

  TRACE1("statfs",path)

  assert(path);
  assert(st);
  assert(dfs);

  memset(st,0,sizeof(struct statvfs));

  ret = fuseConnectAsThreadUid(&conn);
  if (ret) {
    fprintf(stderr, "fuseConnectAsThreadUid: failed to open a libhdfs "
            "connection!  error %d.\n", ret);
    ret = -EIO;
    goto cleanup;
  }
  fs = hdfsConnGetFs(conn);

  const tOffset cap   = hdfsGetCapacity(fs);
  const tOffset used  = hdfsGetUsed(fs);
  const tOffset bsize = hdfsGetDefaultBlockSize(fs);

  st->f_bsize   =  bsize;
  st->f_frsize  =  bsize;
  st->f_blocks  =  cap/bsize;
  st->f_bfree   =  (cap-used)/bsize;
  st->f_bavail  =  (cap-used)/bsize;
  st->f_files   =  1000;
  st->f_ffree   =  500;
  st->f_favail  =  500;

View on GitHub (pinned to 2add963021)

Solutions

Check fuse-dfs stderr for the fuseNewConnect/fuseConnect message that names the underlying error.
Refresh Kerberos tickets (kinit) for the monitoring/monitored uid - df polling often outlives ticket lifetime.
Confirm NameNode is up and fuse_dfs.conf matches: hdfs dfs -ls hdfs://<nn>:<port>/ from the same host.
Ensure the accessing uid has a local passwd entry.
Restart fuse_dfs with a fixed environment if the JVM failed to initialize.

Defensive patterns

Strategy: validation

Validate before calling

# monitoring probe that fails fast when the backend connect is broken
CLASSPATH="$(hdfs classpath --glob)" hdfs dfs -ls "${HDFS_URI:-hdfs://nn:8020/}" >/dev/null || { echo "HDFS backend down"; exit 1; }
klist -s || { echo "ticket expired - df on the mount will fail with EIO"; exit 1; }

Try / catch

struct statvfs st;
if (statvfs("/hdfs_mount", &st) < 0 && errno == EIO) {
    /* fuse-dfs could not connect; do not report '0 bytes free' - alert instead */
    mark_backend_unreachable();
}

Prevention

Alert on EIO from df probes separately from capacity thresholds - it means monitoring lost the HDFS connection, not that space ran out.
Run df probes under a principal with auto-renewed tickets (keytab + k5start).
Track fuse-dfs stderr for fuseNewConnect messages; they precede every mount-wide EIO episode.
Verify NameNode health independently (hdfs dfsadmin -report) when mount probes fail.

When it happens

Trigger: df <mount>, statvfs(), or any tool probing filesystem capacity when getUsername(uid) fails, the uid's Kerberos ticket cache is missing, or hdfsBuilderConnect fails (JVM/CLASSPATH, wrong fuse_dfs.conf namenode URI/port, NameNode down).

Common situations: Monitoring systems polling df on the mount around the clock - they are usually the first to notice expired Kerberos tickets or a NameNode outage; fuse_dfs launched without Hadoop jars on CLASSPATH.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5023fc4eac89cdda. Report an issue: GitHub.