{"record":{"id":"5023fc4eac89cdda","repo":"apache/hadoop","slug":"eio-5023fc","errorCode":"EIO","errorMessage":"fuseConnectAsThreadUid: failed to open a libhdfs connection!  error %d.\n","messagePattern":"fuseConnectAsThreadUid: failed to open a libhdfs connection!  error (.+?)\\.\n","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_statfs.c","lineNumber":41,"sourceCode":"\nint dfs_statfs(const char *path, struct statvfs *st)\n{\n  struct hdfsConn *conn = NULL;\n  hdfsFS fs;\n  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;\n  int ret;\n\n  TRACE1(\"statfs\",path)\n\n  assert(path);\n  assert(st);\n  assert(dfs);\n\n  memset(st,0,sizeof(struct statvfs));\n\n  ret = fuseConnectAsThreadUid(&conn);\n  if (ret) {\n    fprintf(stderr, \"fuseConnectAsThreadUid: failed to open a libhdfs \"\n            \"connection!  error %d.\\n\", ret);\n    ret = -EIO;\n    goto cleanup;\n  }\n  fs = hdfsConnGetFs(conn);\n\n  const tOffset cap   = hdfsGetCapacity(fs);\n  const tOffset used  = hdfsGetUsed(fs);\n  const tOffset bsize = hdfsGetDefaultBlockSize(fs);\n\n  st->f_bsize   =  bsize;\n  st->f_frsize  =  bsize;\n  st->f_blocks  =  cap/bsize;\n  st->f_bfree   =  (cap-used)/bsize;\n  st->f_bavail  =  (cap-used)/bsize;\n  st->f_files   =  1000;\n  st->f_ffree   =  500;\n  st->f_favail  =  500;","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_statfs.c#L23-L59","documentation":"Emitted by the fuse-dfs statfs handler when fuseConnectAsThreadUid() cannot obtain a libhdfs connection for the calling UID; statvfs on the mount fails with EIO, so df and disk-space checks over the mount break. After a successful connect the handler queries hdfsGetCapacity/hdfsGetUsed/hdfsGetDefaultBlockSize to fill the statvfs struct, but none of that runs when the connection step fails.","triggerScenarios":"df <mount>, statvfs(), or any tool probing filesystem capacity when getUsername(uid) fails, the uid's Kerberos ticket cache is missing, or hdfsBuilderConnect fails (JVM/CLASSPATH, wrong fuse_dfs.conf namenode URI/port, NameNode down).","commonSituations":"Monitoring systems polling df on the mount around the clock - they are usually the first to notice expired Kerberos tickets or a NameNode outage; fuse_dfs launched without Hadoop jars on CLASSPATH.","solutions":["Check fuse-dfs stderr for the fuseNewConnect/fuseConnect message that names the underlying error.","Refresh Kerberos tickets (kinit) for the monitoring/monitored uid - df polling often outlives ticket lifetime.","Confirm NameNode is up and fuse_dfs.conf matches: hdfs dfs -ls hdfs://<nn>:<port>/ from the same host.","Ensure the accessing uid has a local passwd entry.","Restart fuse_dfs with a fixed environment if the JVM failed to initialize."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"# monitoring probe that fails fast when the backend connect is broken\nCLASSPATH=\"$(hdfs classpath --glob)\" hdfs dfs -ls \"${HDFS_URI:-hdfs://nn:8020/}\" >/dev/null || { echo \"HDFS backend down\"; exit 1; }\nklist -s || { echo \"ticket expired - df on the mount will fail with EIO\"; exit 1; }","typeGuard":null,"tryCatchPattern":"struct statvfs st;\nif (statvfs(\"/hdfs_mount\", &st) < 0 && errno == EIO) {\n    /* fuse-dfs could not connect; do not report '0 bytes free' - alert instead */\n    mark_backend_unreachable();\n}","preventionTips":["Alert on EIO from df probes separately from capacity thresholds - it means monitoring lost the HDFS connection, not that space ran out.","Run df probes under a principal with auto-renewed tickets (keytab + k5start).","Track fuse-dfs stderr for fuseNewConnect messages; they precede every mount-wide EIO episode.","Verify NameNode health independently (hdfs dfsadmin -report) when mount probes fail."],"tags":["c","fuse-dfs","libhdfs","kerberos","connection","eio","statfs","df"],"backgroundTag":"hdfs-connection-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}