{"record":{"id":"9d46159c883fc298","repo":"apache/hadoop","slug":"eio-9d4615","errorCode":"EIO","errorMessage":"fuseConnectAsThreadUid: failed to open a libhdfs connection!  error %d.\n","messagePattern":"fuseConnectAsThreadUid: failed to open a libhdfs connection!  error (.+?)\\.\n","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_readdir.c","lineNumber":40,"sourceCode":"#include \"fuse_connect.h\"\n\nint dfs_readdir(const char *path, void *buf, fuse_fill_dir_t filler,\n                       off_t offset, struct fuse_file_info *fi)\n{\n  int ret;\n  struct hdfsConn *conn = NULL;\n  hdfsFS fs;\n  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;\n\n  TRACE1(\"readdir\", path)\n\n  assert(dfs);\n  assert(path);\n  assert(buf);\n\n  ret = fuseConnectAsThreadUid(&conn);\n  if (ret) {\n    fprintf(stderr, \"fuseConnectAsThreadUid: failed to open a libhdfs \"\n            \"connection!  error %d.\\n\", ret);\n    ret = -EIO;\n    goto cleanup;\n  }\n  fs = hdfsConnGetFs(conn);\n\n  // Read dirents. Calling a variant that just returns the final path\n  // component (HDFS-975) would save us from parsing it out below.\n  int numEntries = 0;\n  hdfsFileInfo *info = hdfsListDirectory(fs, path, &numEntries);\n\n  // NULL means either the directory doesn't exist or maybe IO error.\n  if (NULL == info) {\n    ret = (errno > 0) ? -errno : -ENOENT;\n    goto cleanup;\n  }\n\n  int i ;","sourceCodeStart":22,"sourceCodeEnd":58,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/fuse_impls_readdir.c#L22-L58","documentation":"Emitted by the fuse-dfs readdir handler when fuseConnectAsThreadUid() cannot obtain a libhdfs connection for the UID making the request; fuse_dfs_readdir returns -EIO, so directory listings on the mount fail with 'Input/output error'. fuseConnectAsThreadUid (fuse_connect.c) resolves the caller UID to a username and creates or reuses a cached hdfsFS via hdfsBuilderConnect, so the real cause is printed on fuse-dfs stderr just above this line.","triggerScenarios":"Running ls/find/glob on the fuse-dfs mount when: (1) getUsername(ctx->uid) fails because the uid has no local passwd entry (returns EIO); (2) Kerberos is configured but the uid's ticket cache (/tmp/krb5cc_<uid> or KRB5CCNAME from the caller's /proc/<pid>/environ) is missing -> fuseNewConnect returns -EACCES; (3) hdfsBuilderConnect fails because the JVM cannot start (broken CLASSPATH), fuse_dfs.conf has a wrong namenode URI/port, or the NameNode is unreachable.","commonSituations":"Kerberized cluster where the accessing user never ran kinit or the ticket expired after mount; fuse_dfs launched without the Hadoop CLASSPATH or with a too-small LIBHDFS_OPTS -Xmx; fuse_dfs.conf pointing at the wrong namenode address; uids that exist in HDFS but have no entry in the local passwd file.","solutions":["Read the fuse-dfs stderr lines immediately above this message - fuseNewConnect/fuseConnect there name the real cause (missing kinit, hdfsBuilderConnect error code, OOM).","If Kerberos: run kinit as the accessing user and confirm the ticket cache file for that uid exists and is fresh (ls /tmp/krb5cc_<uid> or $KRB5CCNAME).","Prove connectivity with the same environment before blaming the mount: CLASSPATH=$(hdfs classpath --glob) hdfs dfs -ls hdfs://<nn>:<port>/","Ensure the accessing uid resolves locally: getent passwd <uid> (fuse-dfs maps uid to username itself).","Restart fuse_dfs with a correct CLASSPATH, sized LIBHDFS_OPTS (-Xmx) and correct fuse_dfs.conf uri/port, then remount."],"exampleFix":"// before\n$ ls /hdfs_mount/data\nls: reading directory /hdfs_mount/data: Input/output error\n\n# fuse-dfs stderr shows the root cause:\n# fuseNewConnect: failed to find Kerberos ticket cache file '/tmp/krb5cc_1001'.\n#   Did you remember to kinit for UID 1001?\n\n// after\n$ kinit alice@EXAMPLE.COM\n$ ls /hdfs_mount/data   # succeeds","handlingStrategy":"validation","validationCode":"# preflight before trusting the fuse-dfs mount (as each accessing user)\nklist -s || kinit \"$USER\"@EXAMPLE.COM\ngetent passwd \"$(id -u)\" >/dev/null || echo \"FATAL: uid has no local passwd entry\"\nCLASSPATH=\"$(hdfs classpath --glob)\" hdfs dfs -ls \"${HDFS_URI:-hdfs://nn:8020/}\" || exit 1\n# only then rely on: ls /hdfs_mount/...","typeGuard":null,"tryCatchPattern":"// users of the mount only see EIO; correlate with fuse-dfs stderr\nDIR *d = opendir(\"/hdfs_mount/data\");\nif (!d && errno == EIO) {\n    /* backend connect failed for this uid: kinit / CLASSPATH / namenode */\n    report_and_retry_after_creds_refresh();\n}","preventionTips":["Keep a valid Kerberos ticket for the mount's lifetime - fuse-dfs condemns cached connections when the ticket cache mtime changes or the file disappears.","Launch fuse_dfs with the full Hadoop CLASSPATH and a sized LIBHDFS_OPTS (-Xmx).","Validate the fuse_dfs.conf namenode URI/port with hdfs dfs -ls before mounting.","Ensure every uid that will use the mount has a local passwd entry.","Monitor fuse-dfs stderr: the fuseNewConnect line above this error carries the actionable cause."],"tags":["c","fuse-dfs","libhdfs","kerberos","connection","eio","readdir"],"backgroundTag":"hdfs-connection-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}