What causes "hdfsOpenFile(%s) failed: error %d (%s)\n" in apache/hadoop?

Read the errno value in the message and resolve it first (ENOENT = wrong/deleted path, EACCES = permissions, ECONNREFUSED/UnknownHost = NameNode config).

apache/hadoop · error

hdfsOpenFile(%s) failed: error %d (%s)\n

Error message

hdfsOpenFile(%s) failed: error %d (%s)\n

What it means

vecsum (libhdfs read microbenchmark) calls hdfsOpenFile(fs, path, O_RDONLY, ...) after ensuring the target file exists at the requested length. If libhdfs returns NULL the tool prints this message with the captured errno and unwinds to its error path, aborting the benchmark run.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/vecsum.c:485

    pinfo = hdfsGetPathInfo(ldata->fs, opts->path);
    if (!pinfo) {
        int err = errno;
        fprintf(stderr, "hdfsGetPathInfo(%s) failed: error %d (%s).  "
                "Attempting to re-create file.\n",
            opts->path, err, strerror(err));
        if (libhdfs_data_create_file(ldata, opts))
            goto error;
    } else if (pinfo->mSize != opts->length) {
        fprintf(stderr, "hdfsGetPathInfo(%s) failed: length was %lld, "
                "but we want length %lld.  Attempting to re-create file.\n",
                opts->path, (long long)pinfo->mSize, (long long)opts->length);
        if (libhdfs_data_create_file(ldata, opts))
            goto error;
    }
    ldata->file = hdfsOpenFile(ldata->fs, opts->path, O_RDONLY, 0, 0, 0);
    if (!ldata->file) {
        int err = errno;
        fprintf(stderr, "hdfsOpenFile(%s) failed: error %d (%s)\n",
            opts->path, err, strerror(err));
        goto error;
    }
    ldata->length = opts->length;
    return ldata;

error:
    if (pinfo)
        hdfsFreeFileInfo(pinfo, 1);
    if (ldata)
        libhdfs_data_free(ldata);
    return NULL;
}

static int check_byte_size(int byte_size, const char *const str)
{
    if (byte_size % sizeof(double)) {
        fprintf(stderr, "%s is not a multiple "

View on GitHub (pinned to 2add963021)

Solutions

Read the errno value in the message and resolve it first (ENOENT = wrong/deleted path, EACCES = permissions, ECONNREFUSED/UnknownHost = NameNode config).
Verify the file exists with the expected size: hdfs dfs -ls <path>.
Confirm connectivity and config used by the native client (CLASSPATH env, fs.defaultFS in the XML on the classpath).
Re-run with a valid Kerberos ticket / correct user if the cluster is secure.

Example fix

# before
vecsum -t libhdfs -p /bad/path/to/file   # hdfsOpenFile failed: error 2 (No such file or directory)
# after
hdfs dfs -ls /bench/vecsum              # confirm path + length, then
vecsum -t libhdfs -p /bench/vecsum

Defensive patterns

Strategy: validation

Validate before calling

/* validate before opening */
if (hdfsExists(fs, path) != 0) {
    fprintf(stderr, "path %s missing; create it first\n", path);
    return -1;
}
hdfsFileInfo *info = hdfsGetPathInfo(fs, path);
if (!info) { /* handle */ }
ldata->file = hdfsOpenFile(fs, path, O_RDONLY, 0, 0, 0);

Try / catch

/* C idiom: NULL check + capture errno immediately (before any other call) */
ldata->file = hdfsOpenFile(fs, path, O_RDONLY, 0, 0, 0);
if (!ldata->file) {
    int err = errno;  /* capture before other syscalls overwrite it */
    switch (err) {
    case ENOENT: /* missing path */ break;
    case EACCES: /* permissions */ break;
    default:     /* connectivity/auth */ break;
    }
}

Prevention

Check hdfsExists()/hdfsGetPathInfo() for the path before hdfsOpenFile().
Capture errno into a local immediately on NULL return; any later libc call can clobber it.
Validate the client config (fs.defaultFS, CLASSPATH) with a trivial hdfsListDirectory call before starting benchmark runs.
In scripts, run vecsum only after confirming the file exists at the expected length via the hdfs CLI.

When it happens

Trigger: hdfsOpenFile() returning NULL: the file was deleted between hdfsGetPathInfo() and the open, permission denied on the path, the NameNode is unreachable or authentication fails, or the libhdfs process is configured with the wrong fs.defaultFS / CLASSPATH so it connects to the wrong cluster.

Common situations: Running the vecsum test binary with a stale CLASSPATH or hdfs-site.xml, typo in the -p path argument, file removed by another process or a previous interrupted run, secure cluster without a valid Kerberos TGT (kinit not done).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0ca23223250cee66. Report an issue: GitHub.