What causes "hdfsRead failed with error %d (%s)\n" in apache/hadoop?

Use the errno text to classify: ECONNRESET/EOF on stream -> DataNode/network issue, check DataNode logs; EACCES/EPERM -> auth/permission; ESTALE/ENOENT -> file removed.

apache/hadoop · error

hdfsRead failed with error %d (%s)\n

Error message

hdfsRead failed with error %d (%s)\n

What it means

In the normal (buffered) read loop vecsum calls hdfsReadFully() for NORMAL_READ_CHUNK_SIZE (8 MiB) bytes; a negative return means the read failed and errno holds the reason (checksum error EIO, connection reset ECONNRESET, lease/auth failures, etc.). The error code is returned up and aborts the pass.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/vecsum.c:689

        length -= ret;
        buf += ret;
    }
    return nread;
}

static int vecsum_normal_loop(int pass, const struct libhdfs_data *ldata,
            const struct options *opts)
{
    double sum = 0.0;

    while (1) {
        int res = hdfsReadFully(ldata->fs, ldata->file, ldata->buf,
                NORMAL_READ_CHUNK_SIZE);
        if (res == 0) // EOF
            break;
        if (res < 0) {
            int err = errno;
            fprintf(stderr, "hdfsRead failed with error %d (%s)\n",
                err, strerror(err));
            return err;
        }
        if (res < NORMAL_READ_CHUNK_SIZE) {
            fprintf(stderr, "hdfsRead got a partial read of "
                "length %d\n", res);
            return EINVAL;
        }
        sum += vecsum(ldata->buf,
                  NORMAL_READ_CHUNK_SIZE / sizeof(double));
    }
    printf("finished normal pass %d.  sum = %g\n", pass, sum);
    return 0;
}

static int vecsum_libhdfs(struct libhdfs_data *ldata,
            const struct options *opts)
{

View on GitHub (pinned to 2add963021)

Solutions

Use the errno text to classify: ECONNRESET/EOF on stream -> DataNode/network issue, check DataNode logs; EACCES/EPERM -> auth/permission; ESTALE/ENOENT -> file removed.
Re-run the benchmark after the cluster is stable; reduce --passes to shorten exposure.
Verify the file still exists and is readable: hdfs dfs -ls <path>.

Defensive patterns

Strategy: retry

Validate before calling

/* before a long benchmark: confirm readability end-to-end */
tSize_t cap = hdfsGetDefaultBlockSize(fs);
hdfsFileInfo *i = hdfsGetPathInfo(fs, path);
if (!i) { /* do not start the run */ }
hdfsFreeFileInfo(i, 1);

Try / catch

int res = hdfsReadFully(fs, file, buf, CHUNK);
if (res < 0) {
    int err = errno;
    if (err == ECONNRESET || err == EIO) {
        /* transient DataNode/network failure: reopen the file and retry the pass */
    } else {
        /* auth/permission/stale: report and stop */
    }
}

Prevention

Schedule long multi-pass runs away from DataNode restarts/decommissioning windows.
Retry at pass granularity: reopen the stream and seek back to offset 0 rather than failing the whole run.
Distinguish retryable errnos (ECONNRESET, EAGAIN-ish) from fatal ones (EACCES, ESTALE) before retrying.

When it happens

Trigger: A DataNode dies or is restarted mid-read, network interruption between client and DataNode, the file is deleted while being read, or a Kerberos token expires during a long benchmark run.

Common situations: Long multi-pass benchmark runs that overlap with cluster maintenance (DN restarts/decommission); flaky networks; running vecsum against a file another job is actively deleting.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/eba571e579bc8aaf. Report an issue: GitHub.