What causes "vecsum failed with error %d\n" in apache/hadoop?

Scan stderr upward for the first specific message (hdfsOpenFile failed / hadoopReadZero failed / partial read / malloc failed) — fix that.

apache/hadoop · error

vecsum failed with error %d\n

Error message

vecsum failed with error %d\n

What it means

Top-level aggregator: after dispatching to the selected mode (local / libhdfs / zcr), main() prints this with the propagated error code and jumps to cleanup, skipping the stopwatch report. The actual failure was printed by the mode-specific code earlier on stderr.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/vecsum.c:804

            goto done;
    }
    watch = stopwatch_create();
    if (!watch)
        goto done;
    switch (opts->ty) {
    case VECSUM_LOCAL:
        vecsum_local(cdata, opts);
        ret = 0;
        break;
    case VECSUM_LIBHDFS:
        ret = vecsum_libhdfs(ldata, opts);
        break;
    case VECSUM_ZCR:
        ret = vecsum_zcr(ldata, opts);
        break;
    }
    if (ret) {
        fprintf(stderr, "vecsum failed with error %d\n", ret);
        goto done;
    }
    ret = 0;
done:
    fprintf(stderr, "cleaning up...\n");
    if (watch && (ret == 0)) {
        long long length = vecsum_length(opts, ldata);
        if (length >= 0) {
            stopwatch_stop(watch, length * opts->passes);
        }
    }
    if (cdata)
        local_data_free(cdata);
    if (ldata)
        libhdfs_data_free(ldata);
    if (opts)
        options_free(opts);
    return ret;

View on GitHub (pinned to 2add963021)

Solutions

Scan stderr upward for the first specific message (hdfsOpenFile failed / hadoopReadZero failed / partial read / malloc failed) — fix that.
If the cause is config-related, test with -t local to confirm the harness itself works, then move to -t libhdfs / -t zcr.
Re-run once the underlying issue is resolved.

Defensive patterns

Strategy: try-catch

Validate before calling

/* fail fast on the common deterministic causes before dispatching */
if (opts->ty != VECSUM_LOCAL && hdfsExists(fs, opts->path) != 0) return ENOENT;
if (opts->ty != VECSUM_LOCAL && opts->length % VECSUM_CHUNK_SIZE) return EINVAL;

Try / catch

switch (opts->ty) {
case VECSUM_LOCAL:  vecsum_local(cdata, opts); ret = 0; break;
case VECSUM_LIBHDFS: ret = vecsum_libhdfs(ldata, opts); break;
case VECSUM_ZCR:     ret = vecsum_zcr(ldata, opts); break;
}
if (ret) { /* ret is the propagated errno: log the FIRST specific stderr line
              above, classify (config vs transient vs alignment), then decide
              whether a restart of the run is worthwhile */ }

Prevention

Treat this code as a summary: always diagnose the first mode-specific error line above it.
Wrap benchmark scripts so a nonzero exit maps to the underlying category (config/alignment/cluster) for reporting.
Run a tiny smoke invocation (small aligned -l, 1 pass) before long benchmark runs to surface config errors early.

When it happens

Trigger: Any nonzero return from vecsum_local, vecsum_libhdfs or vecsum_zcr: file open failure, malloc failure, read failure, partial read, or zero-copy setup failure.

Common situations: First run of vecsum on a new cluster (bad paths/config); runs interrupted by cluster events; unaligned length arguments.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/907379ca3ea706f5. Report an issue: GitHub.