What causes "hadoopRzOptionsAlloc failed.\n" in apache/hadoop?

Check ulimit -v and raise it or remove it; inside containers, raise the memory limit.

ENOMEM

ENOMEM

Error message

hadoopRzOptionsAlloc failed.\n

What it means

hadoopRzOptionsAlloc() only calloc()s a small hadoopRzOptions struct; a NULL return means the allocation failed. vecsum maps this directly to ENOMEM and skips zero-copy setup.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/vecsum.c:625

    }
    printf("finished zcr pass %d.  sum = %g\n", pass, sum);
    ret = 0;

done:
    if (rzbuf)
        hadoopRzBufferFree(ldata->file, rzbuf);
    return ret;
}

static int vecsum_zcr(struct libhdfs_data *ldata,
        const struct options *opts)
{
    int ret, pass;
    struct hadoopRzOptions *zopts = NULL;

    zopts = hadoopRzOptionsAlloc();
    if (!zopts) {
        fprintf(stderr, "hadoopRzOptionsAlloc failed.\n");
        ret = ENOMEM;
        goto done;
    }
    if (hadoopRzOptionsSetSkipChecksum(zopts, 1)) {
        ret = errno;
        perror("hadoopRzOptionsSetSkipChecksum failed: ");
        goto done;
    }
    if (hadoopRzOptionsSetByteBufferPool(zopts, NULL)) {
        ret = errno;
        perror("hadoopRzOptionsSetByteBufferPool failed: ");
        goto done;
    }
    for (pass = 0; pass < opts->passes; ++pass) {
        ret = vecsum_zcr_loop(pass, ldata, zopts, opts);
        if (ret) {
            fprintf(stderr, "vecsum_zcr_loop pass %d failed "
                "with error %d\n", pass, ret);

View on GitHub (pinned to 2add963021)

Solutions

Check ulimit -v and raise it or remove it; inside containers, raise the memory limit.
Free memory on the host and retry; check dmesg for OOM-killer activity.
If persistent, reduce other memory usage of the benchmark (fewer passes, smaller -l length).

Defensive patterns

Strategy: retry

Validate before calling

/* cheap preflight: can we still allocate at all? */
void *p = malloc(4096);
if (!p) { /* host is out of memory — do not start the run */ }
free(p);

Try / catch

zopts = hadoopRzOptionsAlloc();
if (!zopts) {
    if (errno == ENOMEM) { sleep(1); zopts = hadoopRzOptionsAlloc(); } /* one bounded retry */
    if (!zopts) { /* give up with ENOMEM */ }
}

Prevention

Run native benchmarks with realistic memory limits (check ulimit -v and container caps beforehand).
Prefer one bounded retry on ENOMEM for tiny allocations; persistent failure means systemic memory pressure.
Monitor host memory before launching concurrent benchmark instances.

When it happens

Trigger: The process cannot allocate memory at calloc() time: address-space limits (ulimit -v), cgroup/container memory caps, overcommit settings, or a host already deep into memory pressure. Because the struct is tiny, seeing this error indicates severe systemic memory exhaustion, not a big request.

Common situations: Running native libhdfs benchmarks inside containers with tight memory limits; test hosts with restrictive RLIMIT_AS; ulimit -v set by CI sandboxes.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/744b2765c356a88a. Report an issue: GitHub.