What causes "calcEffectiveURI: out of memory" in apache/hadoop?

Compare process RSS against cgroup/rlimit limits; the URI allocation is the victim, not the cause.

ENOMEM

ENOMEM

Error message

calcEffectiveURI: out of memory

What it means

calcEffectiveURI mallocs the assembled 'hdfs://' + namenode + ':port' string before the JVM is contacted; when that allocation fails, hdfsBuilderConnect aborts with ENOMEM and returns NULL. This is client-side memory exhaustion in the process embedding libhdfs - the allocation itself is tiny, so the heap was already exhausted.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:668

        return EINVAL;
    scheme = (strstr(bld->nn, "://")) ? "" : "hdfs://";
    if (bld->port == 0) {
        suffix[0] = '\0';
    } else {
        lastColon = strrchr(bld->nn, ':');
        if (lastColon && (strspn(lastColon + 1, "0123456789") ==
                          strlen(lastColon + 1))) {
            fprintf(stderr, "port %d was given, but URI '%s' already "
                "contains a port!\n", bld->port, bld->nn);
            return EINVAL;
        }
        snprintf(suffix, sizeof(suffix), ":%d", bld->port);
    }

    uriLen = strlen(scheme) + strlen(bld->nn) + strlen(suffix);
    u = malloc((uriLen + 1) * (sizeof(char)));
    if (!u) {
        fprintf(stderr, "calcEffectiveURI: out of memory");
        return ENOMEM;
    }
    snprintf(u, uriLen + 1, "%s%s%s", scheme, bld->nn, suffix);
    *uri = u;
    return 0;
}

static const char *maybeNull(const char *str)
{
    return str ? str : "(NULL)";
}

static const char *hdfsBuilderToStr(const struct hdfsBuilder *bld,
                                    char *buf, size_t bufLen)
{
    snprintf(buf, bufLen, "forceNewInstance=%d, nn=%s, port=%d, "
             "kerbTicketCachePath=%s, userName=%s",
             bld->forceNewInstance, maybeNull(bld->nn), bld->port,

View on GitHub (pinned to 2add963021)

Solutions

Compare process RSS against cgroup/rlimit limits; the URI allocation is the victim, not the cause.
Profile for leaks (valgrind/heap profilers) in the embedding process - libhdfs is rarely the consumer.
Raise memory limits or restart the leaking service, then retry the connect.
On 32-bit hosts, move to 64-bit - this allocation is only the first of many that will fail under pressure.

Defensive patterns

Strategy: retry

Validate before calling

// cheap preflight: fail or defer before calling connect when memory is gone
void *probe = malloc(4096);
if (!probe) {
    /* heap exhausted: shed load instead of calling hdfsBuilderConnect */
    return -ENOMEM;
}
free(probe);
hdfsFS fs = hdfsConnect(nn, port);

Try / catch

hdfsFS fs = hdfsBuilderConnect(bld);
if (!fs && errno == ENOMEM) {
    release_native_caches();          /* application-level relief */
    fs = hdfsBuilderConnect(hdfsNewBuilder_configured());  /* rebuild + retry once */
}
if (!fs) {
    /* permanent failure: report, do not loop */
}

Prevention

Set memory limits (cgroup/rlimit) with headroom above measured peak so trivial allocations never fail.
Fix leaks in the embedding process; libhdfs connection setup allocates almost nothing.
Retry ENOMEM at most once after releasing memory - repeated retries just thrash.
On 32-bit builds under pressure, move to 64-bit.

When it happens

Trigger: hdfsConnect/hdfsConnectAsUser/hdfsBuilderConnect while the C heap of the embedding process is exhausted: rlimit-as/cgroup memory limits hit, a leak elsewhere in the process, or a 32-bit address space fully mapped.

Common situations: Long-running services with slow leaks that finally surface on connect; fuse_dfs on memory-capped hosts; 32-bit clients with large mappings; OOM-adjacent incidents where the first visible failure is a trivial allocation.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f421f3934bc693e5. Report an issue: GitHub.