What causes "getJNIEnv: Unable to create ThreadLocalState " in apache/hadoop?

Treat as OOM: compare process RSS against cgroup MemoryMax / ulimit -v and raise the limit or restart the leaked process

apache/hadoop · critical

getJNIEnv: Unable to create ThreadLocalState

Error message

getJNIEnv: Unable to create ThreadLocalState

What it means

getJNIEnv() could not allocate the per-thread ThreadLocalState: threadLocalStorageCreate() returned NULL because malloc failed (the companion message 'threadLocalStorageCreate: OOM ...' prints just before). This is C-heap exhaustion at the moment of first per-thread initialization; getJNIEnv releases the global lock and returns NULL, failing the current HDFS call.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:818

      return NULL;
    }
    if (state) {
      mutexUnlock(&jvmMutex);

      // Free any stale exception strings.
      free(state->lastExceptionRootCause);
      free(state->lastExceptionStackTrace);
      state->lastExceptionRootCause = NULL;
      state->lastExceptionStackTrace = NULL;

      return state->env;
    }

    /* Create a ThreadLocalState for this thread */
    state = threadLocalStorageCreate();
    if (!state) {
      mutexUnlock(&jvmMutex);
      fprintf(stderr, "getJNIEnv: Unable to create ThreadLocalState\n");
      return NULL;
    }

    state->env = getGlobalJNIEnv();
    if (!state->env) {
        mutexUnlock(&jvmMutex);
        goto fail;
    }

    jthrowable jthr = NULL;
    jthr = initCachedClasses(state->env);
    if (jthr) {
      mutexUnlock(&jvmMutex);
      printExceptionAndFree(state->env, jthr, PRINT_EXC_ALL,
                            "initCachedClasses failed");
      goto fail;
    }

View on GitHub (pinned to 2add963021)

Solutions

Treat as OOM: compare process RSS against cgroup MemoryMax / ulimit -v and raise the limit or restart the leaked process
Run under valgrind or ASan to find the native leak if it recurs
Retry once memory pressure passes; the allocation is small and transient

Defensive patterns

Strategy: retry

Try / catch

hdfsFS fs = hdfsConnectAsUser(nn, port, user);
if (!fs && errno == ENOMEM) {
    sleep(1);              /* let memory pressure pass */
    fs = hdfsConnectAsUser(nn, port, user);
}
if (!fs) return EXIT_FAILURE;

Prevention

Cap concurrent worker threads so per-thread allocations stay bounded
Monitor RSS against container MemoryMax and alert before the OOM boundary
Retry ENOMEM once, then fail loudly instead of looping

When it happens

Trigger: First libhdfs call on a thread when malloc of sizeof(struct ThreadLocalState) fails: cgroup memory cap reached, RLIMIT_AS/RLIMIT_DATA hit, or a severe native leak.

Common situations: Long-running daemons that slowly leak until the container OOM boundary; stress tests under valgrind overhead; containers with tight MemoryMax.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/aaff7b36c50a145d. Report an issue: GitHub.