threadLocalStorageCreate: OOM - Unable to allocate thread lo

Error message

threadLocalStorageCreate: OOM - Unable to allocate thread local state

What it means

threadLocalStorageCreate's malloc for the per-thread ThreadLocalState (env pointer plus exception-string buffers) returned NULL — the C heap is exhausted. getJNIEnv aborts for this thread (its caller also logs 'Unable to create ThreadLocalState') and the current HDFS call fails. This is purely an out-of-memory condition.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix/thread_local_storage.c:157

  snprintf(id, max, "%s:%ld", thr_name_str, thr_id);

  // Release the char*
  (*env)->ReleaseStringUTFChars(env, thr_name, thr_name_str);

done:
  destroyLocalReference(env, thr);
  destroyLocalReference(env, thr_name);

  // Make sure the id is null terminated in case we overflow the max length
  id[max - 1] = '\0';
}

struct ThreadLocalState* threadLocalStorageCreate()
{
  struct ThreadLocalState *state;
  state = (struct ThreadLocalState*)malloc(sizeof(struct ThreadLocalState));
  if (state == NULL) {
    fprintf(stderr,
      "threadLocalStorageCreate: OOM - Unable to allocate thread local state\n");
    return NULL;
  }
  state->lastExceptionStackTrace = NULL;
  state->lastExceptionRootCause = NULL;
  return state;
}

int threadLocalStorageGet(struct ThreadLocalState **state)
{
  int ret = 0;
  if (!gTlsKeyInitialized) {
    ret = pthread_key_create(&gTlsKey, hdfsThreadDestructor);
    if (ret) {
      fprintf(stderr,
        "threadLocalStorageGet: pthread_key_create failed with error %d\n",
        ret);
      return ret;

View on GitHub (pinned to 2add963021)

Solutions

Compare process RSS with cgroup MemoryMax / ulimit -v / -d; raise the limit or restart the leaked process
Profile with valgrind/heaptrack for native leaks if it recurs
Retry the operation after pressure clears; the allocation itself is tiny

Defensive patterns

Strategy: retry

Try / catch

if (!hdfsConnect(nn, port) && errno == ENOMEM) {
    sleep(2);
    fs = hdfsConnect(nn, port); /* one bounded retry */
}

Prevention

Monitor native RSS against cgroup/rlimit memory ceilings
Run periodic leak checks (valgrind/heaptrack) on long-lived daemons
Fail after one bounded retry rather than spinning on ENOMEM

When it happens

Trigger: First libhdfs use on a thread when malloc fails: cgroup memory cap reached, RLIMIT_AS/RLIMIT_DATA hit, or a severe native leak consuming the heap.

Common situations: Containers with tight MemoryMax; long-running daemons with slow leaks; test suites under valgrind's memory overhead.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/67f32e8627171d31. Report an issue: GitHub.