threadLocalStorageGet: pthread_key_create failed with error

Error message

threadLocalStorageGet: pthread_key_create failed with error %d

What it means

On the first thread-local access libhdfs lazily creates its process-wide pthread TLS key, registering hdfsThreadDestructor. pthread_key_create failed: EAGAIN means PTHREAD_KEYS_MAX (typically 1024) keys already exist in the process; ENOMEM means no memory for the key. threadLocalStorageGet returns the error, so getJNIEnv fails for the calling thread.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix/thread_local_storage.c:172

  struct ThreadLocalState *state;
  state = (struct ThreadLocalState*)malloc(sizeof(struct ThreadLocalState));
  if (state == NULL) {
    fprintf(stderr,
      "threadLocalStorageCreate: OOM - Unable to allocate thread local state\n");
    return NULL;
  }
  state->lastExceptionStackTrace = NULL;
  state->lastExceptionRootCause = NULL;
  return state;
}

int threadLocalStorageGet(struct ThreadLocalState **state)
{
  int ret = 0;
  if (!gTlsKeyInitialized) {
    ret = pthread_key_create(&gTlsKey, hdfsThreadDestructor);
    if (ret) {
      fprintf(stderr,
        "threadLocalStorageGet: pthread_key_create failed with error %d\n",
        ret);
      return ret;
    }
    gTlsKeyInitialized = 1;
  }
  *state = pthread_getspecific(gTlsKey);
  return ret;
}

int threadLocalStorageSet(struct ThreadLocalState *state)
{
  int ret = pthread_setspecific(gTlsKey, state);
  if (ret) {
    fprintf(stderr,
      "threadLocalStorageSet: pthread_setspecific failed with error %d\n",
      ret);
    hdfsThreadDestructor(state);

View on GitHub (pinned to 2add963021)

Solutions

If EAGAIN: find key hogs — trace pthread_key_create calls (ltrace/gdb breakpoint) and reduce or fix libraries that create keys per-init without pthread_key_delete
If ENOMEM: address memory exhaustion (limits or leaks)
Load and initialize libhdfs early so it claims its single key before the space fills
Avoid dlopen/dlclose cycles of libhdfs — the key and JVM state are process-wide and only reclaimed at exit

Defensive patterns

Strategy: validation

Try / catch

/* After initializing libhdfs once, cheap calls confirm TLS is healthy */
if (hdfsConfGetStr("dfs.replication", &val) != 0) {
    /* TLS key path broken; report instead of hammering */
    fprintf(stderr, "libhdfs TLS init failed\n");
}

Prevention

Load and initialize libhdfs early, before plugins exhaust the 1024 TLS-key space
Avoid dlopen/dlclose cycles of libhdfs; treat it as process-wide state
Audit native dependencies that allocate pthread keys per initialization

When it happens

Trigger: Embedding libhdfs in a process whose TLS key space is already exhausted by many other runtime libraries/plugins, or under ENOMEM conditions.

Common situations: Large plugin hosts (scripting-language extensions plus many native libs); libraries that leak keys by re-initializing after dlopen/dlclose cycles; general memory exhaustion.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1fae4d936dbd55c5. Report an issue: GitHub.