apache/hadoop · error

mutexLock: pthread_mutex_lock failed with error %d

Error message

mutexLock: pthread_mutex_lock failed with error %d

What it means

mutexLock() is libhdfs' wrapper around pthread_mutex_lock; it prints this when the call returns an error instead of blocking: EINVAL (mutex storage not initialized or corrupted), EDEADLK (thread already owns this non-recursive mutex), or EAGAIN (recursion limit on a recursive mutex). The wrapper still returns the error, so callers continue without mutual exclusion held — a correctness hazard beyond the log line.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix/mutexes.c:37

#include "os/mutexes.h"

#include <pthread.h>
#include <stdio.h>

mutex jvmMutex;
mutex jclassInitMutex = PTHREAD_MUTEX_INITIALIZER;
pthread_mutexattr_t jvmMutexAttr;

__attribute__((constructor)) static void init() {
  pthread_mutexattr_init(&jvmMutexAttr);
  pthread_mutexattr_settype(&jvmMutexAttr, PTHREAD_MUTEX_RECURSIVE);
  pthread_mutex_init(&jvmMutex, &jvmMutexAttr);
}

int mutexLock(mutex *m) {
  int ret = pthread_mutex_lock(m);
  if (ret) {
    fprintf(stderr, "mutexLock: pthread_mutex_lock failed with error %d\n",
      ret);
  }
  return ret;
}

int mutexUnlock(mutex *m) {
  int ret = pthread_mutex_unlock(m);
  if (ret) {
    fprintf(stderr, "mutexUnlock: pthread_mutex_unlock failed with error %d\n",
      ret);
  }
  return ret;
}

View on GitHub (pinned to 2add963021)

Solutions

  1. Map the number: EINVAL means uninitialized/corrupted mutex — hunt overruns near the storage with ASan/valgrind; EDEADLK means recursive locking of a non-recursive mutex — restructure the call path
  2. Never call libhdfs APIs re-entrantly from callbacks or signal handlers on a thread that is already inside an HDFS call
  3. If it appears at startup with odd behavior, assume corruption until proven otherwise
  4. Rebuild/upgrade libhdfs if a build-specific init-ordering bug is suspected

Example fix

/* before */
mutexLock(&m);
/* proceeds even when the lock failed */

/* after */
if (mutexLock(&m) != 0) {
    return -1;  /* lock NOT held — bail instead of running unsynchronized */
}
/* ... */
mutexUnlock(&m);
Defensive patterns

Strategy: validation

Try / catch

/* when hacking on libhdfs internals: never ignore the wrapper return */
if (mutexLock(&m) != 0) {
    return -1; /* lock NOT held — do not proceed */
}
/* critical section */
mutexUnlock(&m);

Prevention

When it happens

Trigger: Recursive locking of jclassInitMutex (a plain PTHREAD_MUTEX_INITIALIZER, non-recursive) from the same thread; a mutex whose storage was reinitialized or overwritten; exceeding recursive depth on jvmMutex (initialized PTHREAD_MUTEX_RECURSIVE by the library constructor).

Common situations: Re-entrant call paths through libhdfs from callbacks; heap/stack corruption smashing static mutex storage (EINVAL plus unrelated crashes); unusual static-link or dlopen ordering where the constructor init of jvmMutexAttr never ran.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/93a51501848226f3. Report an issue: GitHub.