apache/hadoop · critical · IOException

Cannot lock storage {root}. The directory is already locked

Error message

Cannot lock storage {root}. The directory is already locked

What it means

StorageDirectory.lock() gives each HDFS daemon process an exclusive OS-level file lock on <storage-dir>/in_use.lock (FileChannel.tryLock) so two servers never share one storage directory. When tryLock() returns null the lock is already held, and lock() throws this IOException at startup. Note: if isShared() is true the lock is intentionally skipped and this error cannot occur.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java:915

     * <p> Locking is not supported by all file systems.
     * E.g., NFS does not consistently support exclusive locks.
     * 
     * <p> If locking is supported we guarantee exclusive access to the
     * storage directory. Otherwise, no guarantee is given.
     * 
     * @throws IOException if locking fails
     */
    public void lock() throws IOException {
      if (isShared()) {
        LOG.info("Locking is disabled for {}", this.root);
        return;
      }
      FileLock newLock = tryLock();
      if (newLock == null) {
        String msg = "Cannot lock storage " + this.root
          + ". The directory is already locked";
        LOG.info(msg);
        throw new IOException(msg);
      }
      // Don't overwrite lock until success - this way if we accidentally
      // call lock twice, the internal state won't be cleared by the second
      // (failed) lock attempt
      lock = newLock;
    }

    /**
     * Attempts to acquire an exclusive lock on the storage.
     * 
     * @return A lock object representing the newly-acquired lock or
     * <code>null</code> if storage is already locked.
     * @throws IOException if locking fails.
     */
    @SuppressWarnings("resource")
    FileLock tryLock() throws IOException {
      boolean deletionHookAdded = false;
      File lockF = new File(root, STORAGE_FILE_LOCK);

View on GitHub (pinned to 2add963021)

Solutions

Find the holder of <dir>/in_use.lock (lsof +D <storage-dir> or fuser <dir>/in_use.lock) and shut that process down cleanly, then restart
Verify only one NameNode/DataNode instance is configured to use that storage directory (check hdfs-site.xml name.dir/data.dir lists for overlap)
If no live process holds it (typical on NFS), restart the NFS lock manager or move storage to a local filesystem
For HA, confirm the active node fully exited before restarting the standby

Example fix

<!-- before: two daodes share a directory -->
<property><name>dfs.namenode.name.dir</name><value>/mnt/shared/nn</value></property>
<!-- after: each daemon gets its own storage directory -->
<property><name>dfs.namenode.name.dir</name><value>/data/nn</value></property>

Defensive patterns

Strategy: try-catch

Validate before calling

static boolean isStorageDirLocked(File dir) throws IOException {
  File lockF = new File(dir, "in_use.lock");
  try (RandomAccessFile raf = new RandomAccessFile(lockF, "rw");
       FileChannel ch = raf.getChannel()) {
    FileLock l = ch.tryLock();
    if (l == null) return true;
    l.release();
    return false;
  }
}
// run before daemon startup: if (isStorageDirLocked(dir)) abort with holder info

Try / catch

try {
  sd.lock();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("already locked")) {
    // surface which process holds <dir>/in_use.lock (lsof) and abort startup cleanly
  }
  throw e;
}

Prevention

Never list the same directory in two daemons' storage-dir settings
Always stop daemons with the provided scripts so the lock is released
Keep storage directories on local filesystems, not NFS, to avoid stale locks
In HA setups, wait for the old active to exit before starting a replacement

When it happens

Trigger: A second NameNode or DataNode starts with dfs.namenode.name.dir / dfs.datanode.data.dir pointing at a directory whose in_use.lock is held; the previous JVM is still running or was killed without releasing; an HA failover begins before the old process exits; a stale lock is left behind on NFS (where locks are advisory and can outlive the owner).

Common situations: Two daemons configured with overlapping storage directories; accidental double start (supervisor script, docker container restart reusing a mounted volume); NFS-mounted storage directories; a checkpoint/2NN process still holding the directory.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/11252d655950364d. Report an issue: GitHub.