What causes "Client (={}) is not the lease owner (={}: {} (inode {}) {}" in apache/hadoop?

Ensure exactly one writer per path: unique temp names per attempt, rename on commit

apache/hadoop · error · LeaseExpiredException

Client (={}) is not the lease owner (={}: {} (inode {}) {}

Error message

Client (={}) is not the lease owner (={}: {} (inode {}) {}

What it means

Thrown as LeaseExpiredException when the holder string supplied by the client does not equal the client name stored in the file's FileUnderConstructionFeature. HDFS enforces single-writer semantics: only the lease owner may append, allocate blocks, or complete the file, so a mismatching client is rejected.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:3274

    if (!inode.isFile()) {
      throw new LeaseExpiredException("INode is not a regular file: "
          + leaseExceptionString(src, fileId, holder));
    }
    final INodeFile file = inode.asFile();
    if (!file.isUnderConstruction()) {
      throw new LeaseExpiredException("File is not open for writing: "
          + leaseExceptionString(src, fileId, holder));
    }
    // No further modification is allowed on a deleted file.
    // A file is considered deleted, if it is not in the inodeMap or is marked
    // as deleted in the snapshot feature.
    if (isFileDeleted(file)) {
      throw new FileNotFoundException("File is deleted: "
          + leaseExceptionString(src, fileId, holder));
    }
    final String owner = file.getFileUnderConstructionFeature().getClientName();
    if (holder != null && !owner.equals(holder)) {
      throw new LeaseExpiredException("Client (=" + holder
          + ") is not the lease owner (=" + owner + ": "
          + leaseExceptionString(src, fileId, holder));
    }
    return file;
  }
 
  /**
   * Complete in-progress write to the given file.
   * @return true if successful, false if the client should continue to retry
   *         (e.g if not all blocks have reached minimum replication yet)
   * @throws IOException on error (eg lease mismatch, file not open, file deleted)
   */
  boolean completeFile(final String src, String holder,
                       ExtendedBlock last, long fileId)
    throws IOException {
    boolean success = false;
    final String operationName = "completeFile";
    checkOperation(OperationCategory.WRITE);

View on GitHub (pinned to 2add963021)

Solutions

Ensure exactly one writer per path: unique temp names per attempt, rename on commit
If the lease must change hands, call recoverLease(path), wait for it to succeed, then re-open the file with append()
Find and stop the zombie process still holding/renewing the old lease
For append workflows, open the file only after confirming lease ownership via getLeaseState/recovery, not by racing the current holder

Example fix

// before
FSDataOutputStream out = dfs.append(path); // may hit LeaseExpiredException on close/addBlock

// after
if (!dfs.recoverLease(path)) { /* wait, then check isFileClosed */ }
while (!dfs.isFileClosed(path)) { Thread.sleep(1000); }
FSDataOutputStream out = dfs.append(path);

Defensive patterns

Strategy: retry

Validate before calling

if (!dfs.isFileClosed(path)) {
  boolean recovered = dfs.recoverLease(path);
  // wait until isFileClosed before appending
}

Try / catch

try {
  out.close();
} catch (LeaseExpiredException e) {
  // lease was taken over: recover, wait, then append to continue
  recoverAndWaitThenAppend(path);
}

Prevention

Guarantee one writer per path (unique client names, single task attempt)
Kill zombie writer processes before restarting jobs on the same output
For read access to open files, use recoverLease rather than racing the writer

When it happens

Trigger: Calling addBlock, completeFile, or updateBlock with a client name that differs from the creating writer; another client already ran recoverLease and took over the file; a zombie writer resumes after failover once its lease was reassigned.

Common situations: Two DFSOutputStreams opened on the same path (duplicate task attempts, a process that never died); append from a differently named client; a slow writer idle past the soft lease limit while another reader triggers recovery.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/086bcaa66e94918b. Report an issue: GitHub.