What causes "Delete of lock directory %s was unsuccessful" in apache/beam?

Verify HDFS connectivity (fs.defaultFS) and that the NameNode is healthy before releasing.

apache/beam · error · IllegalStateException

Delete of lock directory

Error message

Delete of lock directory %s was unsuccessful

What it means

HDFSSynchronization.releaseJobIdLock() deletes the job's lock directory on HDFS; if the delete fails with an IOException it wraps it in an IllegalStateException("Delete of lock directory %s was unsuccessful"). This indicates cleanup of synchronization state failed.

Solutions

Verify HDFS connectivity (fs.defaultFS) and that the NameNode is healthy before releasing.
Check that the submitting user has write/delete permission on the lock directory; adjust permissions or run as the correct principal.
Make release idempotent/tolerant: check path existence first and treat already-deleted as success.
Retry the release with backoff; if the lock is stale, delete it manually via 'hdfs dfs -rm -r'.

Example fix

// before
sync.releaseJobIdLock(conf);
// after
try {
  sync.releaseJobIdLock(conf);
} catch (IllegalStateException e) {
  LOG.warn("Lock release failed; will retry/stale-clean", e);
  // inspect lock dir manually and remove if stale
}

Defensive patterns

Strategy: retry

Validate before calling

if (!fs.exists(lockPath)) return; // already released
checkHdfsHealthy(fs);

Type guard

null

Try / catch

try { sync.releaseJobIdLock(conf); } catch (IllegalStateException e) { LOG.warn("Lock release failed; inspect {} manually", lockPath, e); }

Prevention

Ensure the job user has delete permission on the sync directory
Verify NameNode health and fs.defaultFS before cleanup
Make lock release idempotent and retried by an external cleanup job

When it happens

Trigger: Calling releaseJobIdLock(conf) when the HDFS delete of the lock directory path throws IOException — e.g. NameNode unreachable, permission denied on the lock dir, or the directory was already removed by another process mid-delete.

Common situations: NameNode outages or network partitions during job cleanup; insufficient HDFS permissions for the job user on the sync directory; concurrent workers releasing the same lock; quota or standby NameNode failover issues.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3464c0e617770b4d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java:105

    return tryCreateFile(conf, path);
  }

  @Override
  public void releaseJobIdLock(Configuration conf) {
    Path path = new Path(locksDir, String.format(LOCKS_DIR_PATTERN, getJobJtIdentifier(conf)));

    try (FileSystem fileSystem = fileSystemFactory.apply(conf)) {
      if (fileSystem.delete(path, true)) {
        LOG.info("Delete of lock directory {} was successful", path);
      } else {
        LOG.warn("Delete of lock directory {} was unsuccessful", path);
      }

    } catch (IOException e) {
      String formattedExceptionMessage =
          String.format("Delete of lock directory %s was unsuccessful", path);
      throw new IllegalStateException(formattedExceptionMessage, e);
    }
  }

  @Override
  public TaskID acquireTaskIdLock(Configuration conf) {
    JobID jobId = HadoopFormats.getJobId(conf);
    boolean lockAcquired = false;
    int taskIdCandidate = 0;

    while (!lockAcquired) {
      taskIdCandidate = RANDOM_GEN.nextInt(Integer.MAX_VALUE);
      Path path =
          new Path(
              locksDir,
              String.format(LOCKS_DIR_TASK_PATTERN, getJobJtIdentifier(conf), taskIdCandidate));
      lockAcquired = tryCreateFile(conf, path);
    }

View on GitHub (pinned to 12126d8942)