{"record":{"id":"3464c0e617770b4d","repo":"apache/beam","slug":"delete-of-lock-directory-s-was-unsuccessful","errorCode":null,"errorMessage":"Delete of lock directory %s was unsuccessful","messagePattern":"Delete of lock directory (.+?) was unsuccessful","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java","lineNumber":105,"sourceCode":"\n    return tryCreateFile(conf, path);\n  }\n\n  @Override\n  public void releaseJobIdLock(Configuration conf) {\n    Path path = new Path(locksDir, String.format(LOCKS_DIR_PATTERN, getJobJtIdentifier(conf)));\n\n    try (FileSystem fileSystem = fileSystemFactory.apply(conf)) {\n      if (fileSystem.delete(path, true)) {\n        LOG.info(\"Delete of lock directory {} was successful\", path);\n      } else {\n        LOG.warn(\"Delete of lock directory {} was unsuccessful\", path);\n      }\n\n    } catch (IOException e) {\n      String formattedExceptionMessage =\n          String.format(\"Delete of lock directory %s was unsuccessful\", path);\n      throw new IllegalStateException(formattedExceptionMessage, e);\n    }\n  }\n\n  @Override\n  public TaskID acquireTaskIdLock(Configuration conf) {\n    JobID jobId = HadoopFormats.getJobId(conf);\n    boolean lockAcquired = false;\n    int taskIdCandidate = 0;\n\n    while (!lockAcquired) {\n      taskIdCandidate = RANDOM_GEN.nextInt(Integer.MAX_VALUE);\n      Path path =\n          new Path(\n              locksDir,\n              String.format(LOCKS_DIR_TASK_PATTERN, getJobJtIdentifier(conf), taskIdCandidate));\n      lockAcquired = tryCreateFile(conf, path);\n    }\n","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java#L87-L123","documentation":"HDFSSynchronization.releaseJobIdLock() deletes the job's lock directory on HDFS; if the delete fails with an IOException it wraps it in an IllegalStateException(\"Delete of lock directory %s was unsuccessful\"). This indicates cleanup of synchronization state failed.","triggerScenarios":"Calling releaseJobIdLock(conf) when the HDFS delete of the lock directory path throws IOException — e.g. NameNode unreachable, permission denied on the lock dir, or the directory was already removed by another process mid-delete.","commonSituations":"NameNode outages or network partitions during job cleanup; insufficient HDFS permissions for the job user on the sync directory; concurrent workers releasing the same lock; quota or standby NameNode failover issues.","solutions":["Verify HDFS connectivity (fs.defaultFS) and that the NameNode is healthy before releasing.","Check that the submitting user has write/delete permission on the lock directory; adjust permissions or run as the correct principal.","Make release idempotent/tolerant: check path existence first and treat already-deleted as success.","Retry the release with backoff; if the lock is stale, delete it manually via 'hdfs dfs -rm -r'."],"exampleFix":"// before\nsync.releaseJobIdLock(conf);\n// after\ntry {\n  sync.releaseJobIdLock(conf);\n} catch (IllegalStateException e) {\n  LOG.warn(\"Lock release failed; will retry/stale-clean\", e);\n  // inspect lock dir manually and remove if stale\n}","handlingStrategy":"retry","validationCode":"if (!fs.exists(lockPath)) return; // already released\ncheckHdfsHealthy(fs);","typeGuard":"null","tryCatchPattern":"try { sync.releaseJobIdLock(conf); } catch (IllegalStateException e) { LOG.warn(\"Lock release failed; inspect {} manually\", lockPath, e); }","preventionTips":["Ensure the job user has delete permission on the sync directory","Verify NameNode health and fs.defaultFS before cleanup","Make lock release idempotent and retried by an external cleanup job"],"tags":["java","hdfs","lock-cleanup","io"],"backgroundTag":"database-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}