{"record":{"id":"4189a14df57bb9b5","repo":"apache/beam","slug":"creation-of-file-on-path-s-failed","errorCode":null,"errorMessage":"Creation of file on path %s failed","messagePattern":"Creation of file on path (.+?) failed","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java","lineNumber":161,"sourceCode":"\n    return HadoopFormats.createTaskAttemptID(jobId, taskId, taskAttemptCandidate);\n  }\n\n  private boolean tryCreateFile(Configuration conf, Path path) {\n    try (FileSystem fileSystem = fileSystemFactory.apply(conf)) {\n      try {\n        return fileSystem.createNewFile(path);\n      } catch (FileAlreadyExistsException | org.apache.hadoop.fs.FileAlreadyExistsException e) {\n        return false;\n      } catch (RemoteException e) {\n        // remote hdfs exception\n        if (e.getClassName().equals(AlreadyBeingCreatedException.class.getName())) {\n          return false;\n        }\n        throw e;\n      }\n    } catch (IOException e) {\n      throw new IllegalStateException(String.format(\"Creation of file on path %s failed\", path), e);\n    }\n  }\n\n  private String getJobJtIdentifier(Configuration conf) {\n    JobID job =\n        Preconditions.checkNotNull(\n            HadoopFormats.getJobId(conf),\n            \"Configuration must contain jobID under key %s.\",\n            HadoopFormatIO.JOB_ID);\n    return job.getJtIdentifier();\n  }\n\n  /**\n   * Function which can throw exception.\n   *\n   * @param <T1> parameter type\n   * @param <T2> result type\n   * @param <X> exception type","sourceCodeStart":143,"sourceCodeEnd":179,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java#L143-L179","documentation":"HDFSSynchronization.tryCreateFile() creates the lock/flag file on HDFS; any IOException other than AlreadyBeingCreatedException is wrapped in IllegalStateException(\"Creation of file on path %s failed\"). It means the synchronization primitive could not create its marker file.","triggerScenarios":"tryAcquireJobLock, acquireTaskIdLock, or acquireTaskAttemptIdLock invoke tryCreateFile and HDFS create throws IOException — permission denied, parent directory missing, NameNode unavailable, or disk/quota errors.","commonSituations":"Missing HDFS write permissions for the job user; the sync base directory was never created; NameNode in safe mode or failover; HDFS quota exceeded.","solutions":["Verify the parent directory exists and the job user has write permission on it (hdfs dfs -chmod/-chown as needed).","Check NameNode health and safe-mode status; wait for safe mode to exit or fix fs.defaultFS misconfiguration.","Retry acquisition with backoff; transient network errors to HDFS often resolve.","Ensure the synchronization directory is pre-created by setup code before workers start."],"exampleFix":"// before\nTaskID id = sync.acquireTaskIdLock(conf);\n// after\nPath syncDir = sync.getJobLockPath(conf); // ensure exists & writable first\nfs.mkdirs(syncDir);\nTaskID id = sync.acquireTaskIdLock(conf);","handlingStrategy":"retry","validationCode":"Path dir = path.getParent();\nif (!fs.exists(dir)) fs.mkdirs(dir);\nfs.access(dir, FsAction.WRITE); // throws if not writable","typeGuard":"null","tryCatchPattern":"try { sync.acquireTaskIdLock(conf); } catch (IllegalStateException e) { retryWithBackoff(() -> sync.acquireTaskIdLock(conf)); }","preventionTips":["Pre-create the synchronization directory with correct permissions in setup","Check NameNode safe mode before acquiring locks","Use retry with backoff for transient HDFS errors"],"tags":["java","hdfs","file-creation","locking"],"backgroundTag":"file-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}