{"record":{"id":"05296cfb0537fcce","repo":"apache/hadoop","slug":"unable-to-release-chunk-at-path-chunkfilepath","errorCode":null,"errorMessage":"Unable to release chunk at path: {chunkFilePath}","messagePattern":"Unable to release chunk at path: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputChunk.java","lineNumber":124,"sourceCode":"  private void openForRead(TaskAttemptContext taskAttemptContext)\n          throws IOException, InterruptedException {\n    reader = new SequenceFileRecordReader<K, V>();\n    reader.initialize(new FileSplit(chunkFilePath, 0,\n            DistCpUtils.getFileSize(chunkFilePath,\n                chunkContext.getConfiguration()), null), taskAttemptContext);\n  }\n\n  /**\n   * Method to be called to relinquish an acquired chunk. All streams open to\n   * the chunk are closed, and the chunk-file is deleted.\n   * @throws IOException Exception thrown on failure to release (i.e. delete)\n   * the chunk file.\n   */\n  public void release() throws IOException {\n    close();\n    if (!chunkContext.getFs().delete(chunkFilePath, false)) {\n      LOG.error(\"Unable to release chunk at path: \" + chunkFilePath);\n      throw new IOException(\"Unable to release chunk at path: \" +\n          chunkFilePath);\n    }\n  }\n\n  /**\n   * Getter for the chunk-file's path, on HDFS.\n   * @return The qualified path to the chunk-file.\n   */\n  public Path getPath() {\n    return chunkFilePath;\n  }\n\n  /**\n   * Getter for the record-reader, opened to the chunk-file.\n   * @return Opened Sequence-file reader.\n   */\n  public SequenceFileRecordReader<K,V> getReader() {\n    assert reader != null : \"Reader un-initialized!\";","sourceCodeStart":106,"sourceCodeEnd":142,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputChunk.java#L106-L142","documentation":"With the dynamic strategy, DistCp splits the listing into chunk files; DynamicInputChunk.release() closes the chunk and deletes its file, throwing when FileSystem.delete(chunkFilePath, false) returns false. delete() returns false when the file no longer exists or permission is lacking, so this most often means a retried task already removed the chunk.","triggerScenarios":"A task-attempt retry releases a chunk that another attempt already deleted; permission on the chunk/meta directory changed mid-job; the meta folder was cleaned externally; filesystem state diverged after HA failover.","commonSituations":"Speculative execution or task retries racing on the same dynamic chunk; a meta folder shared by concurrent distcp jobs; cleanup scripts sweeping the staging directory.","solutions":["Usually transient: re-run the job and give each distcp run its own meta folder","Ensure the job user keeps write+delete permission on the chunk directory for the job's lifetime","Disable speculative execution for distcp if chunk races recur (mapreduce.map.speculative=false)","Exclude the distcp staging area from automated cleanup"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (chunkContext.getFs().exists(chunk.getPath())) {\n  chunk.release();\n}","typeGuard":null,"tryCatchPattern":"try {\n  chunk.release();\n} catch (IOException e) {\n  // commonly a retried attempt already deleted the chunk\n  LOG.warn(\"chunk release failed (already deleted?): {}\", chunk.getPath(), e);\n}","preventionTips":["Give every distcp run its own meta folder","Disable speculative execution for distcp if chunk races recur (mapreduce.map.speculative=false)","Keep delete permission on the chunk directory for the job's lifetime"],"tags":["distcp","hadoop","dynamic","chunk","delete"],"backgroundTag":"file-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}