apache/hadoop · error · IOException

Unable to release chunk at path: {chunkFilePath}

Error message

Unable to release chunk at path: {chunkFilePath}

What it means

With the dynamic strategy, DistCp splits the listing into chunk files; DynamicInputChunk.release() closes the chunk and deletes its file, throwing when FileSystem.delete(chunkFilePath, false) returns false. delete() returns false when the file no longer exists or permission is lacking, so this most often means a retried task already removed the chunk.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputChunk.java:124

  private void openForRead(TaskAttemptContext taskAttemptContext)
          throws IOException, InterruptedException {
    reader = new SequenceFileRecordReader<K, V>();
    reader.initialize(new FileSplit(chunkFilePath, 0,
            DistCpUtils.getFileSize(chunkFilePath,
                chunkContext.getConfiguration()), null), taskAttemptContext);
  }

  /**
   * Method to be called to relinquish an acquired chunk. All streams open to
   * the chunk are closed, and the chunk-file is deleted.
   * @throws IOException Exception thrown on failure to release (i.e. delete)
   * the chunk file.
   */
  public void release() throws IOException {
    close();
    if (!chunkContext.getFs().delete(chunkFilePath, false)) {
      LOG.error("Unable to release chunk at path: " + chunkFilePath);
      throw new IOException("Unable to release chunk at path: " +
          chunkFilePath);
    }
  }

  /**
   * Getter for the chunk-file's path, on HDFS.
   * @return The qualified path to the chunk-file.
   */
  public Path getPath() {
    return chunkFilePath;
  }

  /**
   * Getter for the record-reader, opened to the chunk-file.
   * @return Opened Sequence-file reader.
   */
  public SequenceFileRecordReader<K,V> getReader() {
    assert reader != null : "Reader un-initialized!";

View on GitHub (pinned to 2add963021)

Solutions

1. Usually transient: re-run the job and give each distcp run its own meta folder
2. Ensure the job user keeps write+delete permission on the chunk directory for the job's lifetime
3. Disable speculative execution for distcp if chunk races recur (mapreduce.map.speculative=false)
4. Exclude the distcp staging area from automated cleanup

if (chunkContext.getFs().exists(chunk.getPath())) {
  chunk.release();
}

try {
  chunk.release();
} catch (IOException e) {
  // commonly a retried attempt already deleted the chunk
  LOG.warn("chunk release failed (already deleted?): {}", chunk.getPath(), e);
}

When it happens

Trigger: A task-attempt retry releases a chunk that another attempt already deleted; permission on the chunk/meta directory changed mid-job; the meta folder was cleaned externally; filesystem state diverged after HA failover.

Common situations: Speculative execution or task retries racing on the same dynamic chunk; a meta folder shared by concurrent distcp jobs; cleanup scripts sweeping the staging directory.

Related errors

• Too many chunks created with splitRatio:{splitRatio}, numMap
• key + ": No such file or directory."
• Directory {} is not empty.
• Can not delete root path
• Can not delete the directory: [%s], as it is not empty and o

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/05296cfb0537fcce. Report an issue: GitHub.