apache/hadoop · error · IOException

Could not set replication for:

Error message

Could not set replication for: 

What it means

SetReplication.processPath (SetReplication.java:92) throws IOException('Could not set replication for: <item>') when FileSystem.setReplication(path, newRep) returns false. A false return (rather than an exception) means the filesystem recognized the call but declined — on HDFS this happens when the file no longer exists or is not in a state that allows replication change (classically, a file still open for write).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SetReplication.java:92

  @Override
  protected void processArguments(LinkedList<PathData> args)
  throws IOException {
    super.processArguments(args);
    if (waitOpt) waitForReplication();
  }

  @Override
  protected void processPath(PathData item) throws IOException {
    if (item.stat.isSymlink()) {
      throw new PathIOException(item.toString(), "Symlinks unsupported");
    }
    
    if (item.stat.isFile()) {
      // Do the checking if the file is erasure coded since
      // replication factor for an EC file is meaningless.
      if (!item.stat.isErasureCoded()) {
        if (!item.fs.setReplication(item.path, newRep)) {
          throw new IOException("Could not set replication for: " + item);
        }
        out.println("Replication " + newRep + " set: " + item);
        if (waitOpt) {
          waitList.add(item);
        }
      } else {
        out.println("Did not set replication for: " + item
            + ", because it's an erasure coded file.");
      }
    } 
  }

  /**
   * Wait for all files in waitList to have replication number equal to rep.
   */
  private void waitForReplication() throws IOException {
    for (PathData item : waitList) {
      out.print("Waiting for " + item + " ...");

View on GitHub (pinned to 2add963021)

Solutions

1. Retry later for files under construction: once the writer closes the file, setReplication succeeds (or rely on HDFS replicating the close-time factor)
2. Verify the file still exists and is closed: 'hadoop fs -ls <file>' and check for open leases (hdfs fsck -files -blocks -openforwrite)
3. Exclude active write paths from the -setrep sweep and process them after writers finish
4. If deleting/moving jobs race the sweep, re-run the sweep after they complete

Example fix

# before
hadoop fs -setrep -R 3 /ingest/active   # writers still appending
# setrep: Could not set replication for: /ingest/active/part-0007

# after
hdfs fsck /ingest/active -files -openforwrite   # confirm open files
hadoop fs -setrep -R 3 /ingest/archive          # set on settled dirs now,
hadoop fs -setrep -R 3 /ingest/active           # re-run after writers close

// only settled, closed files are setrep-safe
if (!fs.exists(p)) continue;
FileStatus st = fs.getFileStatus(p);
if (st.isFile() && !st.isSymlink() && !st.isErasureCoded()) {
  if (!fs.setReplication(p, rep)) { /* defer: likely open for write */ }
}

catch (IOException e) when 'Could not set replication for' -> defer the file (writer likely still open), re-run setrep after the write completes; do not hammer-retry

When it happens

Trigger: Running -setrep on a file that is currently being written (open lease) — HDFS returns false for files under construction; the file being deleted/moved between enumeration and the call (symlink and isFile checks passed earlier); path type changing to a directory mid-scan.

Common situations: Setting replication on active ingest directories where writers still hold open files; -setrep -R sweeps racing with compaction jobs that rewrite/delete files; automation that assumes all enumerated files remain static.

Related errors

• No such file or directory
• Input/output error
• replication must be >= 1
• Symlinks unsupported
• ${className} does not support positioned read.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e595280e3b0b529b. Report an issue: GitHub.