{"record":{"id":"e595280e3b0b529b","repo":"apache/hadoop","slug":"could-not-set-replication-for","errorCode":null,"errorMessage":"Could not set replication for: ","messagePattern":"Could not set replication for: ","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SetReplication.java","lineNumber":92,"sourceCode":"  @Override\n  protected void processArguments(LinkedList<PathData> args)\n  throws IOException {\n    super.processArguments(args);\n    if (waitOpt) waitForReplication();\n  }\n\n  @Override\n  protected void processPath(PathData item) throws IOException {\n    if (item.stat.isSymlink()) {\n      throw new PathIOException(item.toString(), \"Symlinks unsupported\");\n    }\n    \n    if (item.stat.isFile()) {\n      // Do the checking if the file is erasure coded since\n      // replication factor for an EC file is meaningless.\n      if (!item.stat.isErasureCoded()) {\n        if (!item.fs.setReplication(item.path, newRep)) {\n          throw new IOException(\"Could not set replication for: \" + item);\n        }\n        out.println(\"Replication \" + newRep + \" set: \" + item);\n        if (waitOpt) {\n          waitList.add(item);\n        }\n      } else {\n        out.println(\"Did not set replication for: \" + item\n            + \", because it's an erasure coded file.\");\n      }\n    } \n  }\n\n  /**\n   * Wait for all files in waitList to have replication number equal to rep.\n   */\n  private void waitForReplication() throws IOException {\n    for (PathData item : waitList) {\n      out.print(\"Waiting for \" + item + \" ...\");","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SetReplication.java#L74-L110","documentation":"SetReplication.processPath (SetReplication.java:92) throws IOException('Could not set replication for: <item>') when FileSystem.setReplication(path, newRep) returns false. A false return (rather than an exception) means the filesystem recognized the call but declined — on HDFS this happens when the file no longer exists or is not in a state that allows replication change (classically, a file still open for write).","triggerScenarios":"Running -setrep on a file that is currently being written (open lease) — HDFS returns false for files under construction; the file being deleted/moved between enumeration and the call (symlink and isFile checks passed earlier); path type changing to a directory mid-scan.","commonSituations":"Setting replication on active ingest directories where writers still hold open files; -setrep -R sweeps racing with compaction jobs that rewrite/delete files; automation that assumes all enumerated files remain static.","solutions":["Retry later for files under construction: once the writer closes the file, setReplication succeeds (or rely on HDFS replicating the close-time factor)","Verify the file still exists and is closed: 'hadoop fs -ls <file>' and check for open leases (hdfs fsck -files -blocks -openforwrite)","Exclude active write paths from the -setrep sweep and process them after writers finish","If deleting/moving jobs race the sweep, re-run the sweep after they complete"],"exampleFix":"# before\nhadoop fs -setrep -R 3 /ingest/active   # writers still appending\n# setrep: Could not set replication for: /ingest/active/part-0007\n\n# after\nhdfs fsck /ingest/active -files -openforwrite   # confirm open files\nhadoop fs -setrep -R 3 /ingest/archive          # set on settled dirs now,\nhadoop fs -setrep -R 3 /ingest/active           # re-run after writers close","handlingStrategy":"retry","validationCode":"// only settled, closed files are setrep-safe\nif (!fs.exists(p)) continue;\nFileStatus st = fs.getFileStatus(p);\nif (st.isFile() && !st.isSymlink() && !st.isErasureCoded()) {\n  if (!fs.setReplication(p, rep)) { /* defer: likely open for write */ }\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) when 'Could not set replication for' -> defer the file (writer likely still open), re-run setrep after the write completes; do not hammer-retry","preventionTips":["Run -setrep on directories whose writers have closed","Re-run sweeps after ingest finishes instead of during","Confirm with fsck -openforwrite when in doubt"],"tags":["setrep","replication-factor","file-under-construction","hdfs","hadoop-shell"],"backgroundTag":"set-replication-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}