apache/hadoop · error · HadoopIllegalArgumentException

"Existing policy " + currentPolicy.getName() + " cannot be c

Error message

"Existing policy " + currentPolicy.getName() + " cannot be changed after file creation."

What it means

Mirror of the copy-on-create check: if the file already carries a copy-on-create policy (LAZY_PERSIST in the default suite), changing it to any other policy after creation is rejected - placement decisions were made when the blocks were written and cannot be switched retroactively. The message names the existing policy that is locked in.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java:458

          + iip.getPath());
    }
    final int snapshotId = iip.getLatestSnapshotId();
    if (inode.isFile()) {
      FSDirectory.LOG.debug("DIR* FSDirAAr.unprotectedSetStoragePolicy for " +
              "File.");
      if (policyId != HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
        BlockStoragePolicy newPolicy = bm.getStoragePolicy(policyId);
        if (newPolicy.isCopyOnCreateFile()) {
          throw new HadoopIllegalArgumentException("Policy " + newPolicy
              + " cannot be set after file creation.");
        }
      }

      BlockStoragePolicy currentPolicy =
          bm.getStoragePolicy(inode.getLocalStoragePolicyID());

      if (currentPolicy != null && currentPolicy.isCopyOnCreateFile()) {
        throw new HadoopIllegalArgumentException(
            "Existing policy " + currentPolicy.getName() +
                " cannot be changed after file creation.");
      }
      inode.asFile().setStoragePolicyID(policyId, snapshotId);
    } else if (inode.isDirectory()) {
      FSDirectory.LOG.debug("DIR* FSDirAAr.unprotectedSetStoragePolicy for " +
              "Directory.");
      setDirStoragePolicy(fsd, iip, policyId);
    } else {
      throw new FileNotFoundException(iip.getPath()
          + " is not a file or directory");
    }
  }

  private static void setDirStoragePolicy(
      FSDirectory fsd, INodesInPath iip, byte policyId) throws IOException {
    INode inode = FSDirectory.resolveLastINode(iip);
    List<XAttr> existingXAttrs = XAttrStorage.readINodeXAttrs(inode);

View on GitHub (pinned to 2add963021)

Solutions

1. Rewrite the file: create a new file under a directory with the desired policy (or default policy), copy the content, then replace the original.
2. Prevent it earlier: scope LAZY_PERSIST to throwaway scratch data you never intend to migrate.
3. For directories carrying the policy, 'hdfs storagepolicies -unsetStoragePolicy <dir>' stops inheritance for future children (existing files still need rewriting).

Example fix

# before: rejected, existing LAZY_PERSIST policy is locked in
hdfs storagepolicies -setStoragePolicy -policy HOT /scratch/part-0

# after: rewrite via copy under a HOT directory, then swap
hdfs dfs -mkdir -p /staging
hdfs storagepolicies -setStoragePolicy -policy HOT /staging
hdfs dfs -cp /scratch/part-0 /staging/part-0
hdfs dfs -mv /staging/part-0 /scratch/part-0

BlockStoragePolicy current = ((DistributedFileSystem) fs).getStoragePolicy(path);
if (current != null && current.isCopyOnCreateFile()) {
  // locked in at creation: must rewrite into a new file with the desired policy
  Path tmp = new Path(path.getParent(), path.getName() + ".migrate");
  FileUtil.copy(fs, path, fs, tmp, false, true, fs.getConf());
  fs.setStoragePolicy(tmp, "HOT");
  fs.delete(path, false);
  fs.rename(tmp, path);
} else {
  fs.setStoragePolicy(path, "HOT");
}

try {
  fs.setStoragePolicy(path, "HOT");
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("cannot be changed after file creation")) {
    // existing copy-on-create policy: rewrite the file instead of retrying
  }
}

When it happens

Trigger: setStoragePolicy(file, otherPolicy) where the file's current policy is LAZY_PERSIST: e.g. 'hdfs storagepolicies -setStoragePolicy -policy HOT <lazy-persist-file>' when RAM_DISK fills and ops try to migrate files to disk-backed storage.

Common situations: Migrating lazy-persist temp data when memory pressure hits; lifecycle scripts walking trees and 'normalizing' all policies to HOT/WARM/COLD without checking the current one.

Related errors

• "Policy " + newPolicy + " cannot be set after file creation.
• "Cannot append to lazy persist file " + path
• The LAZY_PERSIST storage policy has been disabled by the adm
• {} doesn't support satisfyStoragePolicy
• {} doesn't support setStoragePolicy

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/76d5beea67ffff37. Report an issue: GitHub.