apache/hadoop · error · HadoopIllegalArgumentException

"Policy " + newPolicy + " cannot be set after file creation.

Error message

"Policy " + newPolicy + " cannot be set after file creation."

What it means

Policies flagged copyOnCreateFile can only be assigned at file creation because they change where new blocks are placed as the file is written. In the default suite LAZY_PERSIST is the only such policy (BlockStoragePolicySuite marks it 'Cannot be changed on regular files, but inherited'), and applying it to an already-created file throws HadoopIllegalArgumentException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java:449

  }

  static void unprotectedSetStoragePolicy(FSDirectory fsd, BlockManager bm,
      INodesInPath iip, final byte policyId)
      throws IOException {
    assert fsd.hasWriteLock();
    final INode inode = iip.getLastINode();
    if (inode == null) {
      throw new FileNotFoundException("File/Directory does not exist: "
          + iip.getPath());
    }
    final int snapshotId = iip.getLatestSnapshotId();
    if (inode.isFile()) {
      FSDirectory.LOG.debug("DIR* FSDirAAr.unprotectedSetStoragePolicy for " +
              "File.");
      if (policyId != HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
        BlockStoragePolicy newPolicy = bm.getStoragePolicy(policyId);
        if (newPolicy.isCopyOnCreateFile()) {
          throw new HadoopIllegalArgumentException("Policy " + newPolicy
              + " cannot be set after file creation.");
        }
      }

      BlockStoragePolicy currentPolicy =
          bm.getStoragePolicy(inode.getLocalStoragePolicyID());

      if (currentPolicy != null && currentPolicy.isCopyOnCreateFile()) {
        throw new HadoopIllegalArgumentException(
            "Existing policy " + currentPolicy.getName() +
                " cannot be changed after file creation.");
      }
      inode.asFile().setStoragePolicyID(policyId, snapshotId);
    } else if (inode.isDirectory()) {
      FSDirectory.LOG.debug("DIR* FSDirAAr.unprotectedSetStoragePolicy for " +
              "Directory.");
      setDirStoragePolicy(fsd, iip, policyId);
    } else {

View on GitHub (pinned to 2add963021)

Solutions

1. Apply LAZY_PERSIST to the parent directory instead - new children inherit it at creation time (directory policy setting is allowed).
2. For existing data, rewrite: put the directory under the lazy-persist policy (or create with CreateFlag.LAZY_PERSIST), create the new file, copy the bytes.
3. For data that must stay appendable, avoid LAZY_PERSIST entirely - append on lazy-persist files is separately unsupported.

Example fix

# before: rejected, cannot set after creation
hdfs storagepolicies -setStoragePolicy -policy LAZY_PERSIST /data/exists.txt

# after: set on the directory; new files inherit at creation
hdfs storagepolicies -setStoragePolicy -policy LAZY_PERSIST /data

BlockStoragePolicy[] policies = ((DistributedFileSystem) fs).getStoragePolicies();
BlockStoragePolicy target = Arrays.stream(policies)
    .filter(p -> p.getName().equals(policyName)).findFirst().orElse(null);
boolean isExistingFile = fs.exists(path) && fs.getFileStatus(path).isFile();
if (target != null && target.isCopyOnCreateFile() && isExistingFile) {
  // apply to the parent directory instead so children inherit at creation
  fs.setStoragePolicy(path.getParent(), policyName);
} else {
  fs.setStoragePolicy(path, policyName);
}

try {
  fs.setStoragePolicy(path, policyName);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("cannot be set after file creation")) {
    // copy-on-create policy (LAZY_PERSIST): set on the directory or rewrite the file
  }
}

When it happens

Trigger: setStoragePolicy(file, 'LAZY_PERSIST') on an existing file, or 'hdfs storagepolicies -setStoragePolicy -policy LAZY_PERSIST <existing-file>': the target policy isCopyOnCreateFile() and the inode is a regular file (not a directory).

Common situations: Trying to move existing hot data into RAM_DISK to speed up reads; blanket policy rollout scripts applying LAZY_PERSIST across existing trees instead of directories.

Related errors

• "Existing policy " + currentPolicy.getName() + " cannot be c
• "Cannot append to lazy persist file " + path
• The LAZY_PERSIST storage policy has been disabled by the adm
• {} doesn't support satisfyStoragePolicy
• {} doesn't support setStoragePolicy

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2be58b8ee837b7f1. Report an issue: GitHub.