apache/hadoop · error · HadoopIllegalArgumentException

The LAZY_PERSIST storage policy has been disabled by the adm

Error message

The LAZY_PERSIST storage policy has been disabled by the administrator.

What it means

When a file is created with CreateFlag.LAZY_PERSIST, setNewINodeStoragePolicy looks up the LAZY_PERSIST BlockStoragePolicy to stamp on the new inode. If storage policies are administratively disabled (dfs.storage.policy.enabled=false on the NameNode), getStoragePolicy returns null and creation fails with HadoopIllegalArgumentException before the file exists.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java:825

    BlockUnderConstructionFeature uc = b.getUnderConstructionFeature();
    if (uc != null) {
      uc.appendUCPartsConcise(sb);
    }
    sb.append(" for ").append(src);
    NameNode.stateChangeLog.info(sb.toString());
  }

  private static void setNewINodeStoragePolicy(BlockManager bm,
      INodesInPath iip, boolean isLazyPersist) throws IOException {
    INodeFile inode = iip.getLastINode().asFile();
    if (isLazyPersist) {
      BlockStoragePolicy lpPolicy =
          bm.getStoragePolicy("LAZY_PERSIST");

      // Set LAZY_PERSIST storage policy if the flag was passed to
      // CreateFile.
      if (lpPolicy == null) {
        throw new HadoopIllegalArgumentException(
            "The LAZY_PERSIST storage policy has been disabled " +
            "by the administrator.");
      }
      inode.setStoragePolicyID(lpPolicy.getId(),
                                 iip.getLatestSnapshotId());
    } else {
      BlockStoragePolicy effectivePolicy =
          bm.getStoragePolicy(inode.getStoragePolicyID());

      if (effectivePolicy != null &&
          effectivePolicy.isCopyOnCreateFile()) {
        // Copy effective policy from ancestor directory to current file.
        inode.setStoragePolicyID(effectivePolicy.getId(),
                                 iip.getLatestSnapshotId());
      }
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Drop the LAZY_PERSIST flag and create with plain CREATE
2. Ask admins to enable dfs.storage.policy.enabled=true in hdfs-site.xml (NameNode restart; cluster-wide behavior change)
3. If lazy persist is required, also ensure datanodes expose RAM_DISK volumes in dfs.datanode.data.dir

Example fix

// before
FSDataOutputStream out = fs.create(path,
    EnumSet.of(CreateFlag.CREATE, CreateFlag.LAZY_PERSIST), 4096);

// after
FSDataOutputStream out = fs.create(path,
    EnumSet.of(CreateFlag.CREATE), 4096);

boolean policiesEnabled = conf.getBoolean("dfs.storage.policy.enabled", true);
EnumSet<CreateFlag> flags = policiesEnabled
    ? EnumSet.of(CreateFlag.CREATE, CreateFlag.LAZY_PERSIST)
    : EnumSet.of(CreateFlag.CREATE);

try {
  out = fs.create(path, EnumSet.of(CreateFlag.CREATE, CreateFlag.LAZY_PERSIST), 4096);
} catch (HadoopIllegalArgumentException e) {
  // LAZY_PERSIST disabled cluster-wide: fall back to a normal create
  out = fs.create(path, EnumSet.of(CreateFlag.CREATE), 4096);
}

When it happens

Trigger: fs.create(path, EnumSet.of(CreateFlag.CREATE, CreateFlag.LAZY_PERSIST), ...) against a NameNode with dfs.storage.policy.enabled=false.

Common situations: Clusters where admins disabled the storage-policy feature; frameworks or examples that unconditionally pass LAZY_PERSIST; porting applications between clusters with different policy settings.

Related errors

• "Cannot append to lazy persist file " + path
• "Policy " + newPolicy + " cannot be set after file creation.
• "Existing policy " + currentPolicy.getName() + " cannot be c
• {} doesn't support satisfyStoragePolicy
• {} doesn't support setStoragePolicy

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6d1d4bc36e6582d4. Report an issue: GitHub.