apache/hadoop · error · IOException

File has reached the limit on maximum number of blocks (dfs.

Error message

File has reached the limit on maximum number of blocks (dfs.namenode.fs-limits.max-blocks-per-file): <numBlocks> >= <maxBlocksPerFile>

What it means

The NameNode enforces dfs.namenode.fs-limits.max-blocks-per-file (default 1,048,576) on the number of blocks a single file may reference. analyzeFileState throws IOException when the under-construction file already has at least that many blocks, protecting NameNode memory from pathological files. Since the limit is file size divided by block size, small blocks on large files are the usual cause.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java:186

    final byte storagePolicyID;
    String clientMachine;
    final BlockType blockType;

    INodesInPath iip = fsn.dir.resolvePath(pc, src, fileId);
    FileState fileState = analyzeFileState(fsn, iip, fileId, clientName,
                                           previous, onRetryBlock);
    if (onRetryBlock[0] != null && onRetryBlock[0].getLocations().length > 0) {
      // This is a retry. No need to generate new locations.
      // Use the last block if it has locations.
      return null;
    }

    final INodeFile pendingFile = fileState.inode;
    if (!fsn.checkFileProgress(src, pendingFile, false)) {
      throw new NotReplicatedYetException("Not replicated yet: " + src);
    }
    if (pendingFile.getBlocks().length >= fsn.maxBlocksPerFile) {
      throw new IOException("File has reached the limit on maximum number of"
          + " blocks (" + DFSConfigKeys.DFS_NAMENODE_MAX_BLOCKS_PER_FILE_KEY
          + "): " + pendingFile.getBlocks().length + " >= "
          + fsn.maxBlocksPerFile);
    }
    blockSize = pendingFile.getPreferredBlockSize();
    clientMachine = pendingFile.getFileUnderConstructionFeature()
        .getClientMachine();
    blockType = pendingFile.getBlockType();
    ErasureCodingPolicy ecPolicy = null;
    if (blockType == BlockType.STRIPED) {
      ecPolicy =
          FSDirErasureCodingOp.unprotectedGetErasureCodingPolicy(fsn, iip);
      numTargets = (short) (ecPolicy.getSchema().getNumDataUnits()
          + ecPolicy.getSchema().getNumParityUnits());
    } else {
      numTargets = pendingFile.getFileReplication();
    }
    storagePolicyID = pendingFile.getStoragePolicyID();

View on GitHub (pinned to 2add963021)

Solutions

1. Create the file with a bigger block size via fs.create(path, perm, overwrite, bufSize, replication, blockSize, progress) or set dfs.blocksize on the client
2. Raise dfs.namenode.fs-limits.max-blocks-per-file in the NameNode's hdfs-site.xml and restart, weighing NameNode heap usage
3. Split the output into size-rotated files or partitions instead of one unbounded file

Example fix

// before: default block size caps the file at maxBlocks * blockSize
FSDataOutputStream out = fs.create(path, true);

// after: size the block to the expected volume
long expectedBytes = 8L * 1024 * 1024 * 1024 * 1024; // 8 TB
long blockSize = Math.max(128L << 20, (expectedBytes / 1_048_575) + 1);
FSDataOutputStream out = fs.create(path, FsPermission.getFileDefault(),
    true, 4096, (short) 3, blockSize, null);

long maxBlocks = 1_048_576L; // keep in sync with dfs.namenode.fs-limits.max-blocks-per-file
long blockSize = 128L << 20;
if (expectedBytes / blockSize >= maxBlocks) {
  blockSize = (expectedBytes / (maxBlocks - 1)) + 1; // or rotate output files
}

catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("maximum number of")) {
    throw new IllegalStateException(
        "Output too large for one file: raise blockSize or split files", e);
  }
  throw e;
}

When it happens

Trigger: Requesting one more block for a file whose block count has reached the configured maximum: writing more than max-blocks-per-file times blockSize bytes into one file (for example over 1 TiB into a single file with 1 MiB blocks, or over 128 PiB with default 128 MiB blocks).

Common situations: Test clusters with dfs.blocksize set tiny; append-only streams (audit logs, event sinks) running for months into one file; lowering the limit on a namespace that already has bigger files.

Related errors

• "concat: source file " + src + " has preferred block size "
• Cannot truncate to a larger file size. Current size: <oldLen
• The LAZY_PERSIST storage policy has been disabled by the adm
• ConcatDeleteOp can only have {} sources at most.
• Incorrect data format. ConcatDeleteOp can have at most {} so

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e2ac2757bdc255b9. Report an issue: GitHub.