{"record":{"id":"e2ac2757bdc255b9","repo":"apache/hadoop","slug":"file-has-reached-the-limit-on-maximum-number-of-bl","errorCode":null,"errorMessage":"File has reached the limit on maximum number of blocks (dfs.namenode.fs-limits.max-blocks-per-file): <numBlocks> >= <maxBlocksPerFile>","messagePattern":"File has reached the limit on maximum number of blocks \\(dfs\\.namenode\\.fs-limits\\.max-blocks-per-file\\): <numBlocks> >= <maxBlocksPerFile>","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java","lineNumber":186,"sourceCode":"    final byte storagePolicyID;\n    String clientMachine;\n    final BlockType blockType;\n\n    INodesInPath iip = fsn.dir.resolvePath(pc, src, fileId);\n    FileState fileState = analyzeFileState(fsn, iip, fileId, clientName,\n                                           previous, onRetryBlock);\n    if (onRetryBlock[0] != null && onRetryBlock[0].getLocations().length > 0) {\n      // This is a retry. No need to generate new locations.\n      // Use the last block if it has locations.\n      return null;\n    }\n\n    final INodeFile pendingFile = fileState.inode;\n    if (!fsn.checkFileProgress(src, pendingFile, false)) {\n      throw new NotReplicatedYetException(\"Not replicated yet: \" + src);\n    }\n    if (pendingFile.getBlocks().length >= fsn.maxBlocksPerFile) {\n      throw new IOException(\"File has reached the limit on maximum number of\"\n          + \" blocks (\" + DFSConfigKeys.DFS_NAMENODE_MAX_BLOCKS_PER_FILE_KEY\n          + \"): \" + pendingFile.getBlocks().length + \" >= \"\n          + fsn.maxBlocksPerFile);\n    }\n    blockSize = pendingFile.getPreferredBlockSize();\n    clientMachine = pendingFile.getFileUnderConstructionFeature()\n        .getClientMachine();\n    blockType = pendingFile.getBlockType();\n    ErasureCodingPolicy ecPolicy = null;\n    if (blockType == BlockType.STRIPED) {\n      ecPolicy =\n          FSDirErasureCodingOp.unprotectedGetErasureCodingPolicy(fsn, iip);\n      numTargets = (short) (ecPolicy.getSchema().getNumDataUnits()\n          + ecPolicy.getSchema().getNumParityUnits());\n    } else {\n      numTargets = pendingFile.getFileReplication();\n    }\n    storagePolicyID = pendingFile.getStoragePolicyID();","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java#L168-L204","documentation":"The NameNode enforces dfs.namenode.fs-limits.max-blocks-per-file (default 1,048,576) on the number of blocks a single file may reference. analyzeFileState throws IOException when the under-construction file already has at least that many blocks, protecting NameNode memory from pathological files. Since the limit is file size divided by block size, small blocks on large files are the usual cause.","triggerScenarios":"Requesting one more block for a file whose block count has reached the configured maximum: writing more than max-blocks-per-file times blockSize bytes into one file (for example over 1 TiB into a single file with 1 MiB blocks, or over 128 PiB with default 128 MiB blocks).","commonSituations":"Test clusters with dfs.blocksize set tiny; append-only streams (audit logs, event sinks) running for months into one file; lowering the limit on a namespace that already has bigger files.","solutions":["Create the file with a bigger block size via fs.create(path, perm, overwrite, bufSize, replication, blockSize, progress) or set dfs.blocksize on the client","Raise dfs.namenode.fs-limits.max-blocks-per-file in the NameNode's hdfs-site.xml and restart, weighing NameNode heap usage","Split the output into size-rotated files or partitions instead of one unbounded file"],"exampleFix":"// before: default block size caps the file at maxBlocks * blockSize\nFSDataOutputStream out = fs.create(path, true);\n\n// after: size the block to the expected volume\nlong expectedBytes = 8L * 1024 * 1024 * 1024 * 1024; // 8 TB\nlong blockSize = Math.max(128L << 20, (expectedBytes / 1_048_575) + 1);\nFSDataOutputStream out = fs.create(path, FsPermission.getFileDefault(),\n    true, 4096, (short) 3, blockSize, null);","handlingStrategy":"validation","validationCode":"long maxBlocks = 1_048_576L; // keep in sync with dfs.namenode.fs-limits.max-blocks-per-file\nlong blockSize = 128L << 20;\nif (expectedBytes / blockSize >= maxBlocks) {\n  blockSize = (expectedBytes / (maxBlocks - 1)) + 1; // or rotate output files\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"maximum number of\")) {\n    throw new IllegalStateException(\n        \"Output too large for one file: raise blockSize or split files\", e);\n  }\n  throw e;\n}","preventionTips":["Size dfs.blocksize to the expected file volume at create time","Rotate size-bounded files for unbounded streams","Track dfs.namenode.fs-limits.max-blocks-per-file when planning very large files"],"tags":["hdfs","limits","block-size","file-size","namenode-config"],"backgroundTag":"resource-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}