{"record":{"id":"06e4dd2edb50f161","repo":"apache/iceberg","slug":"file-at-offset-contains-records-exceedin-06e4dd","errorCode":null,"errorMessage":"File {} at offset {} contains {} records, exceeding maxRecordsPerMicroBatch limit of {}. This file will be processed entirely to guarantee forward progress. Consider increasing the limit or writing smaller files to avoid unexpected memory usage.","messagePattern":"File (.+?) at offset (.+?) contains (.+?) records, exceeding maxRecordsPerMicroBatch limit of (.+?)\\. This file will be processed entirely to guarantee forward progress\\. Consider increasing the limit or writing smaller files to avoid unexpected memory usage\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java","lineNumber":328,"sourceCode":"        if (filesSeen == 0) {\n          return null;\n        }\n        LOG.debug(\n            \"latestOffset hit file limit at {}, rows: {}, files: {}\",\n            elem.first(),\n            rowsSeen,\n            filesSeen);\n        return elem.first();\n      }\n\n      // Soft limit on rows - include file FIRST, then check\n      rowsSeen += fileRows;\n      filesSeen += 1;\n\n      // Check if we've hit the row limit after including this file\n      if (rowsSeen >= unpackedLimits.getMaxRows()) {\n        if (filesSeen == 1 && rowsSeen > unpackedLimits.getMaxRows()) {\n          LOG.warn(\n              \"File {} at offset {} contains {} records, exceeding maxRecordsPerMicroBatch limit of {}. \"\n                  + \"This file will be processed entirely to guarantee forward progress. \"\n                  + \"Consider increasing the limit or writing smaller files to avoid unexpected memory usage.\",\n              elem.second().file().location(),\n              elem.first(),\n              fileRows,\n              unpackedLimits.getMaxRows());\n        }\n        // Return the offset of the NEXT element (or synthesize tail+1)\n        if (i + 1 < queueSnapshot.size()) {\n          LOG.debug(\n              \"latestOffset hit row limit at {}, rows: {}, files: {}\",\n              queueSnapshot.get(i + 1).first(),\n              rowsSeen,\n              filesSeen);\n          return queueSnapshot.get(i + 1).first();\n        } else {\n          // This is the last element - return tail+1","sourceCodeStart":310,"sourceCodeEnd":346,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java#L310-L346","documentation":"AsyncSparkMicroBatchPlanner.computeLimitedOffset caps micro-batch input at maxRecordsPerMicroBatch, but when a single file alone exceeds the limit it cannot be split here, so it is planned whole to guarantee forward progress. The warning tells the user the memory-safety limit was exceeded and that memory usage may spike. The batch still runs correctly.","triggerScenarios":"A data file in the stream offset range contains more rows than the configured maxRecordsPerMicroBatch; detected in latestOffset while computing the limited batch offset.","commonSituations":"Streaming reads over tables written with very large compaction outputs or unpartitioned bulk backfills; too-low maxRecordsPerMicroBatch setting relative to file sizes.","solutions":["Increase the maxRecordsPerMicroBatch limit to exceed the typical file's record count.","Rewrite/compact the source with smaller files so no single file exceeds the limit.","Accept the warning if occasional large-file batches are tolerable; it is safe but uses more memory."],"exampleFix":"// before\nspark.sql('ALTER TABLE db.t SET TBLPROPERTIES (\\'maxRecordsPerMicroBatch\\'=\\'1000\\')')\n// after\nspark.sql('ALTER TABLE db.t SET TBLPROPERTIES (\\'maxRecordsPerMicroBatch\\'=\\'1000000\\')')","handlingStrategy":"validation","validationCode":"long maxRecords = Long.parseLong(sparkConf.get(\"maxRecordsPerMicroBatch\", \"10000\"));\nif (largestFileRowsInStream(table) > maxRecords) {\n  // raise limit or compact source files\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set maxRecordsPerMicroBatch above your largest expected file's row count.","Keep streaming source files small via regular compaction with a target size.","Monitor the warning rate; a persistent stream of it means limit/file-size mismatch."],"tags":["spark","streaming","memory","microbatch"],"backgroundTag":"value-out-of-range","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}