{"record":{"id":"532fb091f12aed00","repo":"apache/iceberg","slug":"file-at-offset-contains-records-exceedin","errorCode":null,"errorMessage":"File {} at offset {} contains {} records, exceeding maxRecordsPerMicroBatch limit of {}. This file will be processed entirely to guarantee forward progress. Consider increasing the limit or writing smaller files to avoid unexpected memory usage.","messagePattern":"File (.+?) at offset (.+?) contains (.+?) records, exceeding maxRecordsPerMicroBatch limit of (.+?)\\. This file will be processed entirely to guarantee forward progress\\. Consider increasing the limit or writing smaller files to avoid unexpected memory usage\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java","lineNumber":328,"sourceCode":"        if (filesSeen == 0) {\n          return null;\n        }\n        LOG.debug(\n            \"latestOffset hit file limit at {}, rows: {}, files: {}\",\n            elem.first(),\n            rowsSeen,\n            filesSeen);\n        return elem.first();\n      }\n\n      // Soft limit on rows - include file FIRST, then check\n      rowsSeen += fileRows;\n      filesSeen += 1;\n\n      // Check if we've hit the row limit after including this file\n      if (rowsSeen >= unpackedLimits.getMaxRows()) {\n        if (filesSeen == 1 && rowsSeen > unpackedLimits.getMaxRows()) {\n          LOG.warn(\n              \"File {} at offset {} contains {} records, exceeding maxRecordsPerMicroBatch limit of {}. \"\n                  + \"This file will be processed entirely to guarantee forward progress. \"\n                  + \"Consider increasing the limit or writing smaller files to avoid unexpected memory usage.\",\n              elem.second().file().location(),\n              elem.first(),\n              fileRows,\n              unpackedLimits.getMaxRows());\n        }\n        // Return the offset of the NEXT element (or synthesize tail+1)\n        if (i + 1 < queueSnapshot.size()) {\n          LOG.debug(\n              \"latestOffset hit row limit at {}, rows: {}, files: {}\",\n              queueSnapshot.get(i + 1).first(),\n              rowsSeen,\n              filesSeen);\n          return queueSnapshot.get(i + 1).first();\n        } else {\n          // This is the last element - return tail+1","sourceCodeStart":310,"sourceCodeEnd":346,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java#L310-L346","documentation":"During micro-batch offset planning, a single data file already exceeds the maxRecordsPerMicroBatch row limit. The planner cannot split a file, so it processes it whole to guarantee forward progress and logs this warning.","triggerScenarios":"Streaming read with read.limit/max-records-per-microbatch configured smaller than the row count of at least one existing data file; latestOffset -> computeLimitedOffset hits a single file with rowsSeen > maxRows.","commonSituations":"Compaction-less writers producing large files, or users lowering maxRecordsPerMicroBatch for latency without checking file sizes.","solutions":["Increase maxRecordsPerMicroBatch above the largest existing data file's record count","Rewrite/compact input files into smaller files (rewrite_data_files with target-file-size-bytes)","Lower the writer's target file size so new files fit within the batch limit"],"exampleFix":"// before\noption(\"read.limit\", \"1000\")\n// after\noption(\"read.limit\", \"500000\") // or compact large files with rewrite_data_files","handlingStrategy":"validation","validationCode":"// pre-check file sizes before setting the limit\nlong maxFileRecords = files.stream().mapToLong(DataFile::recordCount).max().orElse(0L);\nif (maxFileRecords > maxRecordsPerMicroBatch) { /* raise limit or compact first */ }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set read.limit above the largest existing data file's record count","Run periodic compaction with rewrite_data_files","Keep writer target-file-size-bytes consistent with streaming read limits"],"tags":["spark","streaming","micro-batch","memory"],"backgroundTag":"value-out-of-range","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}