{"record":{"id":"281698a8ba559fc2","repo":"apache/druid","slug":"cannot-read-more-than-d-lines","errorCode":null,"errorMessage":"Cannot read more than %,d lines","messagePattern":"Cannot read more than %,d lines","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"extensions-core/lookups-cached-global/src/main/java/org/apache/druid/data/input/MapPopulator.java","lineNumber":139,"sourceCode":"      final Map<K, V> map,\n      final long byteLimit,\n      final String name\n  ) throws IOException\n  {\n    return source.asCharSource(StandardCharsets.UTF_8).readLines(\n        new LineProcessor<>()\n        {\n          private int lines = 0;\n          private int entries = 0;\n          private long bytes = 0L;\n          private long byteLimitMultiple = 1L;\n          private boolean keyAndValueByteSizesCanBeDetermined = true;\n\n          @Override\n          public boolean processLine(String line)\n          {\n            if (lines == Integer.MAX_VALUE) {\n              throw new ISE(\"Cannot read more than %,d lines\", Integer.MAX_VALUE);\n            }\n            final Map<K, V> kvMap = parser.parseToMap(line);\n            if (kvMap == null) {\n              return true;\n            }\n            map.putAll(kvMap);\n            lines++;\n            entries += kvMap.size();\n            // this top level check so that we dont keep logging inability to determine\n            // byte length for all (key, value) pairs.\n            if (0 < byteLimit && keyAndValueByteSizesCanBeDetermined) {\n              for (Map.Entry<K, V> e : kvMap.entrySet()) {\n                keyAndValueByteSizesCanBeDetermined = canKeyAndValueTypesByteSizesBeDetermined(\n                    e.getKey(),\n                    e.getValue()\n                );\n                if (keyAndValueByteSizesCanBeDetermined) {\n                  bytes += getByteLengthOfObject(e.getKey());","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/lookups-cached-global/src/main/java/org/apache/druid/data/input/MapPopulator.java#L121-L157","documentation":"MapPopulator.processLine counts lines while populating a map from a flat file (used by the global lookup cache loader). The counter is an int; when it reaches Integer.MAX_VALUE it can no longer be incremented, so an IntentionalSQLException-free ISE is thrown instead of silently overflowing. It is effectively a hard cap on lookup file size of ~2.1 billion lines.","triggerScenarios":"Feeding a lookup flatData/parseSpec file with Integer.MAX_VALUE or more lines into MapPopulator.populate; the exception fires on the 2,147,483,647th+ line.","commonSituations":"Pointing a lookup at an entire large dimension/CSV dump instead of a bounded lookup table; misconfigured URI extraction namespace whose poll/refresh reads a huge file into heap memory.","solutions":["Reduce the lookup source file to a bounded set of keys (filter/aggregate the data before publishing it).","Switch to a map-based lookup backed by a database or a different lookup type designed for large datasets.","Increase lookup chunking/splitting of the data file into multiple lookups.","If the file is truly needed in full, do not use the heap-map populator path; consider off-heap or external lookup implementations."],"exampleFix":"// before: lookup points at full 3-billion-line export\n\"flatData\": {\"path\": \"/mnt/full-dimension-export.csv\"}\n// after: pre-filtered lookup file\n\"flatData\": {\"path\": \"/mnt/lookup-keys-only.csv\"} // < 2^31 lines","handlingStrategy":"validation","validationCode":"// bash: count lines before wiring file into a lookup\nlines=$(wc -l < \"$LOOKUP_FILE\")\n[ \"$lines\" -lt 2147483647 ] || echo \"lookup file too large for MapPopulator\"","typeGuard":null,"tryCatchPattern":"// java\ntry { mapPopulator.populate(reader, map); }\ncatch (IllegalStateException e) {\n  if (e.getMessage().contains(\"Cannot read more than\")) {\n    // switch to a db-backed/off-heap lookup\n  }\n}","preventionTips":["Pre-filter lookup source files to the needed key set.","Alert on lookup file size/line count before publishing.","Prefer database or off-heap lookups for very large datasets."],"tags":["lookup","memory","line-limit","int-overflow"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}