apache/druid · error · IllegalStateException

Cannot read more than %,d lines

Error message

Cannot read more than %,d lines

What it means

MapPopulator.processLine counts lines while populating a map from a flat file (used by the global lookup cache loader). The counter is an int; when it reaches Integer.MAX_VALUE it can no longer be incremented, so an IntentionalSQLException-free ISE is thrown instead of silently overflowing. It is effectively a hard cap on lookup file size of ~2.1 billion lines.

Source

Thrown at extensions-core/lookups-cached-global/src/main/java/org/apache/druid/data/input/MapPopulator.java:139

      final Map<K, V> map,
      final long byteLimit,
      final String name
  ) throws IOException
  {
    return source.asCharSource(StandardCharsets.UTF_8).readLines(
        new LineProcessor<>()
        {
          private int lines = 0;
          private int entries = 0;
          private long bytes = 0L;
          private long byteLimitMultiple = 1L;
          private boolean keyAndValueByteSizesCanBeDetermined = true;

          @Override
          public boolean processLine(String line)
          {
            if (lines == Integer.MAX_VALUE) {
              throw new ISE("Cannot read more than %,d lines", Integer.MAX_VALUE);
            }
            final Map<K, V> kvMap = parser.parseToMap(line);
            if (kvMap == null) {
              return true;
            }
            map.putAll(kvMap);
            lines++;
            entries += kvMap.size();
            // this top level check so that we dont keep logging inability to determine
            // byte length for all (key, value) pairs.
            if (0 < byteLimit && keyAndValueByteSizesCanBeDetermined) {
              for (Map.Entry<K, V> e : kvMap.entrySet()) {
                keyAndValueByteSizesCanBeDetermined = canKeyAndValueTypesByteSizesBeDetermined(
                    e.getKey(),
                    e.getValue()
                );
                if (keyAndValueByteSizesCanBeDetermined) {
                  bytes += getByteLengthOfObject(e.getKey());

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Reduce the lookup source file to a bounded set of keys (filter/aggregate the data before publishing it).
  2. Switch to a map-based lookup backed by a database or a different lookup type designed for large datasets.
  3. Increase lookup chunking/splitting of the data file into multiple lookups.
  4. If the file is truly needed in full, do not use the heap-map populator path; consider off-heap or external lookup implementations.

Example fix

// before: lookup points at full 3-billion-line export
"flatData": {"path": "/mnt/full-dimension-export.csv"}
// after: pre-filtered lookup file
"flatData": {"path": "/mnt/lookup-keys-only.csv"} // < 2^31 lines
Defensive patterns

Strategy: validation

Validate before calling

// bash: count lines before wiring file into a lookup
lines=$(wc -l < "$LOOKUP_FILE")
[ "$lines" -lt 2147483647 ] || echo "lookup file too large for MapPopulator"

Try / catch

// java
try { mapPopulator.populate(reader, map); }
catch (IllegalStateException e) {
  if (e.getMessage().contains("Cannot read more than")) {
    // switch to a db-backed/off-heap lookup
  }
}

Prevention

When it happens

Trigger: Feeding a lookup flatData/parseSpec file with Integer.MAX_VALUE or more lines into MapPopulator.populate; the exception fires on the 2,147,483,647th+ line.

Common situations: Pointing a lookup at an entire large dimension/CSV dump instead of a bounded lookup table; misconfigured URI extraction namespace whose poll/refresh reads a huge file into heap memory.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/281698a8ba559fc2. Report an issue: GitHub.