{"record":{"id":"aaf8481e29f6aeba","repo":"apache/druid","slug":"val-d-maxvalue-d-please-don-t-lie-about-max","errorCode":null,"errorMessage":"val[%d] > maxValue[%d], please don't lie about maxValue.  i[%d]","messagePattern":"val\\[(.+?)\\] > maxValue\\[(.+?)\\], please don't lie about maxValue\\.  i\\[(.+?)\\]","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/data/VSizeColumnarInts.java","lineNumber":75,"sourceCode":"  {\n    int numBytes = getNumBytesForMax(maxValue);\n\n    final ByteBuffer buffer = ByteBuffer.allocate((ints.size() * numBytes) + (4 - numBytes));\n    writeToBuffer(buffer, ints, numBytes, maxValue);\n\n    return new VSizeColumnarInts(buffer.asReadOnlyBuffer(), numBytes);\n  }\n\n  private static void writeToBuffer(ByteBuffer buffer, IndexedInts ints, int numBytes, int maxValue)\n  {\n    ByteBuffer helperBuffer = ByteBuffer.allocate(Integer.BYTES);\n    for (int i = 0, size = ints.size(); i < size; i++) {\n      int val = ints.get(i);\n      if (val < 0) {\n        throw new IAE(\"integer values must be positive, got[%d], i[%d]\", val, i);\n      }\n      if (val > maxValue) {\n        throw new IAE(\"val[%d] > maxValue[%d], please don't lie about maxValue.  i[%d]\", val, maxValue, i);\n      }\n\n      helperBuffer.putInt(0, val);\n      buffer.put(helperBuffer.array(), Integer.BYTES - numBytes, numBytes);\n    }\n    buffer.position(0);\n  }\n\n  public static byte getNumBytesForMax(int maxValue)\n  {\n    if (maxValue < 0) {\n      throw new IAE(\"maxValue[%s] must be positive\", maxValue);\n    }\n\n    if (maxValue <= 0xFF) {\n      return 1;\n    } else if (maxValue <= 0xFFFF) {\n      return 2;","sourceCodeStart":57,"sourceCodeEnd":93,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/data/VSizeColumnarInts.java#L57-L93","documentation":"writeToBuffer checks each value against the maxValue declared when the column was created; a value exceeding maxValue would overflow the chosen byte width, corrupting the packed encoding, so it throws this IAE ('don't lie about maxValue'). The declared maxValue must be a true upper bound of all values.","triggerScenarios":"Calling VSizeColumnarInts.fromIndexedInts with a maxValue smaller than the actual maximum in the IndexedInts — e.g. maxValue computed over a sample rather than the full dataset.","commonSituations":"Two-pass dictionary encoding where maxValue came from the first pass, streaming ingestion where later rows exceed earlier max, or off-by-one errors when computing maxValue.","solutions":["Compute maxValue by scanning all values before serialization (use a second pass or the fromIterable factory that computes it)","Increase maxValue to the true maximum and use a byte width that fits it (getNumBytesForMax)","If the true max can exceed 0xFFFFFFFF constraints, switch to VSizeLongSerde / long-based columns"],"exampleFix":"// before: maxValue from partial scan\nint maxValue = sampledMax;\nVSizeColumnarInts.fromIndexedInts(values, maxValue);\n// after: compute true max\nint maxValue = 0; for (int i = 0; i < values.size(); i++) maxValue = Math.max(maxValue, values.get(i));\nVSizeColumnarInts.fromIndexedInts(values, maxValue);","handlingStrategy":"validation","validationCode":"int trueMax = 0; for (int i = 0; i < ints.size(); i++) trueMax = Math.max(trueMax, ints.get(i));\nif (trueMax > maxValue) throw new IllegalArgumentException(\"declared maxValue \" + maxValue + \" < actual \" + trueMax);","typeGuard":"boolean fitsMaxValue(IndexedInts ints, int maxValue) { for (int i = 0; i < ints.size(); i++) { if (ints.get(i) > maxValue) return false; } return true; }","tryCatchPattern":"try { col = VSizeColumnarInts.fromIndexedInts(ints, maxValue); } catch (IAE e) { if (e.getMessage().contains(\"lie about maxValue\")) { maxValue = recomputeMax(ints); col = VSizeColumnarInts.fromIndexedInts(ints, maxValue); } else throw e; }","preventionTips":["Compute maxValue from the full dataset, never a sample or first pass alone","Use the fromIterable factory which derives sizing from the data","Add assertions that declared max >= observed max in encoding utilities"],"tags":["serialization","validation","bounds"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}