apache/druid · error · IllegalStateException

Values must be sorted and unique. Element

Error message

Values must be sorted and unique. Element [%s] with value [%s] is before or equivalent to [%s]

What it means

FrontCodedIntArrayIndexedWriter.write() enforces that int[] values are supplied in strictly ascending, unique order (compared with ARRAY_COMPARATOR). Front coding relies on lexicographic prefix sharing between consecutive sorted values; writing a value that is <= the previous one would corrupt the encoding, so it throws ISE. Null is allowed only before the first non-null value.

Solutions

1. Sort and deduplicate all int[] values lexicographically before writing them.
2. If the writer receives data from a stream, buffer and sort first, or assert ascending order before calling write().
3. Check that only one writer instance writes sequentially and prevObject state is not bypassed.

Example fix

// before
for (int[] v : values) writer.write(v);
// after
Arrays.sort(values, FrontCodedIntArrayIndexedWriter.ARRAY_COMPARATOR);
for (int i = 0; i < values.length; i++) {
  if (i > 0 && ARRAY_COMPARATOR.compare(values[i-1], values[i]) == 0) continue;
  writer.write(values[i]);
}

for (int i = 1; i < values.length; i++) {
  if (FrontCodedIntArrayIndexedWriter.ARRAY_COMPARATOR.compare(values[i-1], values[i]) >= 0)
    throw new IllegalStateException("values not strictly increasing at " + i);
}

try { writer.write(value); } catch (IllegalStateException e) { throw new IOException("unsorted values supplied to FrontCodedIntArrayIndexedWriter", e); }

When it happens

Trigger: Calling write(int[]) with an array that compares equal to or sorts before the previously written value; interleaving writers; re-writing the same value; or a sorter upstream that emits duplicates.

Common situations: Dictionary/dimension value encoding where the input column was not sorted before encoding, duplicate dimension values slipping past a dedup step, or using the writer directly on unsorted data.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• bucketSize must be a power of two (from 1 up to 128) but…
• no value split found with fileSizeLimit
• scratch buffer to big to write buckets
• 08001
• A batch appenderator was already created for this peon's…

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/9345a2945e3c57fa. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/data/FrontCodedIntArrayIndexedWriter.java:119

    this.byteOrder = byteOrder;
    this.bucketBuffer = new int[bucketSize][];
    this.getOffsetBuffer = ByteBuffer.allocate(Integer.BYTES).order(byteOrder);
    this.div = Integer.numberOfTrailingZeros(bucketSize);
  }

  @Override
  public void open() throws IOException
  {
    headerOut = segmentWriteOutMedium.makeWriteOutBytes();
    valuesOut = segmentWriteOutMedium.makeWriteOutBytes();
  }

  @Override
  public int write(@Nullable int[] value) throws IOException
  {

    if (prevObject != null && ARRAY_COMPARATOR.compare(prevObject, value) >= 0) {
      throw new ISE(
          "Values must be sorted and unique. Element [%s] with value [%s] is before or equivalent to [%s]",
          numWritten,
          value == null ? null : Arrays.toString(value),
          Arrays.toString(prevObject)
      );
    }

    if (value == null) {
      if (numWritten != 0) {
        throw DruidException.defensive("Null must come first, got it at numWritten[%,d]!=0", numWritten);
      }
      hasNulls = true;
      return 0;
    }

    // if the bucket buffer is full, write the bucket
    if (numWritten > 0 && (numWritten % bucketSize) == 0) {
      resetScratch();

View on GitHub (pinned to 9b90983fd2)