apache/druid · error · org.apache.druid.java.util.common.ISE

Must serialize value dictionaries before serializing values…

Error message

Must serialize value dictionaries before serializing values for column [%s]

What it means

ScalarNestedCommonFormatColumnSerializer.serialize() writes dictionary IDs into the intermediate value writer, which only makes sense once the value dictionary is final and serialized. If serialize() is called before serializeDictionaries(), IDs assigned now would not match the dictionary that is written later, so an IllegalStateException is thrown.

Solutions

Ensure serializeDictionaries() runs to completion before any serialize() call for the column
Restructure the pipeline into two passes: first build/serialize dictionaries, then stream values
Restart serialization with a fresh serializer instance if the dictionary phase was skipped or failed

Example fix

// before
serializer.serialize(rowSelector); // ISE: dictionaries not serialized
// after
serializer.serializeDictionaries(strings, longs, doubles, arrays);
serializer.open();
serializer.serialize(rowSelector);

Defensive patterns

Strategy: validation

Validate before calling

// Before streaming values, confirm the dictionary phase has completed
if (!writer.isDictionarySerialized()) {
  throw new IllegalStateException("Run serializeDictionaries() before serializing values for "
      + writer.getColumnName());
}
writer.open();
writer.serialize(selector);

Try / catch

try {
  serializer.serialize(rowSelector);
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().contains("before serializing values")) {
    LOG.error(e, "Value phase ran before dictionary phase for %s; restarting segment write", name);
    restartSegmentWriteFromScratch();
  } else {
    throw e;
  }
}

Prevention

Implement the two-pass structure: pass 1 builds and serializes dictionaries, pass 2 streams values
Never resume value serialization on a freshly created serializer; restart the entire write
Guard custom pipelines with an explicit hasDictionaries flag checked before the value loop
Cover the ordering with unit tests mirroring IndexMergerV9's phase sequence

When it happens

Trigger: Calling serialize(ColumnValueSelector) before serializeDictionaries() has completed for the column; a merge pipeline that streams row values before finishing the dictionary pass.

Common situations: Custom ingestion code with out-of-order phase execution (values before dictionaries); partial-failure recovery that resumes at the value phase of a fresh serializer without redoing the dictionary phase.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/acbb124333c440b2. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/nested/ScalarNestedCommonFormatColumnSerializer.java:144

    return hasNulls;
  }

  @Override
  public void open() throws IOException
  {
    if (!dictionarySerialized) {
      throw new IllegalStateException("Dictionary not serialized, cannot open value serializer");
    }
    intermediateValueWriter = new FixedIndexedIntWriter(segmentWriteOutMedium, false);
    intermediateValueWriter.open();
    openValueColumnSerializer();
  }

  @Override
  public void serialize(ColumnValueSelector<? extends StructuredData> selector) throws IOException
  {
    if (!dictionarySerialized) {
      throw new ISE("Must serialize value dictionaries before serializing values for column [%s]", name);
    }

    final Object value = StructuredData.unwrap(selector.getObject());
    final int dictId = processValue(value);
    intermediateValueWriter.write(dictId);
    hasNulls = hasNulls || dictId == 0;
  }

  private void closeForWrite()
  {
    if (!closedForWrite) {
      columnNameBytes = computeFilenameBytes();
      closedForWrite = true;
    }
  }

  @Override
  public long getSerializedSize()

View on GitHub (pinned to 9b90983fd2)