apache/druid · error · java.lang.IllegalStateException

Dictionary not serialized, cannot open value serializer

Error message

Dictionary not serialized, cannot open value serializer

What it means

ScalarNestedCommonFormatColumnSerializer.open() creates the intermediate value writer, but only after the value dictionary has been serialized (dictionarySerialized flag). Opening the value serializer before the dictionary exists would leave the intermediate FixedIndexedIntWriter pointing at a dictionary that never gets written, so it throws IllegalStateException.

Solutions

Call serializeDictionaries() before calling open() so the dictionarySerialized flag is set
Fix the phase ordering in the segment-writing pipeline: write dictionaries, then open, then serialize values
If writing custom merger logic, mirror the ordering used by IndexMergerV9/nested-column support in druid-processing

Example fix

// before
serializer.open();                               // ISE: dictionary not serialized
serializer.serializeDictionaries(s, l, d, a);
// after
serializer.serializeDictionaries(s, l, d, a);
serializer.open();
serializer.serialize(selector);

Defensive patterns

Strategy: validation

Validate before calling

// Enforce phase order in the writing pipeline before open()
Objects.requireNonNull(writer.getDictionaryState(), "dictionary state must exist");
if (!writer.isDictionarySerialized()) {
  writer.serializeDictionaries(strings, longs, doubles, arrays); // run dictionary phase first
}
writer.open();

Try / catch

try {
  serializer.open();
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().contains("Dictionary not serialized")) {
    LOG.error(e, "open() called before serializeDictionaries(); fixing phase order and restarting segment write");
    restartSegmentWriteFromScratch();
  } else {
    throw e;
  }
}

Prevention

Call serializer lifecycle methods strictly in order: serializeDictionaries -> open -> serialize
Centralize serializer lifecycle in one owner class so phases cannot be invoked out of order
Add debug assertions in custom pipelines that dictionary serialization completed before open()
Write a unit test that exercises out-of-order calls to verify failures are caught in CI, not production

When it happens

Trigger: Calling open() on the serializer before calling serializeDictionaries(); an ingestion pipeline that orders phases incorrectly (open -> serialize values -> serialize dictionaries instead of dictionaries first).

Common situations: Custom segment-creation code wiring the serializer lifecycle out of order; generic IndexMerger implementations that forget the nested-column dictionary phase before opening writers.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/76a1d1a52a67d906. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/nested/ScalarNestedCommonFormatColumnSerializer.java:133

  @Override
  public void setDictionaryIdLookup(DictionaryIdLookup dictionaryIdLookup)
  {
    this.dictionaryIdLookup = dictionaryIdLookup;
    this.writeDictionary = false;
    this.dictionarySerialized = true;
  }

  @Override
  public boolean hasNulls()
  {
    return hasNulls;
  }

  @Override
  public void open() throws IOException
  {
    if (!dictionarySerialized) {
      throw new IllegalStateException("Dictionary not serialized, cannot open value serializer");
    }
    intermediateValueWriter = new FixedIndexedIntWriter(segmentWriteOutMedium, false);
    intermediateValueWriter.open();
    openValueColumnSerializer();
  }

  @Override
  public void serialize(ColumnValueSelector<? extends StructuredData> selector) throws IOException
  {
    if (!dictionarySerialized) {
      throw new ISE("Must serialize value dictionaries before serializing values for column [%s]", name);
    }

    final Object value = StructuredData.unwrap(selector.getObject());
    final int dictId = processValue(value);
    intermediateValueWriter.write(dictId);
    hasNulls = hasNulls || dictId == 0;
  }

View on GitHub (pinned to 9b90983fd2)