{"record":{"id":"acbb124333c440b2","repo":"apache/druid","slug":"must-serialize-value-dictionaries-before-serializi","errorCode":null,"errorMessage":"Must serialize value dictionaries before serializing values for column [%s]","messagePattern":"Must serialize value dictionaries before serializing values for column \\[(.+?)\\]","errorType":"exception","errorClass":"org.apache.druid.java.util.common.ISE","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/nested/ScalarNestedCommonFormatColumnSerializer.java","lineNumber":144,"sourceCode":"    return hasNulls;\n  }\n\n  @Override\n  public void open() throws IOException\n  {\n    if (!dictionarySerialized) {\n      throw new IllegalStateException(\"Dictionary not serialized, cannot open value serializer\");\n    }\n    intermediateValueWriter = new FixedIndexedIntWriter(segmentWriteOutMedium, false);\n    intermediateValueWriter.open();\n    openValueColumnSerializer();\n  }\n\n  @Override\n  public void serialize(ColumnValueSelector<? extends StructuredData> selector) throws IOException\n  {\n    if (!dictionarySerialized) {\n      throw new ISE(\"Must serialize value dictionaries before serializing values for column [%s]\", name);\n    }\n\n    final Object value = StructuredData.unwrap(selector.getObject());\n    final int dictId = processValue(value);\n    intermediateValueWriter.write(dictId);\n    hasNulls = hasNulls || dictId == 0;\n  }\n\n  private void closeForWrite()\n  {\n    if (!closedForWrite) {\n      columnNameBytes = computeFilenameBytes();\n      closedForWrite = true;\n    }\n  }\n\n  @Override\n  public long getSerializedSize()","sourceCodeStart":126,"sourceCodeEnd":162,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/nested/ScalarNestedCommonFormatColumnSerializer.java#L126-L162","documentation":"ScalarNestedCommonFormatColumnSerializer.serialize() writes dictionary IDs into the intermediate value writer, which only makes sense once the value dictionary is final and serialized. If serialize() is called before serializeDictionaries(), IDs assigned now would not match the dictionary that is written later, so an IllegalStateException is thrown.","triggerScenarios":"Calling serialize(ColumnValueSelector) before serializeDictionaries() has completed for the column; a merge pipeline that streams row values before finishing the dictionary pass.","commonSituations":"Custom ingestion code with out-of-order phase execution (values before dictionaries); partial-failure recovery that resumes at the value phase of a fresh serializer without redoing the dictionary phase.","solutions":["Ensure serializeDictionaries() runs to completion before any serialize() call for the column","Restructure the pipeline into two passes: first build/serialize dictionaries, then stream values","Restart serialization with a fresh serializer instance if the dictionary phase was skipped or failed"],"exampleFix":"// before\nserializer.serialize(rowSelector); // ISE: dictionaries not serialized\n// after\nserializer.serializeDictionaries(strings, longs, doubles, arrays);\nserializer.open();\nserializer.serialize(rowSelector);","handlingStrategy":"validation","validationCode":"// Before streaming values, confirm the dictionary phase has completed\nif (!writer.isDictionarySerialized()) {\n  throw new IllegalStateException(\"Run serializeDictionaries() before serializing values for \"\n      + writer.getColumnName());\n}\nwriter.open();\nwriter.serialize(selector);","typeGuard":null,"tryCatchPattern":"try {\n  serializer.serialize(rowSelector);\n} catch (IllegalStateException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"before serializing values\")) {\n    LOG.error(e, \"Value phase ran before dictionary phase for %s; restarting segment write\", name);\n    restartSegmentWriteFromScratch();\n  } else {\n    throw e;\n  }\n}","preventionTips":["Implement the two-pass structure: pass 1 builds and serializes dictionaries, pass 2 streams values","Never resume value serialization on a freshly created serializer; restart the entire write","Guard custom pipelines with an explicit hasDictionaries flag checked before the value loop","Cover the ordering with unit tests mirroring IndexMergerV9's phase sequence"],"tags":["segment-serialization","illegal-state","lifecycle-order","druid-processing"],"backgroundTag":"invalid-state-transition","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}