databendlabs/databend · critical

Error reading dictionary

Error message

Error reading dictionary

What it means

Once the dictionary batch header is obtained, `arrow_ipc::reader::read_dictionary(...)` decodes it into `dictionaries_by_id`; the result is unwrapped with `.expect("Error reading dictionary")`. The Arrow reader rejected the dictionary payload (bad buffer lengths, mismatched field ids/types versus `arrow_schema`, or unsupported IPC version). Since packet framing and header parsing already succeeded, this points at payload-level corruption or a schema mismatch between sender and receiver.

Solutions

1. Confirm sender and receiver derive the same `arrow_schema` (check schema synchronization in fragment metadata) so dictionary field ids match.
2. Validate spill-file integrity if the block came from `read_unmanage_spilled_file`; delete corrupt spill data and re-run.
3. Align Arrow/Databend versions across the cluster to eliminate IPC version incompatibilities.
4. Map the `read_dictionary` error into `ErrorCode::Internal` (or `Err` propagation) rather than `.expect` to fail the query gracefully.

Example fix

// before
.expect("Error reading dictionary");

// after
.map_err(|e| ErrorCode::Internal(format!("Error reading dictionary: {}", e)))?;

// Pre-check that the arrow schema on both sides matches before exchange
if sender_schema != receiver_schema {
    return Err(ErrorCode::Internal("exchange schema mismatch: dictionary ids will not resolve"));
}

fn dictionaries_resolvable(schema: &ArrowSchema, dict_ids: &[i64]) -> bool {
    dict_ids.iter().all(|id| schema.fields_with_dict_id(*id).count() > 0)
}

match deserialize_block(&dict, &fragment_data, schema.clone()) {
    Ok(b) => b,
    Err(e) => return Err(ErrorCode::Internal(format!("dictionary decode failed: {e}"))),
}

When it happens

Trigger: `deserialize_block` receiving a dictionary packet whose body buffers don't match the declared batch (truncated `data_body`), whose dictionary field ids don't exist in `arrow_schema`, or whose IPC metadata version isn't supported by the local Arrow reader — reached via `recv_data`, `read`, or `read_unmanage_spilled_file`.

Common situations: Corrupted or truncated spill files; sender schema evolved (new/removed dictionary columns) while a receiver still uses the old schema; cluster nodes on different Arrow versions during rolling upgrades.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• Error parsing first message
• Error parsing dictionary
• DataType::Map should contain a struct field child
• internal error: entered unreachable code
• QueryInfo is none

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/ea6498695a1e6e4e. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/servers/flight/v1/exchange/serde/exchange_deserializer.rs:118

    schema: &DataSchema,
    arrow_schema: Arc<ArrowSchema>,
) -> Result<DataBlock> {
    let mut dictionaries_by_id = HashMap::new();
    for dict_packet in dict {
        if let DataPacket::Dictionary(data) = dict_packet {
            let message =
                root_as_message(&data.data_header[..]).expect("Error parsing first message");
            let buffer = Buffer::from(data.data_body);
            arrow_ipc::reader::read_dictionary(
                &buffer,
                message
                    .header_as_dictionary_batch()
                    .expect("Error parsing dictionary"),
                &arrow_schema,
                &mut dictionaries_by_id,
                &message.version(),
            )
            .expect("Error reading dictionary");
        }
    }

    let batch = flight_data_to_arrow_batch(&fragment_data.data, arrow_schema, &dictionaries_by_id)?;
    let data_block = DataBlock::from_record_batch(schema, &batch)?;
    Ok(data_block)
}

impl BlockMetaTransform<ExchangeDeserializeMeta> for TransformExchangeDeserializer {
    const UNKNOWN_MODE: UnknownMode = UnknownMode::Pass;
    const NAME: &'static str = "TransformExchangeDeserializer";

    fn transform(&mut self, mut meta: ExchangeDeserializeMeta) -> Result<Vec<DataBlock>> {
        match meta.packet.pop().unwrap() {
            DataPacket::ErrorCode(v) => Err(v),
            DataPacket::Dictionary(_) => unreachable!(),
            DataPacket::SerializeProgress { .. } => unreachable!(),
            DataPacket::CopyStatus { .. } => unreachable!(),

View on GitHub (pinned to 288d84d76e)