databendlabs/databend · critical

Error parsing first message

Error message

Error parsing first message

What it means

During block deserialization on the data-exchange path, dictionary `DataPacket`s are parsed as Arrow IPC messages via `root_as_message(&data.data_header[..]).expect("Error parsing first message")`. FlatBuffers verification failed, meaning the header bytes are not a valid Arrow IPC message. This panic indicates corrupted, truncated, or non-dictionary IPC payload exchanged between query nodes (or a schema/serialization version mismatch), and it crashes the deserialization task instead of returning a `Result`.

Solutions

1. Verify all cluster nodes run the same Databend version; rolling upgrades that cross exchange-format changes corrupt packets.
2. Check the source of the dictionary packet: if from a spill file, validate the file is not truncated/corrupted (checksum/size) and re-run the query.
3. Confirm the packet framing logic in `recv_data`/`read` slices `data_header` correctly (no off-by-one or partial reads).
4. Replace the expects with proper error returns (`flatbuffers::root_as_message` error mapped to `ErrorCode::Internal`) so bad packets fail the query cleanly.

Example fix

// before
let message =
    root_as_message(&data.data_header[..]).expect("Error parsing first message");

// after
let message = root_as_message(&data.data_header[..]).map_err(|e| {
    ErrorCode::Internal(format!(
        "Error parsing dictionary IPC message: {}",
        e
    ))
})?;

// Validate dictionary packet before deserialize
if data.data_header.is_empty() {
    return Err(ErrorCode::Internal("empty dictionary data_header in exchange packet"));
}

fn is_valid_dict_packet(p: &DataPacket) -> bool {
    matches!(p, DataPacket::Dictionary(d) if !d.data_header.is_empty())
}

match std::panic::catch_unwind(|| deserialize_block(&dict, &fragment_data, schema.clone())) {
    Ok(res) => res,
    Err(_) => Err(ErrorCode::Internal("corrupt arrow IPC dictionary packet received")),
}

When it happens

Trigger: `deserialize_block(dict, fragment_data, arrow_schema)` receiving a `DataPacket::Dictionary` whose `data_header` is empty, truncated by network send/recv buffering bugs, or produced by an incompatible Arrow IPC writer version; also caused by byte-offset errors in the packet framing upstream in `recv_data` / `read` / spilled-file readers.

Common situations: Mixed Databend versions in a cluster where the exchange serde format changed; corrupted spill files being read back via `read_unmanage_spilled_file`; network issues or buggy proxies mangling flight data packets.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• Error parsing dictionary
• Error reading dictionary
• DataType::Map should contain a struct field child
• internal error: entered unreachable code
• QueryInfo is none

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/38a4573e5c22b34c. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/servers/flight/v1/exchange/serde/exchange_deserializer.rs:107

        let data_block = deserialize_block(dict, fragment_data, &schema, arrow_schema)?;
        if data_block.num_columns() == 0 {
            return Ok(DataBlock::new_with_meta(vec![], row_count as usize, meta));
        }
        data_block.add_meta(meta)
    }
}

pub fn deserialize_block(
    dict: Vec<DataPacket>,
    fragment_data: FragmentData,
    schema: &DataSchema,
    arrow_schema: Arc<ArrowSchema>,
) -> Result<DataBlock> {
    let mut dictionaries_by_id = HashMap::new();
    for dict_packet in dict {
        if let DataPacket::Dictionary(data) = dict_packet {
            let message =
                root_as_message(&data.data_header[..]).expect("Error parsing first message");
            let buffer = Buffer::from(data.data_body);
            arrow_ipc::reader::read_dictionary(
                &buffer,
                message
                    .header_as_dictionary_batch()
                    .expect("Error parsing dictionary"),
                &arrow_schema,
                &mut dictionaries_by_id,
                &message.version(),
            )
            .expect("Error reading dictionary");
        }
    }

    let batch = flight_data_to_arrow_batch(&fragment_data.data, arrow_schema, &dictionaries_by_id)?;
    let data_block = DataBlock::from_record_batch(schema, &batch)?;
    Ok(data_block)
}

View on GitHub (pinned to 288d84d76e)