{"record":{"id":"ea6498695a1e6e4e","repo":"databendlabs/databend","slug":"error-reading-dictionary","errorCode":null,"errorMessage":"Error reading dictionary","messagePattern":"Error reading dictionary","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"src/query/service/src/servers/flight/v1/exchange/serde/exchange_deserializer.rs","lineNumber":118,"sourceCode":"    schema: &DataSchema,\n    arrow_schema: Arc<ArrowSchema>,\n) -> Result<DataBlock> {\n    let mut dictionaries_by_id = HashMap::new();\n    for dict_packet in dict {\n        if let DataPacket::Dictionary(data) = dict_packet {\n            let message =\n                root_as_message(&data.data_header[..]).expect(\"Error parsing first message\");\n            let buffer = Buffer::from(data.data_body);\n            arrow_ipc::reader::read_dictionary(\n                &buffer,\n                message\n                    .header_as_dictionary_batch()\n                    .expect(\"Error parsing dictionary\"),\n                &arrow_schema,\n                &mut dictionaries_by_id,\n                &message.version(),\n            )\n            .expect(\"Error reading dictionary\");\n        }\n    }\n\n    let batch = flight_data_to_arrow_batch(&fragment_data.data, arrow_schema, &dictionaries_by_id)?;\n    let data_block = DataBlock::from_record_batch(schema, &batch)?;\n    Ok(data_block)\n}\n\nimpl BlockMetaTransform<ExchangeDeserializeMeta> for TransformExchangeDeserializer {\n    const UNKNOWN_MODE: UnknownMode = UnknownMode::Pass;\n    const NAME: &'static str = \"TransformExchangeDeserializer\";\n\n    fn transform(&mut self, mut meta: ExchangeDeserializeMeta) -> Result<Vec<DataBlock>> {\n        match meta.packet.pop().unwrap() {\n            DataPacket::ErrorCode(v) => Err(v),\n            DataPacket::Dictionary(_) => unreachable!(),\n            DataPacket::SerializeProgress { .. } => unreachable!(),\n            DataPacket::CopyStatus { .. } => unreachable!(),","sourceCodeStart":100,"sourceCodeEnd":136,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/query/service/src/servers/flight/v1/exchange/serde/exchange_deserializer.rs#L100-L136","documentation":"Once the dictionary batch header is obtained, `arrow_ipc::reader::read_dictionary(...)` decodes it into `dictionaries_by_id`; the result is unwrapped with `.expect(\"Error reading dictionary\")`. The Arrow reader rejected the dictionary payload (bad buffer lengths, mismatched field ids/types versus `arrow_schema`, or unsupported IPC version). Since packet framing and header parsing already succeeded, this points at payload-level corruption or a schema mismatch between sender and receiver.","triggerScenarios":"`deserialize_block` receiving a dictionary packet whose body buffers don't match the declared batch (truncated `data_body`), whose dictionary field ids don't exist in `arrow_schema`, or whose IPC metadata version isn't supported by the local Arrow reader — reached via `recv_data`, `read`, or `read_unmanage_spilled_file`.","commonSituations":"Corrupted or truncated spill files; sender schema evolved (new/removed dictionary columns) while a receiver still uses the old schema; cluster nodes on different Arrow versions during rolling upgrades.","solutions":["Confirm sender and receiver derive the same `arrow_schema` (check schema synchronization in fragment metadata) so dictionary field ids match.","Validate spill-file integrity if the block came from `read_unmanage_spilled_file`; delete corrupt spill data and re-run.","Align Arrow/Databend versions across the cluster to eliminate IPC version incompatibilities.","Map the `read_dictionary` error into `ErrorCode::Internal` (or `Err` propagation) rather than `.expect` to fail the query gracefully."],"exampleFix":"// before\n.expect(\"Error reading dictionary\");\n\n// after\n.map_err(|e| ErrorCode::Internal(format!(\"Error reading dictionary: {}\", e)))?;","handlingStrategy":"validation","validationCode":"// Pre-check that the arrow schema on both sides matches before exchange\nif sender_schema != receiver_schema {\n    return Err(ErrorCode::Internal(\"exchange schema mismatch: dictionary ids will not resolve\"));\n}","typeGuard":"fn dictionaries_resolvable(schema: &ArrowSchema, dict_ids: &[i64]) -> bool {\n    dict_ids.iter().all(|id| schema.fields_with_dict_id(*id).count() > 0)\n}","tryCatchPattern":"match deserialize_block(&dict, &fragment_data, schema.clone()) {\n    Ok(b) => b,\n    Err(e) => return Err(ErrorCode::Internal(format!(\"dictionary decode failed: {e}\"))),\n}","preventionTips":["Ensure fragment metadata propagates an identical arrow schema to all participating nodes.","Delete and regenerate corrupt spill data; monitor disk health on spill volumes.","Upgrade all nodes in lockstep when Arrow versions change."],"tags":["arrow","ipc","serialization","panic","distributed-query"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}