{"record":{"id":"468d4afac2fc067e","repo":"quickwit-oss/quickwit","slug":"append-column","errorCode":null,"errorMessage":"append column: {}","messagePattern":"append column: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/sorted_series/mod.rs","lineNumber":135,"sourceCode":"\n    let old_schema = batch.schema();\n    let mut fields: Vec<Arc<Field>> = old_schema.fields().iter().cloned().collect();\n    let mut columns: Vec<Arc<dyn Array>> = (0..batch.num_columns())\n        .map(|i| Arc::clone(batch.column(i)))\n        .collect();\n\n    fields.push(Arc::new(Field::new(\n        SORTED_SERIES_COLUMN,\n        DataType::Binary,\n        false,\n    )));\n    columns.push(Arc::new(sorted_series));\n\n    let new_schema = Arc::new(Schema::new_with_metadata(\n        fields,\n        old_schema.metadata().clone(),\n    ));\n    RecordBatch::try_new(new_schema, columns).map_err(|e| anyhow!(\"append column: {}\", e))\n}\n\n// -----------------------------------------------------------------------\n// Internal helpers\n// -----------------------------------------------------------------------\n\n/// A resolved key column: its ordinal position in the sort schema,\n/// its index in the RecordBatch, and its sort direction.\nstruct KeyColumn {\n    ordinal: u8,\n    batch_idx: usize,\n    /// Whether this column sorts descending. When true, the storekey\n    /// bytes for this column are bitwise-NOTed so that ascending memcmp\n    /// on the composite key gives the correct descending order.\n    descending: bool,\n}\n\n/// The resolved key schema: tag columns plus mandatory timeseries_id.","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/sorted_series/mod.rs#L117-L153","documentation":"append_sorted_series_column computes the new sorted_series column and then rebuilds the RecordBatch with the extended schema via RecordBatch::try_new. If arrow rejects the rebuilt batch (column/field count or type mismatch, array length mismatch, metadata issues), the arrow error is wrapped as \"append column: {e}\". It indicates the assembled schema and columns are inconsistent — usually a bug rather than bad user input.","triggerScenarios":"Calling append_sorted_series_column where the new Binary column length differs from existing column lengths, or field/column mismatch introduced by concurrent schema handling — surfacing at RecordBatch::try_new.","commonSituations":"A bug after schema/metadata refactors; filtered or projected batch where column lengths diverged; tests constructing mismatched batches.","solutions":["Read the wrapped arrow error text — it names the exact inconsistency (length, type, or count mismatch).","Verify the computed sorted_series column has the same row count as the input batch.","Check that fields pushed match columns pushed one-for-one and types match (Binary, non-nullable)."],"exampleFix":"// before\nRecordBatch::try_new(new_schema, columns).map_err(|e| anyhow!(\"append column: {}\", e))\n// after: guard first\nassert_eq!(sorted_series.len(), batch.num_rows(), \"sorted_series length mismatch\");\nRecordBatch::try_new(new_schema, columns).map_err(|e| anyhow!(\"append column: {}\", e))","handlingStrategy":"validation","validationCode":"assert_eq!(sorted_series.len(), batch.num_rows(), \"row count mismatch\");\nassert_eq!(fields.len(), columns.len());\nassert!(matches!(sorted_series.data_type(), DataType::Binary));\n","typeGuard":null,"tryCatchPattern":"match append_sorted_series_column(sort_fields, &batch) {\n    Ok(out) => out,\n    Err(e) if e.to_string().starts_with(\"append column:\") => {\n        // arrow rejected rebuilt batch: inspect e for length/type mismatch details\n        return Err(e.context(\"schema/column mismatch when appending sorted_series\"));\n    }\n    Err(e) => return Err(e),\n}\n","preventionTips":["Keep sorted_series computation and batch rebuild in one function so lengths cannot diverge.","Assert the computed column length equals batch.num_rows() before try_new.","After schema refactors, run the sorted_series unit tests to catch field/column mismatches."],"tags":["arrow","record-batch","schema","sorted-series"],"backgroundTag":"schema-validation-failed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}