quickwit-oss/quickwit · error

append column: {}

Error message

append column: {}

What it means

append_sorted_series_column computes the new sorted_series column and then rebuilds the RecordBatch with the extended schema via RecordBatch::try_new. If arrow rejects the rebuilt batch (column/field count or type mismatch, array length mismatch, metadata issues), the arrow error is wrapped as "append column: {e}". It indicates the assembled schema and columns are inconsistent — usually a bug rather than bad user input.

Source

Thrown at quickwit/quickwit-parquet-engine/src/sorted_series/mod.rs:135

    let old_schema = batch.schema();
    let mut fields: Vec<Arc<Field>> = old_schema.fields().iter().cloned().collect();
    let mut columns: Vec<Arc<dyn Array>> = (0..batch.num_columns())
        .map(|i| Arc::clone(batch.column(i)))
        .collect();

    fields.push(Arc::new(Field::new(
        SORTED_SERIES_COLUMN,
        DataType::Binary,
        false,
    )));
    columns.push(Arc::new(sorted_series));

    let new_schema = Arc::new(Schema::new_with_metadata(
        fields,
        old_schema.metadata().clone(),
    ));
    RecordBatch::try_new(new_schema, columns).map_err(|e| anyhow!("append column: {}", e))
}

// -----------------------------------------------------------------------
// Internal helpers
// -----------------------------------------------------------------------

/// A resolved key column: its ordinal position in the sort schema,
/// its index in the RecordBatch, and its sort direction.
struct KeyColumn {
    ordinal: u8,
    batch_idx: usize,
    /// Whether this column sorts descending. When true, the storekey
    /// bytes for this column are bitwise-NOTed so that ascending memcmp
    /// on the composite key gives the correct descending order.
    descending: bool,
}

/// The resolved key schema: tag columns plus mandatory timeseries_id.

View on GitHub (pinned to a39730c5cd)

Solutions

1. Read the wrapped arrow error text — it names the exact inconsistency (length, type, or count mismatch).
2. Verify the computed sorted_series column has the same row count as the input batch.
3. Check that fields pushed match columns pushed one-for-one and types match (Binary, non-nullable).

Example fix

// before
RecordBatch::try_new(new_schema, columns).map_err(|e| anyhow!("append column: {}", e))
// after: guard first
assert_eq!(sorted_series.len(), batch.num_rows(), "sorted_series length mismatch");
RecordBatch::try_new(new_schema, columns).map_err(|e| anyhow!("append column: {}", e))

assert_eq!(sorted_series.len(), batch.num_rows(), "row count mismatch");
assert_eq!(fields.len(), columns.len());
assert!(matches!(sorted_series.data_type(), DataType::Binary));

match append_sorted_series_column(sort_fields, &batch) {
    Ok(out) => out,
    Err(e) if e.to_string().starts_with("append column:") => {
        // arrow rejected rebuilt batch: inspect e for length/type mismatch details
        return Err(e.context("schema/column mismatch when appending sorted_series"));
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling append_sorted_series_column where the new Binary column length differs from existing column lengths, or field/column mismatch introduced by concurrent schema handling — surfacing at RecordBatch::try_new.

Common situations: A bug after schema/metadata refactors; filtered or projected batch where column lengths diverged; tests constructing mismatched batches.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• no inputs to align
• type conflict for column '{}': input 0 has {:?}, input {} ha
• sort schema '{}' does not contain a timestamp column
• input {} is missing column '{}'
• `{SORTED_SERIES_COLUMN}` must be Binary-typed

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/468d4afac2fc067e. Report an issue: GitHub.