influxdata/influxdb · critical

schema contains non-existent or column

Error message

schema contains non-existent or column

What it means

MutableBatch::try_into_arrow(projection) converts the batch's columns into an Arrow RecordBatch: it builds a Schema from the projection, then for every schema field looks the column up in the batch's name-to-index map; a missing entry hits this defensive expect (the message is a typo for 'non-existent column'). schema() normally returns a proper error earlier for a projected column that does not exist, so reaching this expect means the projection and batch are inconsistent at a lower level (e.g. a schema evolved and the batch lacks the new column).

Source

Thrown at core/mutable_batch/src/lib.rs:131

    pub fn try_into_arrow(self, projection: Projection<'_>) -> Result<RecordBatch> {
        let schema = self.schema(projection)?;

        let Self {
            column_names,
            columns,
            row_count: _,
        } = self;

        // Convert to Vec<Option<_>> to remove each Column avoid copying the
        // underlying data
        let mut columns = columns.into_iter().map(Some).collect::<Vec<_>>();

        let arrays: Result<Vec<_>, Error> = schema
            .iter()
            .map(|(_, field)| {
                let column_index = column_names
                    .get(field.name())
                    .expect("schema contains non-existent or column");
                std::mem::take(&mut columns[*column_index])
                    .expect("schema contains repeated column name")
                    .try_into_arrow()
                    .context(ColumnSnafu {
                        column: field.name(),
                    })
            })
            .collect();

        RecordBatch::try_new(schema.into(), arrays?).context(ArrowSnafu {})
    }

    /// Returns an iterator over the columns in this batch in no particular order
    pub fn columns(&self) -> impl ExactSizeIterator<Item = (usize, &String, &Column)> + '_ {
        self.column_names
            .iter()
            .map(move |(name, idx)| (*idx, name, &self.columns[*idx]))
    }

View on GitHub (pinned to d28e26e048)

Solutions

1. Build the projection from the batch itself: use batch.columns() names or Projection::All instead of a cached table schema
2. Validate the projection before converting: every projected name must appear in batch.columns()
3. When a table gains a column, materialize it in existing batches (e.g. append a null column) before converting against the wider schema
4. If this panics inside released influxdb3 without custom core code, capture the batch/projection and file an issue

Example fix

// before: projection from a (newer) cached table schema
let rb = batch.try_into_arrow(Projection::Some(&cached_schema_cols))?;  // may name missing columns

// after: project exactly what the batch has
let rb = batch.try_into_arrow(Projection::All)?;
// or filter cached_schema_cols to names present in batch.columns()

// verify the projection matches the batch before converting
let names: HashSet<_> = batch.columns().map(|(_, n, _)| n.clone()).collect();
for col in projection_names {
    assert!(names.contains(col), "projected column {col} missing from batch");
}
let rb = batch.try_into_arrow(Projection::All)?;

When it happens

Trigger: Calling try_into_arrow(Projection::Some(&[..])) with a projection naming columns absent from the batch; converting an old-format batch against a wider table schema after a column was added, without materializing the new column.

Common situations: Schema evolution during writes or compaction; code building projections from a cached table schema instead of the batch itself; merging chunks with different column sets.

Related errors

• schema contains repeated column name
• cannot fit duration into u64
• metric should be in progress
• no metric in progress
• can only accept once

AI-assisted analysis of influxdata/influxdb@d28e26e048 (2026-08-16). Data as JSON: /api/errors/06ee10d1a7630538. Report an issue: GitHub.