influxdata/influxdb · error · Error

Error creating record batch

Error message

Error creating record batch: {0}

What it means

This is a wrapping error: the table buffer wraps any arrow::error::ArrowError raised while constructing an Arrow RecordBatch from the buffered columns. It means the underlying Arrow builder failed — e.g. column length mismatch, schema/data-type mismatch, or an invalid conversion — not a problem with the buffer lookup itself.

Solutions

1. Read the inner ArrowError message (it is embedded in this error's Display) to see which column/constraint failed.
2. Verify all column arrays have exactly the same length as the row count and match the schema's DataTypes field-for-field.
3. Check for mixed-type writes to the same field (Line Protocol allows re-typing a field) and normalize types before buffering.
4. If it appeared after an arrow-rs version bump, review the changelog for new RecordBatch validation rules and update column construction accordingly.

Example fix

// before
let batch = RecordBatch::try_new(schema, cols).map_err(Error::RecordBatchError)?;

// after
assert_eq!(cols.iter().map(|c| c.len()).collect::<std::collections::HashSet<_>>().len(), 1);
let batch = RecordBatch::try_new(schema, cols).map_err(|e| {
    Error::RecordBatchError(arrow::error::ArrowError::InvalidArgumentError(format!(
        "batch build failed: {e}")))
})?;

// pre-validate columns against the schema before RecordBatch::try_new
for (field, col) in schema.fields().iter().zip(cols.iter()) {
    assert_eq!(field.data_type(), col.data_type(), "column {} type mismatch", field.name());
    assert_eq!(col.len(), expected_rows, "column {} length mismatch", field.name());
}

fn columns_match_schema(schema: &arrow::datatypes::Schema, cols: &[arrow::array::ArrayRef]) -> bool {
    schema.fields().len() == cols.len()
        && schema.fields().iter().zip(cols).all(|(f, c)| f.data_type() == c.data_type() && f.len() == c.len())
}

match buffer.to_record_batch() {
    Ok(batch) => process(batch),
    Err(Error::RecordBatchError(inner)) => {
        log::error("arrow failed building batch: {inner}");
        // inspect inner ArrowError variant to find the offending column
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling any TableBuffer path that calls RecordBatch::try_new (or an equivalent builder) with columns that do not conform to the buffer's schema: mismatched array lengths, wrong Arrow DataType for a field, null-buffer inconsistencies, or an Arrow compute/conversion failure during batch assembly.

Common situations: A field written with an inconsistent type (e.g. mixed integer/float) after a schema change; arrays truncated or grown unevenly; upgrading the arrow-rs crate version introduces stricter validation in RecordBatch::try_new; corrupted buffered data after a failed flush.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• unable to compose record batches from databases
• Field not found in table buffer
• unable to compose record batches from retention policies
• unexpected batch schema mismatch: expected
• column id in series key should be valid

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/75c444229d8bdafc. Report an issue: GitHub.

Appendix: source

Thrown at influxdb3_write/src/write_buffer/table_buffer.rs:32

use influxdb3_catalog::catalog::{TableDefinition, legacy};
use influxdb3_id::ColumnId;
use influxdb3_wal::{FieldData, Row};
use schema::{InfluxColumnType, InfluxFieldType, Schema, SchemaBuilder};
use std::collections::BTreeMap;
use std::collections::btree_map::Entry;
use std::mem::size_of;
use std::ops::Range;
use std::sync::Arc;
use thiserror::Error;

use crate::ChunkFilter;

#[derive(Debug, Error)]
pub enum Error {
    #[error("Field not found in table buffer: {0}")]
    FieldNotFound(String),

    #[error("Error creating record batch: {0}")]
    RecordBatchError(#[from] arrow::error::ArrowError),

    #[error("row range {start}..{end} is out of bounds for a chunk of {row_count} rows")]
    RowRangeOutOfBounds {
        start: usize,
        end: usize,
        row_count: usize,
    },
}

pub(crate) type Result<T, E = Error> = std::result::Result<T, E>;

#[derive(Default)]
pub struct TableBuffer {
    chunk_time_to_chunks: BTreeMap<i64, ChunkTimeBuffer>,
    snapshotting_chunks: Vec<SnapshotChunk>,
}

View on GitHub (pinned to 06200ef96b)