influxdata/influxdb · error

invalid statistics

Error message

invalid statistics

What it means

to_parquet_file then reads per-column parquet statistics with decoded.read_statistics(&schema).expect("invalid statistics"). This parses column-chunk statistics and maps them onto IOx ColumnSummary values, requiring IOx field-type metadata in the schema. It fails when the parquet footer's statistics are malformed, absent in an unexpected way, or the columns lack the IOx type metadata needed for interpretation — the documented failure mode for RecordBatches serialized without IOx metadata.

Source

Thrown at core/parquet_file/src/metadata.rs:472

            ?partition_id,
            ?decoded,
            "DecodedIoxParquetMetaData decoded from its IoxParquetMetaData"
        );
        let row_count = decoded.row_count();
        if decoded.md.row_groups().is_empty() {
            debug!(
                ?partition_id,
                "Decoded IoxParquetMetaData has no row groups to provide useful statistics"
            );
        }

        // Derive the min/max timestamp from the Parquet column statistics.
        let schema = decoded
            .read_schema()
            .expect("failed to read encoded schema");
        let stats = decoded
            .read_statistics(&schema)
            .expect("invalid statistics");
        let columns: Vec<_> = stats.iter().map(|v| column_id_map(&v.name)).collect();

        // Extract the min/max timestamps.
        let TimestampRange {
            min: min_time,
            max: max_time,
        } = derive_min_max_time(stats);

        ParquetFileParams {
            namespace_id: self.namespace_id,
            table_id: self.table_id,
            partition_id,
            partition_hash_id,
            object_store_id: self.object_store_id,
            min_time,
            max_time,
            file_size_bytes: file_size_bytes as i64,
            compaction_level: self.compaction_level,

View on GitHub (pinned to d28e26e048)

Solutions

1. Validate first: decode()?, read_schema()?, then read_statistics(&schema) — handle Err by quarantining or re-ingesting the file instead of panicking.
2. Re-write foreign parquet through the IOx write path so statistics carry IOx-compatible metadata.
3. For files written with statistics disabled, re-generate them with statistics enabled (writer option) before cataloging.
4. File an upstream issue asking to_parquet_file to propagate these as errors; expects on external data are sharp edges.

Example fix

// before
let params = table_meta.to_parquet_file(pid, phid, size, &iox_meta, &col_map);

// after
let probe = iox_meta
    .decode()
    .and_then(|d| d.read_schema().map(|s| (d, s)))
    .and_then(|(d, s)| d.read_statistics(&s).map(|_| ()));
if let Err(e) = probe {
    return Err(format!("invalid parquet statistics: {e}").into());
}
let params = table_meta.to_parquet_file(pid, phid, size, &iox_meta, &col_map);

// full pre-flight: decode, schema, and statistics must all parse
fn statistics_parse(m: &IoxParquetMetaData) -> bool {
    m.decode()
        .and_then(|d| d.read_schema().map(|s| (d, s)))
        .and_then(|(d, s)| d.read_statistics(&s).map(|_| ()))
        .is_ok()
}

if !statistics_parse(&iox_meta) {
    return quarantine(object_store_id, "invalid parquet statistics");
}

let ok = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
    table_meta.to_parquet_file(pid, phid, size, &iox_meta, &col_map)
}));
if let Err(_) = ok {
    return Err(CatalogError::InvalidParquetStatistics.into());
}

When it happens

Trigger: to_parquet_file over files written by non-IOx writers (statistics shaped differently or missing IOx type annotations), footers with truncated statistics structures, or statistics produced by parquet writer versions this parser rejects.

Common situations: Bulk-loading externally produced parquet; fixtures written with plain arrow writers; parquet-crate version upgrades changing statistics representation; catalog compaction touching legacy files.

Related errors

• invalid IOx metadata
• failed to read encoded schema
• no min time statistic
• no time column in metadata statistics
• num_row_groups_in_parallel should be above zero

AI-assisted analysis of influxdata/influxdb@d28e26e048 (2026-08-16). Data as JSON: /api/errors/ef0483812ace09d8. Report an issue: GitHub.