{"record":{"id":"ef0483812ace09d8","repo":"influxdata/influxdb","slug":"invalid-statistics","errorCode":null,"errorMessage":"invalid statistics","messagePattern":"invalid statistics","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"core/parquet_file/src/metadata.rs","lineNumber":472,"sourceCode":"            ?partition_id,\n            ?decoded,\n            \"DecodedIoxParquetMetaData decoded from its IoxParquetMetaData\"\n        );\n        let row_count = decoded.row_count();\n        if decoded.md.row_groups().is_empty() {\n            debug!(\n                ?partition_id,\n                \"Decoded IoxParquetMetaData has no row groups to provide useful statistics\"\n            );\n        }\n\n        // Derive the min/max timestamp from the Parquet column statistics.\n        let schema = decoded\n            .read_schema()\n            .expect(\"failed to read encoded schema\");\n        let stats = decoded\n            .read_statistics(&schema)\n            .expect(\"invalid statistics\");\n        let columns: Vec<_> = stats.iter().map(|v| column_id_map(&v.name)).collect();\n\n        // Extract the min/max timestamps.\n        let TimestampRange {\n            min: min_time,\n            max: max_time,\n        } = derive_min_max_time(stats);\n\n        ParquetFileParams {\n            namespace_id: self.namespace_id,\n            table_id: self.table_id,\n            partition_id,\n            partition_hash_id,\n            object_store_id: self.object_store_id,\n            min_time,\n            max_time,\n            file_size_bytes: file_size_bytes as i64,\n            compaction_level: self.compaction_level,","sourceCodeStart":454,"sourceCodeEnd":490,"githubUrl":"https://github.com/influxdata/influxdb/blob/d28e26e048401c53cbb98cf2d6ab0cf1e98048ca/core/parquet_file/src/metadata.rs#L454-L490","documentation":"to_parquet_file then reads per-column parquet statistics with decoded.read_statistics(&schema).expect(\"invalid statistics\"). This parses column-chunk statistics and maps them onto IOx ColumnSummary values, requiring IOx field-type metadata in the schema. It fails when the parquet footer's statistics are malformed, absent in an unexpected way, or the columns lack the IOx type metadata needed for interpretation — the documented failure mode for RecordBatches serialized without IOx metadata.","triggerScenarios":"to_parquet_file over files written by non-IOx writers (statistics shaped differently or missing IOx type annotations), footers with truncated statistics structures, or statistics produced by parquet writer versions this parser rejects.","commonSituations":"Bulk-loading externally produced parquet; fixtures written with plain arrow writers; parquet-crate version upgrades changing statistics representation; catalog compaction touching legacy files.","solutions":["Validate first: decode()?, read_schema()?, then read_statistics(&schema) — handle Err by quarantining or re-ingesting the file instead of panicking.","Re-write foreign parquet through the IOx write path so statistics carry IOx-compatible metadata.","For files written with statistics disabled, re-generate them with statistics enabled (writer option) before cataloging.","File an upstream issue asking to_parquet_file to propagate these as errors; expects on external data are sharp edges."],"exampleFix":"// before\nlet params = table_meta.to_parquet_file(pid, phid, size, &iox_meta, &col_map);\n\n// after\nlet probe = iox_meta\n    .decode()\n    .and_then(|d| d.read_schema().map(|s| (d, s)))\n    .and_then(|(d, s)| d.read_statistics(&s).map(|_| ()));\nif let Err(e) = probe {\n    return Err(format!(\"invalid parquet statistics: {e}\").into());\n}\nlet params = table_meta.to_parquet_file(pid, phid, size, &iox_meta, &col_map);","handlingStrategy":"validation","validationCode":"// full pre-flight: decode, schema, and statistics must all parse\nfn statistics_parse(m: &IoxParquetMetaData) -> bool {\n    m.decode()\n        .and_then(|d| d.read_schema().map(|s| (d, s)))\n        .and_then(|(d, s)| d.read_statistics(&s).map(|_| ()))\n        .is_ok()\n}\n\nif !statistics_parse(&iox_meta) {\n    return quarantine(object_store_id, \"invalid parquet statistics\");\n}","typeGuard":null,"tryCatchPattern":"let ok = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {\n    table_meta.to_parquet_file(pid, phid, size, &iox_meta, &col_map)\n}));\nif let Err(_) = ok {\n    return Err(CatalogError::InvalidParquetStatistics.into());\n}","preventionTips":["Write parquet with column statistics enabled (default in arrow-rs) so read_statistics has data.","Reject or re-ingest files from external writers before they reach catalog conversion.","Prefer pre-flight decode/read_schema/read_statistics over catch_unwind — it is deterministic and cheap relative to IO.","Maintain fixture files with valid IOx metadata for regression tests of the catalog path."],"tags":["rust","influxdb-iox","parquet","statistics","deserialization","panic"],"backgroundTag":"metadata-decode-failed","analyzedSha":"d28e26e048401c53cbb98cf2d6ab0cf1e98048ca","analyzedAt":"2026-08-16T19:53:34.623Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}