{"record":{"id":"63979b21a60377f2","repo":"risingwavelabs/risingwave","slug":"too-many-cdc-snapshot-splits","errorCode":null,"errorMessage":"too many CDC snapshot splits","messagePattern":"too many CDC snapshot splits","errorType":"exception","errorClass":"ConnectorError","httpStatus":null,"severity":"error","filePath":"src/connector/src/source/cdc/external/postgres.rs","lineNumber":844,"sourceCode":"\nfn to_int_scalar(i: i64, data_type: &DataType) -> ScalarImpl {\n    match data_type {\n        DataType::Int16 => ScalarImpl::Int16(i.try_into().unwrap()),\n        DataType::Int32 => ScalarImpl::Int32(i.try_into().unwrap()),\n        DataType::Int64 => ScalarImpl::Int64(i),\n        _ => {\n            panic!(\"Can't convert int {} to ScalarImpl::{}\", i, data_type)\n        }\n    }\n}\n\nfn try_increase_split_id(split_id: &mut i64) -> ConnectorResult<()> {\n    match split_id.checked_add(1) {\n        Some(s) => {\n            *split_id = s;\n            Ok(())\n        }\n        None => Err(anyhow::anyhow!(\"too many CDC snapshot splits\").into()),\n    }\n}\n\n/// Use the first column of primary keys to split table.\nfn is_supported_even_split_data_type(data_type: &DataType) -> bool {\n    matches!(\n        data_type,\n        DataType::Int16 | DataType::Int32 | DataType::Int64\n    )\n}\n\npub fn type_name_to_pg_type(ty_name: &str) -> Option<PgType> {\n    let ty_name_lower = ty_name.to_lowercase();\n    // Handle array types (prefixed with _)\n    if let Some(base_type) = ty_name_lower.strip_prefix('_') {\n        match base_type {\n            \"int2\" => Some(PgType::INT2_ARRAY),\n            \"int4\" => Some(PgType::INT4_ARRAY),","sourceCodeStart":826,"sourceCodeEnd":862,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/source/cdc/external/postgres.rs#L826-L862","documentation":"`try_increase_split_id` increments the running snapshot split id (i64) for each generated split and guards against overflow. Incrementing beyond `i64::MAX` is treated as an unrecoverable 'too many CDC snapshot splits' error. In practice this requires an astronomically large table or a looping id assignment bug.","triggerScenarios":"Calling `try_increase_split_id` (from `as_even_splits` or `as_uneven_splits`) when `split_id` is already `i64::MAX` so `checked_add(1)` returns None.","commonSituations":"Extremely large tables combined with a tiny `backfill_num_rows_per_split`; a bug in split-id assignment causing unbounded splitting; artificially crafted tests with a pre-set max split id.","solutions":["Increase `backfill_num_rows_per_split` so fewer splits are generated.","Verify split-id assignment logic is not looping/reusing ids incorrectly.","Backfill the table in phases (partition upstream) instead of one giant snapshot.","If you maintain the code, consider widening the id space or reusing ids after completion."],"exampleFix":"// before: tiny split size on a huge table\nWITH (connector = 'postgres-cdc', backfill_num_rows_per_split = '1');\n// after\nWITH (connector = 'postgres-cdc', backfill_num_rows_per_split = '100000');","handlingStrategy":"validation","validationCode":"// Estimate split count before generating\nlet est_splits = total_rows / backfill_num_rows_per_split;\nif est_splits > i64::MAX as f64 as i64 { /* unreachable, but guard anyway */ }","typeGuard":null,"tryCatchPattern":"match try_increase_split_id(&mut split_id) {\n    Err(e) if e.to_string().contains(\"too many CDC snapshot splits\") => {\n        // stop splitting, use the splits generated so far\n        break;\n    },\n    r => r?,\n}","preventionTips":["Use a reasonable backfill_num_rows_per_split for very large tables.","Backfill in phases rather than one monolithic snapshot.","Monitor split counts during backfill and alert on abnormal growth."],"tags":["cdc","postgres","overflow","splits"],"backgroundTag":"value-out-of-range","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}