{"record":{"id":"73cbf738791a1330","repo":"risingwavelabs/risingwave","slug":"unexpected-split-count-job-id-job-id-split-tot","errorCode":null,"errorMessage":"unexpected split count: job_id={job_id}, split_total_count={}, split_completed_count={split_completed_count}","messagePattern":"unexpected split count: job_id=(.+?), split_total_count=(.+?), split_completed_count=(.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/stream/cdc.rs","lineNumber":292,"sourceCode":"    job_id: JobId,\n    splits: Vec<(i64, Vec<u8>, Vec<u8>, i16)>,\n) -> MetaResult<CdcTableSnapshotSplits> {\n    let split_completed_count = splits\n        .iter()\n        .filter(|(_, _, _, is_backfill_finished)| *is_backfill_finished == 1)\n        .count();\n    assert!(\n        split_completed_count <= 1,\n        \"split_completed_count = {}\",\n        split_completed_count\n    );\n    let is_backfill_finished = split_completed_count == 1;\n    let splits = if is_backfill_finished {\n        if splits.len() != 1 {\n            // CdcTableBackfillTracker::complete_job rewrites splits in a transaction.\n            // This error should only happen when the meta store reads uncommitted data.\n            tracing::error!(%job_id, ?splits, \"unexpected split count\");\n            bail!(\n                \"unexpected split count: job_id={job_id}, split_total_count={}, split_completed_count={split_completed_count}\",\n                splits.len()\n            );\n        }\n        CdcTableSnapshotSplits::Finished\n    } else {\n        let splits: Vec<_> = splits\n            .into_iter()\n            // The try_init_parallel_cdc_table_snapshot_splits ensures that split with a larger split_id will always have a larger left bound.\n            // Assigning consecutive splits to the same actor enables potential optimization in CDC backfill executor.\n            .sorted_by_key(|(split_id, _, _, _)| *split_id)\n            .map(\n                |(split_id, left_bound_inclusive, right_bound_exclusive, _)| {\n                    CdcTableSnapshotSplitRaw {\n                        split_id,\n                        left_bound_inclusive,\n                        right_bound_exclusive,\n                    }","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/stream/cdc.rs#L274-L310","documentation":"compose_job_splits checks that when a CDC table backfill is finished (split_completed_count == 1, the completion marker), the persisted split list contains exactly that one marker row. Multiple or zero rows indicate the meta store served uncommitted or partially rewritten data — CdcTableBackfillTracker::complete_job rewrites splits atomically in a transaction, so this should never happen on a healthy store.","triggerScenarios":"reload_cdc_table_snapshot_splits -> compose_job_splits reads the splits for a job whose completed count is 1 but whose split list has a count other than 1 — i.e. reading torn/uncommitted transaction state or a corrupted split rewrite.","commonSituations":"Meta store (sql/pg/mysql/sqlite) isolation anomalies or a crashed complete_job leaving partial state; reading a replica with stale/dirty data; version upgrade migrating split storage mid-flight.","solutions":["Check meta store health and logs for transaction errors around the job; verify you are not reading from a lagging replica.","Retry reload — if the rewrite was in flight, a subsequent read should see committed state.","Reset the job's split state via meta recovery: restart the meta node / run recovery so the backfill tracker rebuilds.","If reproducible, file a bug including job_id; the atomic rewrite contract is violated."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match reload_cdc_table_snapshot_splits(job_id).await {\n    Err(e) if e.to_string().contains(\"unexpected split count\") => {\n        // likely read torn state from an in-flight rewrite; wait and retry\n        tokio::time::sleep(Duration::from_millis(500)).await;\n        reload_cdc_table_snapshot_splits(job_id).await?;\n    }\n    other => other?,\n}","preventionTips":["Ensure the meta store runs on the primary with read-committed-or-stronger isolation.","Do not read election/split tables from lagging replicas.","Avoid killing the meta node during split rewrites; rely on clean shutdown."],"tags":["cdc","backfill","meta-store","transaction"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}