unexpected split count: job_id=

Error message

unexpected split count: job_id={job_id}, split_total_count={}, split_completed_count={split_completed_count}

What it means

compose_job_splits checks that when a CDC table backfill is finished (split_completed_count == 1, the completion marker), the persisted split list contains exactly that one marker row. Multiple or zero rows indicate the meta store served uncommitted or partially rewritten data — CdcTableBackfillTracker::complete_job rewrites splits atomically in a transaction, so this should never happen on a healthy store.

Solutions

Check meta store health and logs for transaction errors around the job; verify you are not reading from a lagging replica.
Retry reload — if the rewrite was in flight, a subsequent read should see committed state.
Reset the job's split state via meta recovery: restart the meta node / run recovery so the backfill tracker rebuilds.
If reproducible, file a bug including job_id; the atomic rewrite contract is violated.

Defensive patterns

Strategy: retry

Try / catch

match reload_cdc_table_snapshot_splits(job_id).await {
    Err(e) if e.to_string().contains("unexpected split count") => {
        // likely read torn state from an in-flight rewrite; wait and retry
        tokio::time::sleep(Duration::from_millis(500)).await;
        reload_cdc_table_snapshot_splits(job_id).await?;
    }
    other => other?,
}

Prevention

Ensure the meta store runs on the primary with read-committed-or-stronger isolation.
Do not read election/split tables from lagging replicas.
Avoid killing the meta node during split rewrites; rely on clean shutdown.

When it happens

Trigger: reload_cdc_table_snapshot_splits -> compose_job_splits reads the splits for a job whose completed count is 1 but whose split list has a count other than 1 — i.e. reading torn/uncommitted transaction state or a corrupted split rewrite.

Common situations: Meta store (sql/pg/mysql/sqlite) isolation anomalies or a crashed complete_job leaving partial state; reading a replica with stale/dirty data; version upgrade migrating split storage mid-flight.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/73cbf738791a1330. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/stream/cdc.rs:292

    job_id: JobId,
    splits: Vec<(i64, Vec<u8>, Vec<u8>, i16)>,
) -> MetaResult<CdcTableSnapshotSplits> {
    let split_completed_count = splits
        .iter()
        .filter(|(_, _, _, is_backfill_finished)| *is_backfill_finished == 1)
        .count();
    assert!(
        split_completed_count <= 1,
        "split_completed_count = {}",
        split_completed_count
    );
    let is_backfill_finished = split_completed_count == 1;
    let splits = if is_backfill_finished {
        if splits.len() != 1 {
            // CdcTableBackfillTracker::complete_job rewrites splits in a transaction.
            // This error should only happen when the meta store reads uncommitted data.
            tracing::error!(%job_id, ?splits, "unexpected split count");
            bail!(
                "unexpected split count: job_id={job_id}, split_total_count={}, split_completed_count={split_completed_count}",
                splits.len()
            );
        }
        CdcTableSnapshotSplits::Finished
    } else {
        let splits: Vec<_> = splits
            .into_iter()
            // The try_init_parallel_cdc_table_snapshot_splits ensures that split with a larger split_id will always have a larger left bound.
            // Assigning consecutive splits to the same actor enables potential optimization in CDC backfill executor.
            .sorted_by_key(|(split_id, _, _, _)| *split_id)
            .map(
                |(split_id, left_bound_inclusive, right_bound_exclusive, _)| {
                    CdcTableSnapshotSplitRaw {
                        split_id,
                        left_bound_inclusive,
                        right_bound_exclusive,
                    }

View on GitHub (pinned to 6469eb736d)