risingwavelabs/risingwave · error

Multiple jobs found in no-shuffle ensemble

Error message

Multiple jobs found in no-shuffle ensemble

What it means

Each no-shuffle ensemble must belong to exactly one streaming job. render_actors_with_allocator applies Itertools::exactly_one to the deduplicated (job_id, distribution_type, vnode_count) triples of the ensemble's entry fragments; more than one distinct triple causes this error.

Source

Thrown at src/meta/src/controller/scale.rs:974

            entry_fragments
                .iter()
                .map(|fragment| fragment.parallelism.clone())
                .dedup(),
        )
        .map_err(|_| {
            anyhow!(
                "entry fragments {:?} have inconsistent parallelism settings",
                entries.iter().copied().collect_vec()
            )
        })?;

        let (job_id, distribution_type, vnode_count) = Itertools::exactly_one(
            entry_fragments
                .iter()
                .map(|f| (f.job_id, f.distribution_type, f.vnode_count))
                .dedup(),
        )
        .map_err(|_| anyhow!("Multiple jobs found in no-shuffle ensemble"))?;

        let job = job_map
            .get(&job_id)
            .ok_or_else(|| anyhow!("streaming job {job_id} not found"))?;

        let database_resource_group = streaming_job_databases
            .get(&job_id)
            .and_then(|database_id| database_map.get(database_id))
            .unwrap()
            .resource_group
            .clone();

        let source_entry_fragment = entry_fragments.iter().find(|f| {
            let mask = f.fragment_type_mask;
            if mask.contains(FragmentTypeFlag::Source) {
                assert!(!mask.contains(FragmentTypeFlag::SourceScan))
            }
            mask.contains(FragmentTypeFlag::Source) && !mask.contains(FragmentTypeFlag::Dml)

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the ensemble in the loaded context and ensure its components all belong to a single job/distribution; fix the ensemble-building code if it merged across jobs.
2. Reload the context freshly to rule out stale cached ensembles.
3. Check catalog consistency for fragments listed in the ensemble and their job_id.
4. Report a bug with the ensemble contents if this occurs without migration/upgrade.

let keys: HashSet<_> = entry_fragments.iter()
    .map(|f| (f.job_id, f.distribution_type, f.vnode_count)).collect();
if keys.len() != 1 { bail!("ensemble spans multiple jobs/distributions: {keys:?}"); }

match render_actors_with_allocator(&ctx, ...) {
    Err(e) if e.to_string().contains("no-shuffle ensemble") => {
        // rebuild ensembles from a fresh catalog read
        rebuild_ensembles_then_retry().await
    }
    r => r,
}

When it happens

Trigger: Ensemble construction grouped fragments of different jobs (or different distribution types / vnode counts) into one no-shuffle ensemble — e.g. buggy ensemble merging during load, or catalog corruption mixing fragments from multiple jobs into shared exchange groups.

Common situations: Upgrades or migrations where the no-shuffle grouping logic changed; meta store rows referencing fragments of other jobs; concurrent job replace leaving mixed ensembles.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• job fragments should exist for streaming job
• streaming job should exist for loaded context
• entry fragments {:?} have inconsistent parallelism settings
• streaming job {job_id} not found
• missing source id in source fragment {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/8fd0144802282ea4. Report an issue: GitHub.