{"record":{"id":"8fd0144802282ea4","repo":"risingwavelabs/risingwave","slug":"multiple-jobs-found-in-no-shuffle-ensemble","errorCode":null,"errorMessage":"Multiple jobs found in no-shuffle ensemble","messagePattern":"Multiple jobs found in no-shuffle ensemble","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/controller/scale.rs","lineNumber":974,"sourceCode":"            entry_fragments\n                .iter()\n                .map(|fragment| fragment.parallelism.clone())\n                .dedup(),\n        )\n        .map_err(|_| {\n            anyhow!(\n                \"entry fragments {:?} have inconsistent parallelism settings\",\n                entries.iter().copied().collect_vec()\n            )\n        })?;\n\n        let (job_id, distribution_type, vnode_count) = Itertools::exactly_one(\n            entry_fragments\n                .iter()\n                .map(|f| (f.job_id, f.distribution_type, f.vnode_count))\n                .dedup(),\n        )\n        .map_err(|_| anyhow!(\"Multiple jobs found in no-shuffle ensemble\"))?;\n\n        let job = job_map\n            .get(&job_id)\n            .ok_or_else(|| anyhow!(\"streaming job {job_id} not found\"))?;\n\n        let database_resource_group = streaming_job_databases\n            .get(&job_id)\n            .and_then(|database_id| database_map.get(database_id))\n            .unwrap()\n            .resource_group\n            .clone();\n\n        let source_entry_fragment = entry_fragments.iter().find(|f| {\n            let mask = f.fragment_type_mask;\n            if mask.contains(FragmentTypeFlag::Source) {\n                assert!(!mask.contains(FragmentTypeFlag::SourceScan))\n            }\n            mask.contains(FragmentTypeFlag::Source) && !mask.contains(FragmentTypeFlag::Dml)","sourceCodeStart":956,"sourceCodeEnd":992,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/controller/scale.rs#L956-L992","documentation":"Each no-shuffle ensemble must belong to exactly one streaming job. render_actors_with_allocator applies Itertools::exactly_one to the deduplicated (job_id, distribution_type, vnode_count) triples of the ensemble's entry fragments; more than one distinct triple causes this error.","triggerScenarios":"Ensemble construction grouped fragments of different jobs (or different distribution types / vnode counts) into one no-shuffle ensemble — e.g. buggy ensemble merging during load, or catalog corruption mixing fragments from multiple jobs into shared exchange groups.","commonSituations":"Upgrades or migrations where the no-shuffle grouping logic changed; meta store rows referencing fragments of other jobs; concurrent job replace leaving mixed ensembles.","solutions":["Inspect the ensemble in the loaded context and ensure its components all belong to a single job/distribution; fix the ensemble-building code if it merged across jobs.","Reload the context freshly to rule out stale cached ensembles.","Check catalog consistency for fragments listed in the ensemble and their job_id.","Report a bug with the ensemble contents if this occurs without migration/upgrade."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"let keys: HashSet<_> = entry_fragments.iter()\n    .map(|f| (f.job_id, f.distribution_type, f.vnode_count)).collect();\nif keys.len() != 1 { bail!(\"ensemble spans multiple jobs/distributions: {keys:?}\"); }","typeGuard":null,"tryCatchPattern":"match render_actors_with_allocator(&ctx, ...) {\n    Err(e) if e.to_string().contains(\"no-shuffle ensemble\") => {\n        // rebuild ensembles from a fresh catalog read\n        rebuild_ensembles_then_retry().await\n    }\n    r => r,\n}","preventionTips":["Build ensembles only from fragments of a single job","Validate ensemble composition right after load","Re-read ensembles after upgrades that change grouping logic","Never hand-merge ensembles across jobs"],"tags":["rescale","ensemble","allocator","invariant"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}