{"record":{"id":"098573a69eea6b03","repo":"risingwavelabs/risingwave","slug":"streaming-job-should-exist-for-loaded-context","errorCode":null,"errorMessage":"streaming job should exist for loaded context","messagePattern":"streaming job should exist for loaded context","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"src/meta/src/controller/scale.rs","lineNumber":387,"sourceCode":"                .expect(\"job fragments should exist for streaming job\");\n            for fragment_id in fragments.keys().copied() {\n                fragment_databases.insert(fragment_id, database_id);\n                if let Some(source_id) = fragment_source_ids.remove(&fragment_id) {\n                    context.fragment_source_ids.insert(fragment_id, source_id);\n                }\n                if let Some(splits) = fragment_splits.remove(&fragment_id) {\n                    context.fragment_splits.insert(fragment_id, splits);\n                }\n            }\n\n            assert!(\n                context\n                    .job_map\n                    .insert(\n                        job_id,\n                        job_map\n                            .remove(&job_id)\n                            .expect(\"streaming job should exist for loaded context\"),\n                    )\n                    .is_none(),\n                \"duplicated streaming job\"\n            );\n            assert!(\n                context.job_fragments.insert(job_id, fragments).is_none(),\n                \"duplicated job fragments\"\n            );\n            assert!(\n                context\n                    .streaming_job_databases\n                    .insert(job_id, database_id)\n                    .is_none(),\n                \"duplicated job database mapping\"\n            );\n        }\n\n        for ensemble in ensembles {","sourceCodeStart":369,"sourceCodeEnd":405,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/controller/scale.rs#L369-L405","documentation":"In the same per-database split of LoadedFragmentContext, job_map.remove(&job_id) is expected to return the StreamingJob model. This expect() panics when streaming_job_databases maps a job to a database but the corresponding StreamingJob row was not loaded into job_map by the load stage.","triggerScenarios":"into_database_contexts receives a context where job_fragments and streaming_job_databases contain a job_id but job_map does not — typically a metadata read that loaded fragments for the job but skipped/lost its streaming-job row (concurrent deletion, or build_loaded_context divergence between the two maps).","commonSituations":"Rescale or actor re-assignment running while the job is being dropped; corrupted meta store state where fragment rows survive job deletion; a regression in load_fragment_context's job query filters.","solutions":["Check the meta store for the job_id from the panic: if the streaming job row is gone but fragments remain, clean up orphaned fragments or restore from backup.","Re-run the rescale outside of concurrent DDL (cancel/drop window) to rule out a race.","Audit build_loaded_context so every job placed in job_fragments/streaming_job_databases is also inserted into job_map.","Report as an internal invariant violation with the meta snapshot if it reproduces."],"exampleFix":"// before\njob_map.remove(&job_id).expect(\"streaming job should exist for loaded context\")\n// after\njob_map.remove(&job_id).ok_or_else(|| {\n    anyhow!(\"streaming job {job_id} missing from job_map in loaded context\")\n})?","handlingStrategy":"try-catch","validationCode":"for (job_id, _db) in &context.streaming_job_databases {\n    assert!(context.job_map.contains_key(job_id), \"job {job_id} missing from job_map\");\n}","typeGuard":"fn job_loaded(ctx: &LoadedFragmentContext, job_id: JobId) -> bool { ctx.job_map.contains_key(&job_id) }","tryCatchPattern":"let Some(job) = context.job_map.get(&job_id) else {\n    return Err(anyhow!(\"streaming job {job_id} missing from loaded context\"));\n};","preventionTips":["Avoid rescaling during DROP/CREATE of streaming jobs","Keep load_fragment_context fetching jobs for every mapped job","Detect orphaned rows with a periodic catalog consistency check","Never manually delete streaming_jobs rows while fragments remain"],"tags":["meta","rescale","panic","invariant"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}