risingwavelabs/risingwave · critical

streaming job should exist for loaded context

Error message

streaming job should exist for loaded context

What it means

In the same per-database split of LoadedFragmentContext, job_map.remove(&job_id) is expected to return the StreamingJob model. This expect() panics when streaming_job_databases maps a job to a database but the corresponding StreamingJob row was not loaded into job_map by the load stage.

Source

Thrown at src/meta/src/controller/scale.rs:387

                .expect("job fragments should exist for streaming job");
            for fragment_id in fragments.keys().copied() {
                fragment_databases.insert(fragment_id, database_id);
                if let Some(source_id) = fragment_source_ids.remove(&fragment_id) {
                    context.fragment_source_ids.insert(fragment_id, source_id);
                }
                if let Some(splits) = fragment_splits.remove(&fragment_id) {
                    context.fragment_splits.insert(fragment_id, splits);
                }
            }

            assert!(
                context
                    .job_map
                    .insert(
                        job_id,
                        job_map
                            .remove(&job_id)
                            .expect("streaming job should exist for loaded context"),
                    )
                    .is_none(),
                "duplicated streaming job"
            );
            assert!(
                context.job_fragments.insert(job_id, fragments).is_none(),
                "duplicated job fragments"
            );
            assert!(
                context
                    .streaming_job_databases
                    .insert(job_id, database_id)
                    .is_none(),
                "duplicated job database mapping"
            );
        }

        for ensemble in ensembles {

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check the meta store for the job_id from the panic: if the streaming job row is gone but fragments remain, clean up orphaned fragments or restore from backup.
2. Re-run the rescale outside of concurrent DDL (cancel/drop window) to rule out a race.
3. Audit build_loaded_context so every job placed in job_fragments/streaming_job_databases is also inserted into job_map.
4. Report as an internal invariant violation with the meta snapshot if it reproduces.

Example fix

// before
job_map.remove(&job_id).expect("streaming job should exist for loaded context")
// after
job_map.remove(&job_id).ok_or_else(|| {
    anyhow!("streaming job {job_id} missing from job_map in loaded context")
})?

for (job_id, _db) in &context.streaming_job_databases {
    assert!(context.job_map.contains_key(job_id), "job {job_id} missing from job_map");
}

fn job_loaded(ctx: &LoadedFragmentContext, job_id: JobId) -> bool { ctx.job_map.contains_key(&job_id) }

let Some(job) = context.job_map.get(&job_id) else {
    return Err(anyhow!("streaming job {job_id} missing from loaded context"));
};

When it happens

Trigger: into_database_contexts receives a context where job_fragments and streaming_job_databases contain a job_id but job_map does not — typically a metadata read that loaded fragments for the job but skipped/lost its streaming-job row (concurrent deletion, or build_loaded_context divergence between the two maps).

Common situations: Rescale or actor re-assignment running while the job is being dropped; corrupted meta store state where fragment rows survive job deletion; a regression in load_fragment_context's job query filters.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• job fragments should exist for streaming job
• expect fn
• logical join should store predicate as Condition
• core predicate must exist
• checked above

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/098573a69eea6b03. Report an issue: GitHub.