risingwavelabs/risingwave · error

downstream relation missing for {} -> {}

Error message

downstream relation missing for {} -> {}

What it means

Thrown by build_reschedule_commands during rescale when a downstream relation entry expected in the `downstream_relations` map is absent for a (fragment_id, downstream_fragment_id) pair. The map is built from the previous fragment graph, so its absence means the plan was not assembled with all upstream->downstream edges intact. It is an internal consistency check in the scale (parallelism/rewrite) path.

Source

Thrown at src/meta/src/stream/scale.rs:930

                            .collect(),
                    };

                let target_fragment_distribution = *distribution_type;

                let fragment_relation::Model {
                    source_fragment_id: _,
                    target_fragment_id: _,
                    dispatcher_type,
                    dist_key_indices,
                    output_indices,
                    output_type_mapping,
                } = downstream_relations
                    .remove(&(
                        *fragment_id as FragmentId,
                        *downstream_fragment_id as FragmentId,
                    ))
                    .ok_or_else(|| {
                        MetaError::from(anyhow!(
                            "downstream relation missing for {} -> {}",
                            fragment_id,
                            downstream_fragment_id
                        ))
                    })?;

                let pb_mapping = PbDispatchOutputMapping {
                    indices: output_indices.into_u32_array(),
                    types: output_type_mapping.unwrap_or_default().to_protobuf(),
                };

                let (dispatchers, _) = compose_dispatchers(
                    *distribution_type,
                    &source_fragment_actors,
                    *downstream_fragment_id,
                    target_fragment_distribution,
                    &target_fragment_actors,
                    dispatcher_type,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Retry the rescale operation once other DDL on the job has settled
2. Check meta logs for concurrent DDL on the same table/mv during rescale
3. Verify the job's fragment graph is intact via RW_FRAGMENT_STATS / meta node metadata
4. Upgrade to a version with the fix if this reproduces deterministically

Example fix

// No caller-side code fix; retry rescale after no other DDL is running
// before: ALTER TABLE t SET PARALLELISM 4;  (while CREATE MV on t is running)
// after:  wait for CREATE MV to finish, then ALTER TABLE t SET PARALLELISM 4;

-- before rescaling, ensure no other DDL is running on the job
SELECT * FROM rw_catalog.rw_ddl_progress WHERE table_name = 'my_mv';

match rescale_result {
    Err(e) if e.to_string().contains("downstream relation missing") => {
        // wait for concurrent DDL, then retry rescale
    }
    r => r?,
}

When it happens

Trigger: Rescale (ALTER ... SET PARALLELISM / pause-resume migration) where a downstream fragment lookup key is missing because the edge was already removed by an earlier iteration, or the reschedule context omitted the edge.

Common situations: Running rescale on a streaming job whose fragment graph changed concurrently (another DDL in flight), or a meta-service bug/state corruption after partial migration.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• should not update vnode bitmap during consuming log store
• next offset {:?} should be later than current offset {:?}
• new item epoch {} does not match current chunk offset epoch
• new item epoch {} does not exceed barrier offset epoch {}
• reschedule failed

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/3046ab48bd5b391a. Report an issue: GitHub.