risingwavelabs/risingwave · error

source backfill fragment's upstream fragment should have…

Error message

source backfill fragment's upstream fragment should have one-on-one no_shuffle mapping, fragment_id: {fragment_id}, upstream_fragment_id: {upstream_source_fragment_id}, upstream_actor: {upstream_actor}, source_new_no_shuffle: {source_new_no_shuffle:?}

What it means

Companion check to the fragment-level one: for each upstream actor in the no_shuffle mapping there must be a corresponding downstream backfill actor in `source_new_no_shuffle`. This error names the specific upstream_actor whose 1:1 counterpart is missing.

Solutions

1. Check parallelism of upstream vs backfill fragments matches for no_shuffle edges
2. Retry the operation after any in-flight rescale completes
3. Inspect source_new_no_shuffle in logs to find the missing actor pair
4. Recreate the backfill MV to restore a consistent actor mapping

// ensure upstream actor count equals downstream no_shuffle actor count
assert_eq!(upstream_actors.len(), source_new_no_shuffle.len());

if let Err(e) = resolve_backfill_splits(...) {
    if e.to_string().contains("upstream_actor") && e.to_string().contains("no_shuffle") {
        // actor mapping misaligned: recreate backfill MV
    }
}

When it happens

Trigger: resolve_backfill_splits iterates upstream actors and one upstream actor has no entry in the downstream no_shuffle map — actor counts differ between upstream and downstream fragments (not strictly 1:1).

Common situations: Parallelism mismatch between upstream fragment and backfill fragment; a partial actor rewrite during source replacement or rescale leaving upstream/downstream actor sets unaligned.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• no upstream actors found from fragment
• source backfill fragment's upstream fragment should have…
• cannot find StreamActor of actor
• concurrent backup job is not supported: existent job
• `debug_splits` is not allowed in release mode

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/4750b38197544939. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/stream/source_manager/split_assignment.rs:312

                let mut backfill_actors = vec![];
                let Some(source_new_no_shuffle) = actor_no_shuffle
                    .get(&upstream_source_fragment_id)
                    .and_then(|source_upstream_actor_no_shuffle| {
                        source_upstream_actor_no_shuffle.get(&fragment_id)
                    })
                else {
                    return Err(anyhow::anyhow!(
                            "source backfill fragment's upstream fragment should have one-on-one no_shuffle mapping, fragment_id: {fragment_id}, upstream_fragment_id: {upstream_source_fragment_id}, actor_no_shuffle: {actor_no_shuffle:?}",
                            fragment_id = fragment_id,
                            upstream_source_fragment_id = upstream_source_fragment_id,
                            actor_no_shuffle = actor_no_shuffle,
                        ).into());
                };
                for upstream_actor in &upstream_actors {
                    let Some(no_shuffle_backfill_actor) = source_new_no_shuffle.get(upstream_actor)
                    else {
                        return Err(anyhow::anyhow!(
                            "source backfill fragment's upstream fragment should have one-on-one no_shuffle mapping, fragment_id: {fragment_id}, upstream_fragment_id: {upstream_source_fragment_id}, upstream_actor: {upstream_actor}, source_new_no_shuffle: {source_new_no_shuffle:?}",
                            fragment_id = fragment_id,
                            upstream_source_fragment_id = upstream_source_fragment_id,
                            upstream_actor = upstream_actor,
                            source_new_no_shuffle = source_new_no_shuffle
                        ).into());
                    };
                    backfill_actors.push((*no_shuffle_backfill_actor, *upstream_actor));
                }
                assigned.insert(
                    fragment_id,
                    align_splits(
                        backfill_actors,
                        |actor_id| get_upstream_actor_splits(upstream_source_fragment_id, actor_id),
                        fragment_id,
                        upstream_source_fragment_id,
                    )?,
                );

View on GitHub (pinned to 6469eb736d)