locality provider upstream ended unexpectedly during backfil

Error message

locality provider upstream ended unexpectedly during backfill

What it means

During backfill, the locality provider executor selects between its upstream message stream and the snapshot stream, expecting the upstream to stay alive until a barrier is delivered. If the upstream stream terminates (yields None) before any barrier arrives, the provider cannot make progress — the backfill loop has no barrier to break on — and aborts with this error. A stream ending without a barrier is a protocol violation for a running actor.

Source

Thrown at src/stream/src/executor/locality_provider.rs:600

                    let snapshot_next = snapshot_stream_ref.next();
                    pin_mut!(upstream_next);
                    pin_mut!(snapshot_next);

                    match select(upstream_next, snapshot_next).await {
                        FutureEither::Left((msg, _)) => match msg.transpose()? {
                            Some(Message::Barrier(barrier)) => {
                                // Process the barrier after draining the snapshot builders.
                                break barrier;
                            }
                            Some(Message::Chunk(chunk)) => {
                                // Buffer the upstream chunk.
                                upstream_chunk_buffer.push(chunk.compact_vis());
                            }
                            Some(Message::Watermark(_)) => {
                                // Ignore watermark during backfill.
                            }
                            None => {
                                return Err(anyhow::anyhow!(
                                    "locality provider upstream ended unexpectedly during backfill"
                                )
                                .into());
                            }
                        },
                        FutureEither::Right((msg, _)) => match msg.transpose()? {
                            Some((vnode, row)) => {
                                // Use builder to batch rows efficiently
                                let builder = builders.get_mut(&vnode).unwrap();
                                if let Some(data_chunk) = builder.append_one_row(row) {
                                    // Builder is full, handle the chunk
                                    let chunk = Self::handle_snapshot_chunk(
                                        data_chunk,
                                        vnode,
                                        &pk_indices,
                                        &mut backfill_state,
                                        &mut cur_barrier_snapshot_processed_rows,
                                    )?;

View on GitHub (pinned to 6469eb736d)

Solutions

Check the upstream actor's logs for the original failure/panic that ended its stream and fix that root cause.
Retry recovery — RisingWave's barrier-based recovery restarts the fragment and the backfill resumes from its persisted state.
Verify no concurrent rescale/schema change removed the upstream mid-backfill; re-run the backfill after the topology is stable.
If reproducible with a healthy upstream, file a bug with the fragment graph — it indicates an executor/scheduler protocol violation.

Defensive patterns

Strategy: try-catch

Try / catch

match locality_provider.run().await {
    Err(e) if e.to_string().contains("locality provider upstream ended unexpectedly during backfill") => {
        // upstream died mid-backfill: surface root cause, then restart the fragment
        tracing::error!(%e, "locality provider upstream lost during backfill; recovering");
        recover_fragment().await?;
    }
    other => other?,
}

Prevention

Avoid rescaling or DDL/fragment rewrites while a backfill is in progress.
Monitor upstream actor health and restart policies so stream failures are recovered promptly.
Confirm the upstream always emits barriers (never terminates silently) in custom executors feeding the locality provider.
Enable structured logging on upstream actors to correlate their termination with the backfill failure.

When it happens

Trigger: execute_inner's backfill loop selects on upstream.next(); the upstream yields None (actor finished, failed, or was cancelled) before emitting Message::Barrier. Chunks are buffered and watermarks ignored, but stream end without a barrier is fatal.

Common situations: Upstream actor crash/panic during a backfill; rescale or fragment rewrite removing the upstream mid-backfill; upstream reaching a terminal error state; scheduler bugs that stop an actor without emitting a final barrier.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/94ef741b4e6015d6. Report an issue: GitHub.