when any of the stream reaches the end, other stream should

Error message

when any of the stream reaches the end, other stream should also reaches the end, but poll result: {:?}

What it means

In debug builds only (`cfg!(debug_assertions)`), after a stream reaches the end the KV log store reader drains the remaining row streams and expects every one to return `None` (already ended). If any stream still yields a row (`Some(result)`), streams ended at different points — contradicting the invariant that all streams end together right after a shared barrier. This is a debug-time consistency check for multi-stream replay correctness.

Source

Thrown at src/stream/src/common/log_store_impl/kv_log_store/serde.rs:1133

                return Err(anyhow!(
                    "when any of the stream reaches the end, it should be right after emitting an barrier. Current state: {:?}",
                    s
                ));
            }
        }
        assert!(
            self.barrier_streams.is_empty(),
            "should not have any pending barrier received stream after barrier emit"
        );
        if !self.not_started_streams.is_empty() {
            return Err(anyhow!(
                "a stream has reached the end but some other stream has not started yet"
            ));
        }
        if cfg!(debug_assertions) {
            while let Some((opt, _stream)) = self.row_streams.next().await {
                if let Some(result) = opt {
                    return Err(anyhow!(
                        "when any of the stream reaches the end, other stream should also reaches the end, but poll result: {:?}",
                        result
                    ));
                }
            }
        }
        Ok(None)
    }
}

#[cfg(test)]
mod tests {
    use std::future::poll_fn;
    use std::iter::once;
    use std::sync::Arc;
    use std::task::Poll;

    use bytes::Bytes;

View on GitHub (pinned to 6469eb736d)

Solutions

Reproduce in a debug build, then dump the per-stream rows around the final barrier epoch and check barrier alignment of the upstream streams.
Recover all streams from a common snapshot/barrier so they restart from the same epoch and end together.
Align writer versions across nodes (rolling upgrade with mismatched serde/log-store versions can misalign barriers).
If seen only in release builds (check disabled), still treat as a correctness bug and file it with repro data.

Example fix

// before: streams independent, can end at different epochs
StreamA::write(rows).await;
StreamB::write(rows).await; // continued past StreamA's end
// after: route all writes through the shared barrier-aligned writer
log_store.write_all_aligned(vec![rows_a, rows_b]).await; // ends together after barrier

Defensive patterns

Strategy: try-catch

Validate before calling

// After end-of-stream, drain in a controlled way and log any laggard stream instead of failing hard
while let Some((opt, stream)) = row_streams.next().await {
    if opt.is_some() { tracing::warn!(?stream, "stream outlived end-of-stream"); }
}

Type guard

// Rust
fn all_streams_ended(results: &[Option<PollResult>]) -> bool {
    results.iter().all(|r| r.is_none())
}

Try / catch

// Rust
match replay_to_end().await {
    Err(e) if e.to_string().contains("other stream should also reaches the end") => {
        recover_from_common_barrier().await?; // re-align streams, then replay again
    }
    other => other,
}

Prevention

Align barrier emission so all parallel streams write barriers at the same epoch
Avoid rolling upgrades that mix writer versions with different barrier cadences
Never hand-patch Hummock data in dev/test clusters
Run replay consistency tests in debug builds before release

When it happens

Trigger: In debug builds, draining the reader after end-of-stream while at least one row stream still polls to `Some(Ok(row))` — i.e. one stream kept producing rows after another had terminated.

Common situations: Mixed-version clusters where one writer continues past a barrier another stopped at; bugs in barrier alignment across parallel streams; manually rewritten or patched Hummock data in test/dev clusters.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/391df4e270695af7. Report an issue: GitHub.