risingwavelabs/risingwave · error · anyhow::Error

receive AlignInitialEpoch after initialization

Error message

receive AlignInitialEpoch after initialization

What it means

In the steady-state coordination loop (`run_coordination`), receiving `AlignInitialEpoch` after initialization has completed is invalid — epoch alignment only happens during the init handshake. The coordinator bails because continuing would mean an unaligned writer joins mid-stream, which can break epoch ordering.

Source

Thrown at src/meta/src/manager/sink_coordination/coordinator_worker.rs:731

                    }
                    CoordinationHandleManagerEvent::Stop => {
                        self.handle_manager.stop_handle(handle_id)?;
                        running_handles = self
                            .handle_manager
                            .alter_parallelisms(pending_new_handles.drain(..))
                            .await?;
                        self.try_handle_init_requests(&running_handles, &mut two_phase_handler)
                            .await?;

                        continue;
                    }
                    CoordinationHandleManagerEvent::CommitRequest {
                        epoch,
                        metadata,
                        schema_change,
                    } => (handle_id, epoch, (metadata, schema_change)),
                    CoordinationHandleManagerEvent::AlignInitialEpoch(_) => {
                        bail!("receive AlignInitialEpoch after initialization")
                    }
                },
                CoordinatorWorkerEvent::ReadyToCommit(epoch, metadata, schema_change) => {
                    let start_time = Instant::now();
                    let commit_fut = async {
                        match &mut coordinator {
                            SinkCommitCoordinator::SinglePhase(coordinator) => {
                                assert!(metadata.is_none());
                                if let Some(schema_change) = schema_change {
                                    coordinator
                                        .commit_schema_change(epoch, schema_change)
                                        .instrument_await(Self::commit_span(
                                            "single_phase_schema_change",
                                            sink_id,
                                            epoch,
                                        ))
                                        .await?;
                                }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Fix the writer's reconnect logic: after restart it should wait for the coordinator to run a new init/alignment round rather than sending AlignInitialEpoch spontaneously.
2. Restart the sink so writer and coordinator resynchronize through the normal handshake.
3. Check coordinator failover logic — after meta failover the coordinator should re-run `wait_init_handles` for existing handles.
4. Verify rolling upgrades: an older writer binary may still send AlignInitialEpoch in steady state.

// writer-side precheck before sending alignment
if !self.session.is_init_phase() { return Err(anyhow!("alignment only during init")); }

fn in_steady_state(phase: &Phase) -> bool { matches!(phase, Phase::Running) }

// on this error, restart the sink so writer and coordinator redo the handshake
if err.contains("receive AlignInitialEpoch after initialization") { restart_sink(); }

When it happens

Trigger: A handle that joins after the coordinator finished init sends an AlignInitialEpoch event — e.g. a writer reconnecting and replaying its startup sequence, or a writer that missed the transition from init to running phase.

Common situations: Writer reconnect after network blip that replays old handshake logic; coordinator failover where a writer thinks init is still in progress; new sink workers added without going through a fresh coordinator init round.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• receive AlignInitialEpoch on epoch {} from handle {} during
• should have been handled
• receive commit request on epoch {} from handle {} during alt
• infinite
• failed to find handle {} to start

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/5ed222ecb3014df7. Report an issue: GitHub.