risingwavelabs/risingwave · error

receive AlignInitialEpoch on epoch {} from handle {} during

Error message

receive AlignInitialEpoch on epoch {} from handle {} during alter parallelism

What it means

During `alter_parallelisms`, receiving an `AlignInitialEpoch` event is unexpected: epoch alignment is an initialization-time event, but the coordinator is in the parallelism-change phase where existing handles only re-register via NewHandle. The coordinator bails to protect the protocol.

Source

Thrown at src/meta/src/manager/sink_coordination/coordinator_worker.rs:491

                    requests.add_new_request(handle_id, (), self.vnode_bitmap(handle_id))?;
                }
                CoordinationHandleManagerEvent::UpdateVnodeBitmap => {
                    assert!(remaining_handles.remove(&handle_id));
                    requests.add_new_request(handle_id, (), self.vnode_bitmap(handle_id))?;
                }
                CoordinationHandleManagerEvent::Stop => {
                    assert!(remaining_handles.remove(&handle_id));
                    self.stop_handle(handle_id)?;
                }
                CoordinationHandleManagerEvent::CommitRequest { epoch, .. } => {
                    bail!(
                        "receive commit request on epoch {} from handle {} during alter parallelism",
                        epoch,
                        handle_id
                    );
                }
                CoordinationHandleManagerEvent::AlignInitialEpoch(epoch) => {
                    bail!(
                        "receive AlignInitialEpoch on epoch {} from handle {} during alter parallelism",
                        epoch,
                        handle_id
                    );
                }
            }
        }
        Ok(requests.handle_ids)
    }
}

/// Represents the coordinator worker's state machine for handling schema changes.
///
/// - `Running`: Normal operation, handles can be started immediately
/// - `WaitingForFlushed`: Waiting for all pending two-phase commits to complete before starting new handles. This
///   ensures new sink executors load the correct schema.
enum CoordinatorWorkerState {
    Running,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Route new handles spawned during `alter_parallelisms` through the correct NewHandle path so they do not run the initial-alignment flow.
2. Make the writer send AlignInitialEpoch only when it receives an AlignInitialEpochResponse request from the coordinator, never proactively.
3. Ensure writer and coordinator versions match (rolling-upgrade skew) — check for recent protocol changes.
4. Retry the alter operation once no writers are (re)connecting.

// writer-side precheck
if !self.init_phase {
    // never send AlignInitialEpoch outside init
    return Err(anyhow!("alignment only valid during init"));
}

fn expects_alignment(phase: &Phase) -> bool { matches!(phase, Phase::Init | Phase::WaitingInitHandles) }

// retry the alter after all writers have re-registered through the proper path
match alter_result { Err(e) if e.contains("AlignInitialEpoch") => retry_after_restart(), _ => {} }

When it happens

Trigger: A handle sends `AlignInitialEpoch` while parallelism alteration is running — typically a brand-new handle that skipped the init handshake (e.g. registered after init completed) and thinks it must align epochs, or a writer reusing an old session's alignment flow.

Common situations: Sink worker restarts overlapping with a parallelism change; new writers spawned mid-alter that bypass `wait_init_handles`; version-skew between a writer binary that still performs initial alignment and a coordinator that no longer expects it in this phase.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• receive AlignInitialEpoch after initialization
• should have been handled
• receive commit request on epoch {} from handle {} during alt
• infinite
• failed to find handle {} to start

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/1f05f79d9bd42c27. Report an issue: GitHub.