risingwavelabs/risingwave · error

failed to rewind the log after ALTER SINK CONFIG

Error message

failed to rewind the log after ALTER SINK CONFIG

What it means

On ALTER SINK CONFIG the reader attempts to rewind the log to a position valid under the new config so consumption can continue. If rewind fails, the only safe path is dropping the reader and forcing recovery; the executor logs the underlying rewind error and returns this wrapped error to trigger that recovery.

Solutions

1. Read the preceding 'failed to rewind the log reader for ALTER SINK CONFIG' log line for the root-cause rewind_err and fix that underlying issue
2. Retry the recovery — after rebuild the reader starts from a consistent position and the new config applies
3. Re-issue ALTER SINK CONFIG during low traffic if rewind races with log writes
4. If rewind deterministically fails, check sink log store backend health (e.g. Kafka/compactor state)

match reader.rewind(pos).await {
    Ok(()) => {},
    Err(rewind_err) => {
        error!(error = %rewind_err.as_report(), "failed to rewind the log reader for ALTER SINK CONFIG");
        return Err(anyhow!("failed to rewind the log after ALTER SINK CONFIG").into());
    }
}

When it happens

Trigger: The rewind operation on the sink's log reader returns Err (rewind_err) while applying an ALTER SINK CONFIG with vnode bitmap changes, so the code takes the Err(rewind_err) branch and deliberately errors.

Common situations: ALTER SINK CONFIG concurrent with heavy log traffic or an incompatible requested position, log reader already advanced past a rewirable point, or an underlying storage failure during rewind.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• This is not an actual error condition. The system is…
• LanceDB pre-commit epoch
• a stream has reached the end but some other stream has not…
• adhoc recovery triggered
• ALTER SINK_RATE_LIMIT is not for sink into table

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/3d50311a837b472c. Report an issue: GitHub.

Appendix: source

Thrown at src/stream/src/executor/sink.rs:941

                RebuildSinkMessage::UpdateConfig(config) => {
                    if F::ALLOW_REWIND {
                        match log_reader.rewind().await {
                            Ok(()) => {
                                sink_param.properties.extend(config);
                                sink = TryFrom::try_from(sink_param.clone()).map_err(|e| StreamExecutorError::from((e, sink_param.sink_id)))?;
                                info!(
                                    executor_id = %sink_writer_param.executor_id,
                                    sink_id = %sink_param.sink_id,
                                    "alter sink config successfully with rewind"
                                );
                                Ok(())
                            }
                            Err(rewind_err) => {
                                error!(
                                    error = %rewind_err.as_report(),
                                    "failed to rewind the log reader for ALTER SINK CONFIG"
                                );
                                Err(anyhow!("failed to rewind the log after ALTER SINK CONFIG").into())
                            }
                        }
                    } else {
                        sink_param.properties.extend(config);
                        sink = TryFrom::try_from(sink_param.clone()).map_err(|e| StreamExecutorError::from((e, sink_param.sink_id)))?;
                        Err(anyhow!("This is not an actual error condition. The system is intentionally triggering recovery procedures to ensure ALTER SINK CONFIG are fully applied.").into())
                    }
                    .map_err(|e| StreamExecutorError::from((e, sink_param.sink_id)))?;
                }
            }
        }
    }
}

enum RebuildSinkMessage {
    RebuildSink(Arc<Bitmap>, oneshot::Sender<()>),
    UpdateConfig(HashMap<String, String>),
}

View on GitHub (pinned to 6469eb736d)