{"record":{"id":"3d50311a837b472c","repo":"risingwavelabs/risingwave","slug":"failed-to-rewind-the-log-after-alter-sink-config","errorCode":null,"errorMessage":"failed to rewind the log after ALTER SINK CONFIG","messagePattern":"failed to rewind the log after ALTER SINK CONFIG","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/stream/src/executor/sink.rs","lineNumber":941,"sourceCode":"                RebuildSinkMessage::UpdateConfig(config) => {\n                    if F::ALLOW_REWIND {\n                        match log_reader.rewind().await {\n                            Ok(()) => {\n                                sink_param.properties.extend(config);\n                                sink = TryFrom::try_from(sink_param.clone()).map_err(|e| StreamExecutorError::from((e, sink_param.sink_id)))?;\n                                info!(\n                                    executor_id = %sink_writer_param.executor_id,\n                                    sink_id = %sink_param.sink_id,\n                                    \"alter sink config successfully with rewind\"\n                                );\n                                Ok(())\n                            }\n                            Err(rewind_err) => {\n                                error!(\n                                    error = %rewind_err.as_report(),\n                                    \"failed to rewind the log reader for ALTER SINK CONFIG\"\n                                );\n                                Err(anyhow!(\"failed to rewind the log after ALTER SINK CONFIG\").into())\n                            }\n                        }\n                    } else {\n                        sink_param.properties.extend(config);\n                        sink = TryFrom::try_from(sink_param.clone()).map_err(|e| StreamExecutorError::from((e, sink_param.sink_id)))?;\n                        Err(anyhow!(\"This is not an actual error condition. The system is intentionally triggering recovery procedures to ensure ALTER SINK CONFIG are fully applied.\").into())\n                    }\n                    .map_err(|e| StreamExecutorError::from((e, sink_param.sink_id)))?;\n                }\n            }\n        }\n    }\n}\n\nenum RebuildSinkMessage {\n    RebuildSink(Arc<Bitmap>, oneshot::Sender<()>),\n    UpdateConfig(HashMap<String, String>),\n}","sourceCodeStart":923,"sourceCodeEnd":959,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/stream/src/executor/sink.rs#L923-L959","documentation":"On ALTER SINK CONFIG the reader attempts to rewind the log to a position valid under the new config so consumption can continue. If rewind fails, the only safe path is dropping the reader and forcing recovery; the executor logs the underlying rewind error and returns this wrapped error to trigger that recovery.","triggerScenarios":"The rewind operation on the sink's log reader returns Err (rewind_err) while applying an ALTER SINK CONFIG with vnode bitmap changes, so the code takes the Err(rewind_err) branch and deliberately errors.","commonSituations":"ALTER SINK CONFIG concurrent with heavy log traffic or an incompatible requested position, log reader already advanced past a rewirable point, or an underlying storage failure during rewind.","solutions":["Read the preceding 'failed to rewind the log reader for ALTER SINK CONFIG' log line for the root-cause rewind_err and fix that underlying issue","Retry the recovery — after rebuild the reader starts from a consistent position and the new config applies","Re-issue ALTER SINK CONFIG during low traffic if rewind races with log writes","If rewind deterministically fails, check sink log store backend health (e.g. Kafka/compactor state)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match reader.rewind(pos).await {\n    Ok(()) => {},\n    Err(rewind_err) => {\n        error!(error = %rewind_err.as_report(), \"failed to rewind the log reader for ALTER SINK CONFIG\");\n        return Err(anyhow!(\"failed to rewind the log after ALTER SINK CONFIG\").into());\n    }\n}","preventionTips":["Apply ALTER SINK CONFIG during low write traffic windows","Always read the underlying rewind_err log line before escalating","Verify sink log backend (Kafka etc.) health before altering config"],"tags":["sink","recovery","alter-sink"],"backgroundTag":"invalid-state-transition","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}