{"record":{"id":"2c686e35af8c24ee","repo":"tikv/tikv","slug":"store-failed-to-write-to-raft-engine","errorCode":null,"errorMessage":"store {}: {} failed to write to raft engine: {:?}","messagePattern":"store (.+?): (.+?) failed to write to raft engine: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"components/raftstore/src/store/async_io/write.rs","lineNumber":1177,"sourceCode":"        }\n        fail_point!(\"raft_between_save\");\n\n        let mut write_raft_time = 0f64;\n        if !self.batch.raft_wbs[0].is_empty() {\n            fail_point!(\"raft_before_save_on_store_1\", self.store_id == 1, |_| {});\n\n            let now = Instant::now();\n            self.perf_context.start_observe();\n            for i in 0..self.batch.raft_wbs.len() {\n                self.raft_engine\n                    .consume_and_shrink(\n                        &mut self.batch.raft_wbs[i],\n                        true,\n                        RAFT_WB_SHRINK_SIZE,\n                        RAFT_WB_DEFAULT_SIZE,\n                    )\n                    .unwrap_or_else(|e| {\n                        panic!(\n                            \"store {}: {} failed to write to raft engine: {:?}\",\n                            self.store_id, self.tag, e\n                        );\n                    });\n            }\n            self.batch.raft_wbs.truncate(1);\n            let trackers: Vec<_> = self\n                .batch\n                .tasks\n                .iter()\n                .flat_map(|task| task.trackers.iter().flat_map(|t| t.as_tracker_token()))\n                .collect();\n            let _ = self.perf_context.report_metrics(&trackers);\n            write_raft_time = duration_to_sec(now.saturating_elapsed());\n            STORE_WRITE_RAFTDB_DURATION_HISTOGRAM.observe(write_raft_time);\n            // Record the last confirmed raft-log append progress on a monotonic\n            // raw clock. Fail-fast reads this timestamp later to answer a very\n            // specific question: \"even if the dedicated disk probe is slow, are","sourceCodeStart":1159,"sourceCodeEnd":1195,"githubUrl":"https://github.com/tikv/tikv/blob/78aedc1c81ef3f7d8bacc6e9d09f56460f134937/components/raftstore/src/store/async_io/write.rs#L1159-L1195","documentation":"Analogous to the KV-engine failure: the worker writes the raft write batch (raft_wbs) with sync enabled, and on error panics because undurable raft log writes break Raft guarantees. The node aborts rather than continue with a raft engine that lost writes.","triggerScenarios":"write_opt (sync) on the raft engine write batch returns Err during the raft-batch flush — raft disk full, device failure, WAL corruption, or FS errors on the raft column family volume.","commonSituations":"Raft-engine/RocksDB raft CF volume out of space (common when raft and data share a disk), NVMe failure, or corruption after power loss.","solutions":["Check the embedded error plus kernel logs for I/O errors on the raft storage volume; free space or replace the faulty disk.","If using raft-engine, check its logs directory separately; run integrity checks (tikv-ctl raft-engine commands) before restart.","If corruption is found, rebootstrap the store: remove the node from the cluster, wipe its data, and let it re-sync via snapshot.","After the disk issue is fixed, restart; monitor raft log disk usage to prevent recurrence."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"# Ensure the raft storage path has space and is writable before startup\ndf -h /path/to/tikv/raft && test -w /path/to/tikv/raft && echo OK","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Put raft data on a dedicated, monitored volume with headroom","Alert on raft-engine/raft CF size growth and disk usage","Run fs device health monitoring; watch for WAL write latency spikes","After power loss, verify raft engine integrity before rejoining the cluster"],"tags":["raftstore","raft-engine","disk-io","panic","durability"],"backgroundTag":"raft-engine-write-failed","analyzedSha":"78aedc1c81ef3f7d8bacc6e9d09f56460f134937","analyzedAt":"2026-09-03T23:31:32.398Z","contentChangedAt":"2026-09-03T23:31:32.398Z","schemaVersion":2},"datasetVersion":"2026-09-11T07:07:21.782Z"}