tikv/tikv · critical

store {}: {} failed to write to raft engine: {:?}

Error message

store {}: {} failed to write to raft engine: {:?}

What it means

Analogous to the KV-engine failure: the worker writes the raft write batch (raft_wbs) with sync enabled, and on error panics because undurable raft log writes break Raft guarantees. The node aborts rather than continue with a raft engine that lost writes.

Source

Thrown at components/raftstore/src/store/async_io/write.rs:1177

        }
        fail_point!("raft_between_save");

        let mut write_raft_time = 0f64;
        if !self.batch.raft_wbs[0].is_empty() {
            fail_point!("raft_before_save_on_store_1", self.store_id == 1, |_| {});

            let now = Instant::now();
            self.perf_context.start_observe();
            for i in 0..self.batch.raft_wbs.len() {
                self.raft_engine
                    .consume_and_shrink(
                        &mut self.batch.raft_wbs[i],
                        true,
                        RAFT_WB_SHRINK_SIZE,
                        RAFT_WB_DEFAULT_SIZE,
                    )
                    .unwrap_or_else(|e| {
                        panic!(
                            "store {}: {} failed to write to raft engine: {:?}",
                            self.store_id, self.tag, e
                        );
                    });
            }
            self.batch.raft_wbs.truncate(1);
            let trackers: Vec<_> = self
                .batch
                .tasks
                .iter()
                .flat_map(|task| task.trackers.iter().flat_map(|t| t.as_tracker_token()))
                .collect();
            let _ = self.perf_context.report_metrics(&trackers);
            write_raft_time = duration_to_sec(now.saturating_elapsed());
            STORE_WRITE_RAFTDB_DURATION_HISTOGRAM.observe(write_raft_time);
            // Record the last confirmed raft-log append progress on a monotonic
            // raw clock. Fail-fast reads this timestamp later to answer a very
            // specific question: "even if the dedicated disk probe is slow, are

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Check the embedded error plus kernel logs for I/O errors on the raft storage volume; free space or replace the faulty disk.
2. If using raft-engine, check its logs directory separately; run integrity checks (tikv-ctl raft-engine commands) before restart.
3. If corruption is found, rebootstrap the store: remove the node from the cluster, wipe its data, and let it re-sync via snapshot.
4. After the disk issue is fixed, restart; monitor raft log disk usage to prevent recurrence.

# Ensure the raft storage path has space and is writable before startup
df -h /path/to/tikv/raft && test -w /path/to/tikv/raft && echo OK

When it happens

Trigger: write_opt (sync) on the raft engine write batch returns Err during the raft-batch flush — raft disk full, device failure, WAL corruption, or FS errors on the raft column family volume.

Common situations: Raft-engine/RocksDB raft CF volume out of space (common when raft and data share a disk), NVMe failure, or corruption after power loss.

Related errors

• store {}: {} failed to write to kv engine: {:?}
• invalid pd configuration: {:?}
• Number of threads can't be 0
• read meta file {} failed, error {:?}
• parse from bytes error {:?}

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/2c686e35af8c24ee. Report an issue: GitHub.