hashicorp/nomad · critical

failed to read log entry at index %d: %v

Error message

failed to read log entry at index %d: %v

What it means

ApplyNext wraps the error from store.GetLog when the Raft log entry at nextIdx cannot be read from the underlying log store (BoltDB/WAL). This indicates backend read failure at a specific index — distinct from compaction gaps or end-of-log.

Source

Thrown at helper/raftutil/fsm.go:138

		if index != 0 {
			f.nextIdx = index + 1
			return index, term, nil
		}
	}

	if f.nextIdx < f.logFirstIdx {
		return 0, 0, fmt.Errorf("missing logs [%v, %v]", f.nextIdx, f.logFirstIdx-1)
	}

	if f.nextIdx > f.logLastIdx {
		return 0, 0, ErrNoMoreLogs
	}

	var e raft.Log
	err = f.store.GetLog(f.nextIdx, &e)
	if err != nil {
		return 0, 0, fmt.Errorf("failed to read log entry at index %d: %v", f.nextIdx, err)
	}

	defer func() {
		r := recover()
		if r != nil && strings.HasPrefix(fmt.Sprint(r), "failed to apply request") {
			// Enterprise specific log entries will fail to load in OSS repository with "failed to apply request."
			// If not relevant to investigation, we can ignore them and simply worn.
			f.logger.Warn("failed to apply log; loading Enterprise data-dir in OSS binary?", "index", e.Index)

			f.nextIdx++
		} else if r != nil {
			panic(r)
		}
	}()

	if e.Type == raft.LogCommand {
		f.fsm.Apply(&e)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped error and run integrity checks on the storage backend (bolt check for BoltDB)
2. Restore the data dir from the latest Raft snapshot or backup
3. Check disk health (smartctl/dmesg) and filesystem errors if corruption is indicated

Example fix

// before
idx, term, err := f.ApplyNext()
if err != nil { log.Fatal(err) }
// after
idx, term, err := f.ApplyNext()
if err != nil && strings.Contains(err.Error(), "failed to read log entry") {
    log.Printf("log store corrupt at index %d: %v — restoring from snapshot", nextIdx, err)
    return restoreFromSnapshot(dataDir)
}

if err := boltIntegrityCheck(filepath.Join(dataDir, "raft.db")); err != nil {
    return fmt.Errorf("raft.db corrupt — restore from snapshot before replay: %w", err)
}

if _, _, err := f.ApplyNext(); err != nil && strings.Contains(err.Error(), "failed to read log entry") {
    log.Printf("log store read failure: %v — falling back to snapshot restore", err)
    return restoreFromSnapshot(dataDir)
}

When it happens

Trigger: Calling ApplyNext when the log store returns an error for GetLog at the current nextIdx: corrupt BoltDB page, WAL read error, I/O failure, or entry missing despite passing the range checks.

Common situations: Corrupted raft.db after a crash or power loss; failing disk sectors; WAL file truncation; reading a data dir on damaged storage.

Related errors

• failed to find raft store in %v: %v
• failed to open raft store %v: %v
• failed to open raft logs: %v
• failed to fetch first index: %v
• failed to fetch last index: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4cfb2b89e65307d8. Report an issue: GitHub.