hashicorp/nomad · warning

failed to read log entry at index

Error message

failed to read log entry at index %d (firstIdx: %d, lastIdx: %d): %v

What it means

Inside LogEntries' goroutine, each index from firstIdx to lastIdx is fetched via store.GetLog. If GetLog returns an error for a specific index, it is sent on the warnings channel (not returned as a fatal error) and iteration continues. This means the store advertised an index range but the entry at that index could not be read back.

Solutions

1. Read the warnings channel alongside entries and log each failed index; a few isolated gaps may be tolerable for inspection.
2. Determine the extent: many consecutive failures means broad corruption — restore raft.db / the wal/ directory from a healthy backup or peer.
3. Run filesystem/disk health checks on the volume; replace failing hardware before rebuilding.
4. If the server's own state is affected, retire the server and re-join a fresh one so raft re-replicates logs from the leader.
5. Keep the store open read-only and stop the agent to avoid read failures racing with compaction/writes.

Example fix

// before
entries, _, err := raftutil.LogEntries(p)
// warnings discarded — unreadable indexes silently dropped
// after
entriesCh, warningsCh, err := raftutil.LogEntries(p)
if err != nil {
	return err
}
for w := range warningsCh {
	log.Printf("raft log gap: %v", w) // surface corrupt/missing indexes
}

func readRangeWithFallback(p string) ([]*logMessage, []error, error) {
	entriesCh, warnCh, err := raftutil.LogEntries(p)
	if err != nil {
		return nil, nil, err
	}
	var entries []*logMessage
	var gaps []error
	for entriesCh != nil || warnCh != nil {
		select {
		case e, ok := <-entriesCh:
			if !ok { entriesCh = nil; continue }
			entries = append(entries, e.(*logMessage))
		case w, ok := <-warnCh:
			if !ok { warnCh = nil; continue }
			gaps = append(gaps, w)
		}
	}
	if len(gaps) > len(entries) {
		return nil, gaps, fmt.Errorf("majority of raft log entries unreadable; restore from backup")
	}
	return entries, gaps, nil
}

func isLogReadWarning(w error) bool {
	return w != nil && strings.Contains(w.Error(), "failed to read log entry at index")
}

for w := range warnings {
	if isLogReadWarning(w) {
		var idx uint64
		fmt.Sscanf(w.Error(), "failed to read log entry at index %d", &idx)
		log.Printf("gap at index %d — continuing", idx)
	}
}

When it happens

Trigger: A gap or corruption in the middle of the log range: an index present in metadata (First/LastIndex) but whose record is missing or unreadable — e.g. corrupt segment in the WAL, corrupted bolt page, or a bolt bucket inconsistency.

Common situations: Disk corruption in the middle of raft.db or a WAL segment; data dir partially restored from backup leaving index gaps; hardware failure (bad blocks); interrupted copy of the store to another machine.

Related errors

• command did not include data
• failed to read log entry at index
• existing BoltDB raft store found at
• failed to create WAL directory
• failed to fetch first index

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8327ee1825977014. Report an issue: GitHub.

Appendix: source

Thrown at helper/raftutil/state.go:121

// warnings. If opening the raft state returns an error, both channels
// will be nil.
func LogEntries(p string) (<-chan interface{}, <-chan error, error) {
	store, firstIdx, lastIdx, err := RaftStateInfo(p)
	if err != nil {
		return nil, nil, fmt.Errorf("failed to open raft logs: %v", err)
	}

	entries := make(chan interface{})
	warnings := make(chan error)

	go func() {
		defer store.Close()
		defer close(entries)
		for i := firstIdx; i <= lastIdx; i++ {
			var e raft.Log
			err := store.GetLog(i, &e)
			if err != nil {
				warnings <- fmt.Errorf(
					"failed to read log entry at index %d (firstIdx: %d, lastIdx: %d): %v",
					i, firstIdx, lastIdx, err)
				continue
			}

			entry, err := decode(&e)
			if err != nil {
				warnings <- fmt.Errorf(
					"failed to decode log entry at index %d: %v", i, err)
				continue
			}

			entries <- entry
		}
	}()

	return entries, warnings, nil
}

View on GitHub (pinned to 482b49bf1a)