hashicorp/nomad · warning

failed to read log entry at index %d (firstIdx: %d, lastIdx:

Error message

failed to read log entry at index %d (firstIdx: %d, lastIdx: %d): %v

What it means

Inside LogEntries' goroutine, each index from firstIdx to lastIdx is fetched via store.GetLog. If GetLog returns an error for a specific index, it is sent on the warnings channel (not returned as a fatal error) and iteration continues. This means the store advertised an index range but the entry at that index could not be read back.

Source

Thrown at helper/raftutil/state.go:121

// warnings. If opening the raft state returns an error, both channels
// will be nil.
func LogEntries(p string) (<-chan interface{}, <-chan error, error) {
	store, firstIdx, lastIdx, err := RaftStateInfo(p)
	if err != nil {
		return nil, nil, fmt.Errorf("failed to open raft logs: %v", err)
	}

	entries := make(chan interface{})
	warnings := make(chan error)

	go func() {
		defer store.Close()
		defer close(entries)
		for i := firstIdx; i <= lastIdx; i++ {
			var e raft.Log
			err := store.GetLog(i, &e)
			if err != nil {
				warnings <- fmt.Errorf(
					"failed to read log entry at index %d (firstIdx: %d, lastIdx: %d): %v",
					i, firstIdx, lastIdx, err)
				continue
			}

			entry, err := decode(&e)
			if err != nil {
				warnings <- fmt.Errorf(
					"failed to decode log entry at index %d: %v", i, err)
				continue
			}

			entries <- entry
		}
	}()

	return entries, warnings, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the warnings channel alongside entries and log each failed index; a few isolated gaps may be tolerable for inspection.
2. Determine the extent: many consecutive failures means broad corruption — restore raft.db / the wal/ directory from a healthy backup or peer.
3. Run filesystem/disk health checks on the volume; replace failing hardware before rebuilding.
4. If the server's own state is affected, retire the server and re-join a fresh one so raft re-replicates logs from the leader.
5. Keep the store open read-only and stop the agent to avoid read failures racing with compaction/writes.

Example fix

// before
entries, _, err := raftutil.LogEntries(p)
// warnings discarded — unreadable indexes silently dropped
// after
entriesCh, warningsCh, err := raftutil.LogEntries(p)
if err != nil {
	return err
}
for w := range warningsCh {
	log.Printf("raft log gap: %v", w) // surface corrupt/missing indexes
}

func readRangeWithFallback(p string) ([]*logMessage, []error, error) {
	entriesCh, warnCh, err := raftutil.LogEntries(p)
	if err != nil {
		return nil, nil, err
	}
	var entries []*logMessage
	var gaps []error
	for entriesCh != nil || warnCh != nil {
		select {
		case e, ok := <-entriesCh:
			if !ok { entriesCh = nil; continue }
			entries = append(entries, e.(*logMessage))
		case w, ok := <-warnCh:
			if !ok { warnCh = nil; continue }
			gaps = append(gaps, w)
		}
	}
	if len(gaps) > len(entries) {
		return nil, gaps, fmt.Errorf("majority of raft log entries unreadable; restore from backup")
	}
	return entries, gaps, nil
}

func isLogReadWarning(w error) bool {
	return w != nil && strings.Contains(w.Error(), "failed to read log entry at index")
}

for w := range warnings {
	if isLogReadWarning(w) {
		var idx uint64
		fmt.Sscanf(w.Error(), "failed to read log entry at index %d", &idx)
		log.Printf("gap at index %d — continuing", idx)
	}
}

When it happens

Trigger: A gap or corruption in the middle of the log range: an index present in metadata (First/LastIndex) but whose record is missing or unreadable — e.g. corrupt segment in the WAL, corrupted bolt page, or a bolt bucket inconsistency.

Common situations: Disk corruption in the middle of raft.db or a WAL segment; data dir partially restored from backup leaving index gaps; hardware failure (bad blocks); interrupted copy of the store to another machine.

Related errors

• command did not include data
• failed to read log entry at index %d: %v
• failed to find raft store in %v: %v
• failed to open raft store %v: %v
• failed to get destination uint64 key %s: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8327ee1825977014. Report an issue: GitHub.