hashicorp/nomad · error

missing logs [ , ]

Error message

missing logs [%v, %v]

What it means

ApplyNext returns this error when the next index to apply is below the store's first available log index, meaning the tail of the log has been compacted/truncated and the requested entries are gone. The missing index range [nextIdx, logFirstIdx-1] is reported.

Solutions

  1. Start replay from the store's first index (RaftStateInfo's firstIdx) instead of an old cursor
  2. Restore state from a snapshot at or before the desired index, then replay only subsequent logs
  3. If the gap is unexpected, investigate compaction settings or store truncation

Example fix

// before
f, _ := raftutil.NewFSM(dir)
f.Seek(100) // may be compacted
// after
firstIdx, lastIdx, _ := raftutil.RaftStateInfo(storePath) // via helper
start := max(desiredIdx, firstIdx)
f.Seek(start)
Defensive patterns

Strategy: validation

Validate before calling

firstIdx, _, err := storeFirstIndex(storePath) // via RaftStateInfo
if err != nil {
    return err
}
if replayStartIdx < firstIdx {
    replayStartIdx = firstIdx // skip compacted prefix
}

Try / catch

idx, term, err := f.ApplyNext()
if err != nil && strings.HasPrefix(err.Error(), "missing logs") {
    return fmt.Errorf("log %d is compacted; restore from snapshot then replay from first available index: %w", f.NextIndex(), err)
}

Prevention

When it happens

Trigger: Calling ApplyNext (or ApplyUntil/ApplyAll which loop through it) with a starting index earlier than the store's first retained log, e.g. after snapshot compaction removed old entries.

Common situations: Trying to replay from an old index after Raft compaction; resuming an interrupted replay whose cursor predates the compacted prefix; pointing tooling at a store whose early logs were truncated.

Related errors


AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b9408b45a3bdd7c0. Report an issue: GitHub.

Appendix: source

Thrown at helper/raftutil/fsm.go:128

}

func (f *FSMHelper) ApplyNext() (index uint64, term uint64, err error) {
	if f.nextIdx == 1 {
		// check snapshots first
		index, term, err := f.restoreFromSnapshot()
		if err != nil {
			return 0, 0, err
		}

		if index != 0 {
			f.nextIdx = index + 1
			return index, term, nil
		}
	}

	if f.nextIdx < f.logFirstIdx {
		return 0, 0, fmt.Errorf("missing logs [%v, %v]", f.nextIdx, f.logFirstIdx-1)
	}

	if f.nextIdx > f.logLastIdx {
		return 0, 0, ErrNoMoreLogs
	}

	var e raft.Log
	err = f.store.GetLog(f.nextIdx, &e)
	if err != nil {
		return 0, 0, fmt.Errorf("failed to read log entry at index %d: %v", f.nextIdx, err)
	}

	defer func() {
		r := recover()
		if r != nil && strings.HasPrefix(fmt.Sprint(r), "failed to apply request") {
			// Enterprise specific log entries will fail to load in OSS repository with "failed to apply request."
			// If not relevant to investigation, we can ignore them and simply worn.
			f.logger.Warn("failed to apply log; loading Enterprise data-dir in OSS binary?", "index", e.Index)

View on GitHub (pinned to 482b49bf1a)