hashicorp/nomad · critical

panic(r)

Error message

panic(r)

What it means

FSMHelper.ApplyNext replays a Raft log entry through the embedded FSM, recovering panics from Apply. It swallows panics starting with 'failed to apply request' (enterprise-only log entries that OSS binaries can't apply), but any other panic is deliberately re-panicked because it represents a genuine, unexpected FSM bug.

Source

Thrown at helper/raftutil/fsm.go:150

		return 0, 0, ErrNoMoreLogs
	}

	var e raft.Log
	err = f.store.GetLog(f.nextIdx, &e)
	if err != nil {
		return 0, 0, fmt.Errorf("failed to read log entry at index %d: %v", f.nextIdx, err)
	}

	defer func() {
		r := recover()
		if r != nil && strings.HasPrefix(fmt.Sprint(r), "failed to apply request") {
			// Enterprise specific log entries will fail to load in OSS repository with "failed to apply request."
			// If not relevant to investigation, we can ignore them and simply worn.
			f.logger.Warn("failed to apply log; loading Enterprise data-dir in OSS binary?", "index", e.Index)

			f.nextIdx++
		} else if r != nil {
			panic(r)
		}
	}()

	if e.Type == raft.LogCommand {
		f.fsm.Apply(&e)
	}

	f.nextIdx++
	return e.Index, e.Term, nil
}

// ApplyUntil applies all raft entries until (inclusive) the passed index.
func (f *FSMHelper) ApplyUntil(stopIdx uint64) (idx uint64, term uint64, err error) {
	var lastIdx, lastTerm uint64
	for {
		idx, term, err := f.ApplyNext()
		if err == ErrNoMoreLogs {
			return lastIdx, lastTerm, nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the panic message: if it is 'failed to apply request', it is intentionally ignored as an Enterprise entry in OSS
2. If re-panicked, inspect the Raft log at the reported index and the FSM code path that panicked; fix or remove the offending entry
3. Ensure the binary and data-dir versions match (version skew between log writer and FSM applier is a common cause)
4. Use a Nomad-enterprise binary if the data-dir contains enterprise log entries

defer func() {
    if r := recover(); r != nil {
        logger.Error("FSM replay panicked on log entry", "panic", r, "index", nextIdx)
        // abort replay / quarantine the log entry
    }
}()

When it happens

Trigger: Calling ApplyNext/ApplyUntil/ApplyAll (e.g. during raftutil snapshot/restore tooling or dev-mode FSM replay) when f.fsm.Apply panics with a message other than 'failed to apply request'.

Common situations: Loading an OSS data-dir containing malformed or corrupted Raft log entries whose application trips a nil map/nil pointer inside the FSM; replaying logs with entry types the local FSM code can't handle due to version skew.

Related errors

• Failed to create redacted snapshot: %v
• Raft error when restoring snapshot: %v
• failed adding job to periodic dispatcher: %v
• periodicDispatcher.Remove failed: %w
• index update failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c6934fd3a593fbaa. Report an issue: GitHub.