What causes "failed to decode log entry at index %d: %v" in hashicorp/nomad?

Read the warnings channel and note which indexes failed; the rest of the log can still be inspected.

failed to decode log entry at index %d: %v

Error message

failed to decode log entry at index %d: %v

What it means

After GetLog succeeds, LogEntries decodes each raft.Log into a logMessage via decode(); if decode fails (e.g. a LogCommand entry with empty Data), the error is sent on the warnings channel and the entry is skipped. Note that msgpack body decode failures inside decode are swallowed (body replaced with "FAILED TO DECODE DATA"), so this warning fires only for structural failures like missing command data or an unknown log type with no mapping.

Source

Thrown at helper/raftutil/state.go:129

	entries := make(chan interface{})
	warnings := make(chan error)

	go func() {
		defer store.Close()
		defer close(entries)
		for i := firstIdx; i <= lastIdx; i++ {
			var e raft.Log
			err := store.GetLog(i, &e)
			if err != nil {
				warnings <- fmt.Errorf(
					"failed to read log entry at index %d (firstIdx: %d, lastIdx: %d): %v",
					i, firstIdx, lastIdx, err)
				continue
			}

			entry, err := decode(&e)
			if err != nil {
				warnings <- fmt.Errorf(
					"failed to decode log entry at index %d: %v", i, err)
				continue
			}

			entries <- entry
		}
	}()

	return entries, warnings, nil
}

type logMessage struct {
	LogType string
	Term    uint64
	Index   uint64

	CommandType           string      `json:",omitempty"`
	IgnoreUnknownTypeFlag bool        `json:",omitempty"`

View on GitHub (pinned to 482b49bf1a)

Solutions

Read the warnings channel and note which indexes failed; the rest of the log can still be inspected.
Check the Nomad version that wrote the data and use the matching version of the inspect tooling.
If entries are corrupt (empty command data), restore the raft store from a healthy backup or peer.
Report/inspect the raw raft.Log for those indexes with a custom reader if you need the bytes for forensic analysis.
Treat the server's log as suspect if many entries fail to decode; re-add the server to force log replication.

Example fix

// before
entry, err := decode(&e) // inside LogEntries; empty LogCommand data -> warning
// after
// caller-side: collect decode warnings and fall back to raw inspection
for w := range warningsCh {
	log.Printf("skipping entry: %v", w)
}
// keep raw entries for forensic dump:
raw := &raft.Log{}
if err := store.GetLog(i, raw); err == nil {
	hexdump.Raw(raw.Data)
}

Defensive patterns

Strategy: fallback

Validate before calling

func readEntriesTolerantDecoding(p string) ([]*logMessage, []error, error) {
	entriesCh, warnCh, err := raftutil.LogEntries(p)
	if err != nil {
		return nil, nil, err
	}
	var entries []*logMessage
	var undecodable []error
	for entriesCh != nil || warnCh != nil {
		select {
		case e, ok := <-entriesCh:
			if !ok { entriesCh = nil; continue }
			entries = append(entries, e.(*logMessage))
		case w, ok := <-warnCh:
			if !ok { warnCh = nil; continue }
			undecodable = append(undecodable, w)
		}
	}
	return entries, undecodable, nil
}

Type guard

func isDecodeWarning(w error) bool {
	return w != nil && strings.Contains(w.Error(), "failed to decode log entry at index")
}

Try / catch

for w := range warnings {
	if isDecodeWarning(w) {
		log.Printf("undecodable entry skipped (version mismatch?): %v", w)
		// fall back to raw byte inspection for those indexes
	}
}

Prevention

Use the same Nomad version to inspect logs as the server that wrote them.
Always drain the warnings channel to avoid goroutine deadlock.
Remember msgpack body errors are downgraded to "FAILED TO DECODE DATA" — only structural decode errors warn.
Check for empty/truncated command data as a sign of on-disk corruption.
Keep version-compatible tooling and backups for forensic reads.

When it happens

Trigger: A LogCommand entry whose Data is empty (decode returns 'command did not include data'); or entries written by an incompatible Nomad version whose Data does not conform to the expected layout when decoded.

Common situations: Reading logs produced by a much older/newer Nomad version with different message encoding; entries corrupted on disk so the command payload was truncated to zero length; inspecting logs written by third-party forks using unknown message types.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ffb06b2382afa3a7. Report an issue: GitHub.