hashicorp/consul · critical

failed to decode batch updates: %v

Error message

failed to decode batch updates: %v

What it means

FSM apply handler for network-coordinate batch updates sent by the Serf health syncer. It decodes the committed structs.Coordinates batch with structs.Decode and panics if decoding fails. Coordinate batches are internal agent-to-server traffic, so an undecodable batch nearly always means schema disagreement between agents/servers of different Consul versions, or raft log corruption.

Source

Thrown at agent/consul/fsm/commands_ce.go:311

	defer metrics.MeasureSinceWithLabels([]string{"fsm", "tombstone"}, time.Now(),
		[]metrics.Label{{Name: "op", Value: string(req.Op)}})
	switch req.Op {
	case structs.TombstoneReap:
		return c.state.ReapTombstones(index, req.ReapIndex)
	default:
		c.logger.Warn("Invalid Tombstone operation", "operation", req.Op)
		return fmt.Errorf("Invalid Tombstone operation '%s'", req.Op)
	}
}

// applyCoordinateBatchUpdate processes a batch of coordinate updates and applies
// them in a single underlying transaction. This interface isn't 1:1 with the outer
// update interface that the coordinate endpoint exposes, so we made it single
// purpose and avoided the opcode convention.
func (c *FSM) applyCoordinateBatchUpdate(buf []byte, index uint64) interface{} {
	var updates structs.Coordinates
	if err := structs.Decode(buf, &updates); err != nil {
		panic(fmt.Errorf("failed to decode batch updates: %v", err))
	}
	defer metrics.MeasureSince([]string{"fsm", "coordinate", "batch-update"}, time.Now())
	if err := c.state.CoordinateBatchUpdate(index, updates); err != nil {
		return err
	}
	return nil
}

// applyPreparedQueryOperation applies the given prepared query operation to the
// state store.
func (c *FSM) applyPreparedQueryOperation(buf []byte, index uint64) interface{} {
	var req structs.PreparedQueryRequest
	if err := decodePreparedQueryRequest(buf, &req); err != nil {
		if errors.Is(err, ErrDroppingTenantedReq) {
			c.logger.Warn("dropping tenanted prepared query request")
			return nil
		}
		panic(fmt.Errorf("failed to decode request: %v", err))

View on GitHub (pinned to 2397ff0d76)

Solutions

1. Check that agents and servers run compatible Consul versions (coordinate structs must match)
2. Finish the upgrade so all components share one version
3. Restore from a compatible snapshot or reseed raft data if logs are corrupt
4. Report with raft logs if versions are uniform

# coordinate batches flow agent -> server; verify both run compatible versions
consul version          # on servers
consul info | grep -i version   # cross-check clients where possible

// embedded usage only; recovering a coordinate batch loses Serf health
// position data — acceptable only in a controlled test harness
func (w *WrappedFSM) Apply(log *raft.Log) (resp interface{}) {
    defer func() {
        if r := recover(); r != nil {
            w.log.Error("coordinate batch panic", "index", log.Index, "panic", r)
            resp = fmt.Errorf("fsm apply failed: %v", r)
        }
    }()
    return w.inner.Apply(log)
}

When it happens

Trigger: A committed coordinate-batch raft log that fails msgpack decoding: agents on a different Consul version than servers (coordinate struct fields changed), replaying incompatible snapshots/raft logs, or corrupted raft storage.

Common situations: Rolling upgrades where clients/servers briefly disagree on the coordinate schema; restoring snapshots across versions; failing storage.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to decode request: %v
• ClusterSize not set
• all indexers must have a non-empty name
• no indexer was supplied when creating a new cache Index
• The Indexer must also implement one of the SingleIndexer or

AI-assisted analysis of hashicorp/consul@2397ff0d76 (2026-08-15). Data as JSON: /api/errors/6af180599afea838. Report an issue: GitHub.