{"record":{"id":"6af180599afea838","repo":"hashicorp/consul","slug":"failed-to-decode-batch-updates-v","errorCode":null,"errorMessage":"failed to decode batch updates: %v","messagePattern":"failed to decode batch updates: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"agent/consul/fsm/commands_ce.go","lineNumber":311,"sourceCode":"\tdefer metrics.MeasureSinceWithLabels([]string{\"fsm\", \"tombstone\"}, time.Now(),\n\t\t[]metrics.Label{{Name: \"op\", Value: string(req.Op)}})\n\tswitch req.Op {\n\tcase structs.TombstoneReap:\n\t\treturn c.state.ReapTombstones(index, req.ReapIndex)\n\tdefault:\n\t\tc.logger.Warn(\"Invalid Tombstone operation\", \"operation\", req.Op)\n\t\treturn fmt.Errorf(\"Invalid Tombstone operation '%s'\", req.Op)\n\t}\n}\n\n// applyCoordinateBatchUpdate processes a batch of coordinate updates and applies\n// them in a single underlying transaction. This interface isn't 1:1 with the outer\n// update interface that the coordinate endpoint exposes, so we made it single\n// purpose and avoided the opcode convention.\nfunc (c *FSM) applyCoordinateBatchUpdate(buf []byte, index uint64) interface{} {\n\tvar updates structs.Coordinates\n\tif err := structs.Decode(buf, &updates); err != nil {\n\t\tpanic(fmt.Errorf(\"failed to decode batch updates: %v\", err))\n\t}\n\tdefer metrics.MeasureSince([]string{\"fsm\", \"coordinate\", \"batch-update\"}, time.Now())\n\tif err := c.state.CoordinateBatchUpdate(index, updates); err != nil {\n\t\treturn err\n\t}\n\treturn nil\n}\n\n// applyPreparedQueryOperation applies the given prepared query operation to the\n// state store.\nfunc (c *FSM) applyPreparedQueryOperation(buf []byte, index uint64) interface{} {\n\tvar req structs.PreparedQueryRequest\n\tif err := decodePreparedQueryRequest(buf, &req); err != nil {\n\t\tif errors.Is(err, ErrDroppingTenantedReq) {\n\t\t\tc.logger.Warn(\"dropping tenanted prepared query request\")\n\t\t\treturn nil\n\t\t}\n\t\tpanic(fmt.Errorf(\"failed to decode request: %v\", err))","sourceCodeStart":293,"sourceCodeEnd":329,"githubUrl":"https://github.com/hashicorp/consul/blob/2397ff0d763d34f2fe37fe59fde6a7f7fc430a3e/agent/consul/fsm/commands_ce.go#L293-L329","documentation":"FSM apply handler for network-coordinate batch updates sent by the Serf health syncer. It decodes the committed structs.Coordinates batch with structs.Decode and panics if decoding fails. Coordinate batches are internal agent-to-server traffic, so an undecodable batch nearly always means schema disagreement between agents/servers of different Consul versions, or raft log corruption.","triggerScenarios":"A committed coordinate-batch raft log that fails msgpack decoding: agents on a different Consul version than servers (coordinate struct fields changed), replaying incompatible snapshots/raft logs, or corrupted raft storage.","commonSituations":"Rolling upgrades where clients/servers briefly disagree on the coordinate schema; restoring snapshots across versions; failing storage.","solutions":["Check that agents and servers run compatible Consul versions (coordinate structs must match)","Finish the upgrade so all components share one version","Restore from a compatible snapshot or reseed raft data if logs are corrupt","Report with raft logs if versions are uniform"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"# coordinate batches flow agent -> server; verify both run compatible versions\nconsul version          # on servers\nconsul info | grep -i version   # cross-check clients where possible","typeGuard":null,"tryCatchPattern":"// embedded usage only; recovering a coordinate batch loses Serf health\n// position data — acceptable only in a controlled test harness\nfunc (w *WrappedFSM) Apply(log *raft.Log) (resp interface{}) {\n    defer func() {\n        if r := recover(); r != nil {\n            w.log.Error(\"coordinate batch panic\", \"index\", log.Index, \"panic\", r)\n            resp = fmt.Errorf(\"fsm apply failed: %v\", r)\n        }\n    }()\n    return w.inner.Apply(log)\n}","preventionTips":["Upgrade clients and servers together so coordinate structs never disagree","Avoid restoring snapshots written by other major versions","Alert on any FSM panic in servers — it always indicates divergence or corruption"],"tags":["go","consul","raft","fsm","coordinates","msgpack","panic","data-corruption"],"backgroundTag":null,"analyzedSha":"2397ff0d763d34f2fe37fe59fde6a7f7fc430a3e","analyzedAt":"2026-08-15T19:19:47.700Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}