hashicorp/consul · critical

failed to decode request: %v

Error message

failed to decode request: %v

What it means

FSM apply handler for catalog registration RPCs on Consul servers. Every committed Raft log entry of type RegisterRequest is msgpack-decoded here; if decoding fails — and the error is not the ErrDroppingTenantedReq sentinel that CE downgrade mode maps to a warning and drop — the FSM panics deliberately. An undecodable committed log means the cluster's state machines disagree, and halting the server is considered safer than skipping the entry.

Source

Thrown at agent/consul/fsm/commands_ce.go:166

	registerCommand(structs.PeeringWriteType, (*FSM).applyPeeringWrite)
	registerCommand(structs.PeeringDeleteType, (*FSM).applyPeeringDelete)
	registerCommand(structs.PeeringTerminateByIDType, (*FSM).applyPeeringTerminate)
	registerCommand(structs.PeeringTrustBundleWriteType, (*FSM).applyPeeringTrustBundleWrite)
	registerCommand(structs.PeeringTrustBundleDeleteType, (*FSM).applyPeeringTrustBundleDelete)
	registerCommand(structs.PeeringSecretsWriteType, (*FSM).applyPeeringSecretsWrite)
	registerCommand(structs.ResourceOperationType, (*FSM).applyResourceOperation)
	registerCommand(structs.UpdateVirtualIPRequestType, (*FSM).applyManualVirtualIPs)
}

func (c *FSM) applyRegister(buf []byte, index uint64) interface{} {
	defer metrics.MeasureSince([]string{"fsm", "register"}, time.Now())
	var req structs.RegisterRequest
	if err := decodeRegistrationReq(buf, &req); err != nil {
		if errors.Is(err, ErrDroppingTenantedReq) {
			c.logger.Warn("dropping tenanted register request")
			return nil
		}
		panic(fmt.Errorf("failed to decode request: %v", err))
	}

	// Apply all updates in a single transaction
	if err := c.state.EnsureRegistration(index, &req); err != nil {
		c.logger.Warn("EnsureRegistration failed", "error", err)
		return err
	}
	return nil
}

func (c *FSM) applyDeregister(buf []byte, index uint64) interface{} {
	defer metrics.MeasureSince([]string{"fsm", "deregister"}, time.Now())
	var req structs.DeregisterRequest
	if err := decodeDeregistrationReq(buf, &req); err != nil {
		if errors.Is(err, ErrDroppingTenantedReq) {
			c.logger.Warn("dropping tenanted deregister request")
			return nil
		}

View on GitHub (pinned to 2397ff0d76)

Solutions

1. Verify all servers run the same Consul version and agree on the raft protocol (consul version, consul members, operator raft list-peers)
2. Check preceding logs for 'dropping tenanted register request' warnings — they indicate enterprise tenanted data hitting a CE build in downgrade mode
3. If versions are mixed, complete or roll back the upgrade so a single version writes and applies raft logs
4. If corruption is suspected, restore the node from a known-good snapshot or reseed its raft directory
5. If versions are consistent and no tenanted data is involved, preserve the raft logs and open an issue with HashiCorp

# before upgrade/restore, verify snapshot integrity and version compatibility
consul snapshot inspect /path/to/snapshot.tgz
consul version   # run on every server; all peers must match

// embedded-Consul usage ONLY: recover in an FSM wrapper to convert the panic
// to a crash report. Never do this in a real server: skipping a committed
// register log forks the state machine.
func (w *WrappedFSM) Apply(log *raft.Log) (resp interface{}) {
    defer func() {
        if r := recover(); r != nil {
            w.log.Error("fsm panic", "index", log.Index, "panic", r)
            resp = fmt.Errorf("fsm apply failed: %v", r)
        }
    }()
    return w.inner.Apply(log)
}

When it happens

Trigger: A committed RegisterRequest log that this binary cannot msgpack-decode: mixed server versions writing/reading different schemas, replaying a snapshot or raft logs from an incompatible Consul version, corrupted raft storage, or tenanted enterprise data replayed on a CE build whose downgrade decode path fails outside the recognized sentinel.

Common situations: Rolling upgrades where servers temporarily run different versions; restoring snapshots across major versions; running enterprise (namespaced/partitioned) data against a CE binary; disk corruption in the raft/data directories.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to decode batch updates: %v
• ClusterSize not set
• all indexers must have a non-empty name
• no indexer was supplied when creating a new cache Index
• The Indexer must also implement one of the SingleIndexer or

AI-assisted analysis of hashicorp/consul@2397ff0d76 (2026-08-15). Data as JSON: /api/errors/503bd57ae9a2c4e5. Report an issue: GitHub.