hashicorp/nomad · error

Invalid variable operation '%s'

Error message

Invalid variable operation '%s'

What it means

The FSM's variable apply dispatcher switches on the Variables operation type (Set/Delete/CAS/Lock/Release). If req.Op does not match any known VarOp, it returns "Invalid variable operation '<op>'" and logs a warning. Since the FSM applies replicated Raft entries, an unknown op means malformed or version-incompatible data reached the state machine.

Source

Thrown at nomad/fsm.go:2404

		panic(fmt.Errorf("failed to decode request: %v", err))
	}
	defer metrics.MeasureSinceWithLabels([]string{"nomad", "fsm", "apply_sv_operation"}, time.Now(),
		[]metrics.Label{{Name: "op", Value: string(req.Op)}})
	switch req.Op {
	case structs.VarOpSet:
		return n.state.VarSet(msgType, index, &req)
	case structs.VarOpDelete:
		return n.state.VarDelete(msgType, index, &req)
	case structs.VarOpDeleteCAS:
		return n.state.VarDeleteCAS(msgType, index, &req)
	case structs.VarOpCAS:
		return n.state.VarSetCAS(msgType, index, &req)
	case structs.VarOpLockAcquire:
		return n.state.VarLockAcquire(msgType, index, &req)
	case structs.VarOpLockRelease:
		return n.state.VarLockRelease(msgType, index, &req)
	default:
		err := fmt.Errorf("Invalid variable operation '%s'", req.Op)
		n.logger.Warn("Invalid variable operation", "operation", req.Op)
		return err
	}
}

func (n *nomadFSM) applyRootKeyMetaUpsert(msgType structs.MessageType, buf []byte, index uint64) any {
	defer metrics.MeasureSince([]string{"nomad", "fsm", "apply_root_key_meta_upsert"}, time.Now())

	var req structs.KeyringUpdateRootKeyMetaRequest
	if err := structs.Decode(buf, &req); err != nil {
		panic(fmt.Errorf("failed to decode request: %v", err))
	}

	wrappedRootKeys := structs.NewRootKey(req.RootKeyMeta)

	if err := n.state.UpsertRootKey(index, wrappedRootKeys, req.Rekey); err != nil {
		n.logger.Error("UpsertWrappedRootKeys failed", "error", err)
		return err

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check `nomad version` on all servers and upgrade the reporting server to match the cluster's latest version.
2. Identify which Op string appears in the warning log line; confirm it exists in your Nomad version's structs.VarOp constants.
3. Ensure clients/tools submit variables via supported RPCs only, never fabricating Op values.
4. If one server repeatedly logs this while others do not, drain/replace that server as it may be lagging or corrupted.

Example fix

// upgrade the lagging server so it recognizes the variable op
// before: nomad 1.6.x in a 1.7.x cluster
nomad agent -server -consul ...  # v1.6
// after
nomad agent -server -consul ...  # v1.7 (supports the new variable op)

// ensure the whole server fleet supports the variables feature before use
for _, m := range members.Members {
    if version.Compare(version.Must(version.Parse(m.Tags["build"]), minVarSupportVersion) < 0) {
        return fmt.Errorf("server %s too old for variables ops", m.Name)
    }
}

_, _, err := client.Variables().Create(var)
if err != nil && strings.Contains(err.Error(), "Invalid variable operation") {
    // a server cannot parse the op: halt writes, upgrade fleet
}

When it happens

Trigger: applyVariableOperation receives a structs.VarOpSet/VarOpDelete/VarOpCAS/VarOpLockAcquire/VarOpLockRelease-unknown Op value — typically a Raft entry written by a newer Nomad version with a variable op this server does not understand.

Common situations: Mixed-version server fleets where a newer server replicated a new variable op to older servers; hand-crafted or corrupted Raft log entries; custom tooling submitting variables RPCs with invalid Op values.

Related errors

• LockNoPathErr
• variable not found
• root key not found
• root key in use, cannot delete
• variable already holds a lock

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e6b43cb5c5328baf. Report an issue: GitHub.