hashicorp/nomad · error

raft apply failed: %w

Error message

raft apply failed: %w

What it means

After validation and encryption, Apply commits the variable through Raft (VarApplyStateRequestType). Any error from raftApply — no leader, lost quorum, FSM apply rejection — is wrapped as 'raft apply failed: %w'. The variable write did not commit.

Source

Thrown at nomad/variables_endpoint.go:141

			VariableMetadata: structs.VariableMetadata{
				Namespace:   args.Var.Namespace,
				Path:        args.Var.Path,
				ModifyIndex: args.Var.ModifyIndex,
			},
		}
	}

	// Make a SVEArgs
	sveArgs := structs.VarApplyStateRequest{
		Op:           args.Op,
		Var:          ev,
		WriteRequest: args.WriteRequest,
	}

	// Apply the update.
	o, index, err := sv.srv.raftApply(structs.VarApplyStateRequestType, sveArgs)
	if err != nil {
		return fmt.Errorf("raft apply failed: %w", err)
	}

	out, _ := o.(*structs.VarApplyStateResponse)

	// The return value depends on the operation results and the callers permissions
	r, err := sv.makeVariablesApplyResponse(args, out, aclObj)
	if err != nil {
		return err
	}

	*reply = *r
	reply.Index = index

	if out.IsOk() {
		switch args.Op {
		case structs.VarOpLockAcquire:
			sv.timers.CreateVariableLockTTLTimer(ev.Copy())
		case structs.VarOpLockRelease:

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check cluster leadership (`nomad operator api /v1/status/leader`) and retry after a leader exists
2. Restore quorum or fix failing Raft servers; inspect server logs for the root error
3. Retry the apply with backoff — writes are safe to reissue
4. For CAS ops, re-fetch the current ModifyIndex and retry with the fresh index

Example fix

// before
_, err := vars.Apply(req) // fails during election

// after
err := retry.Do(func() error { _, err := vars.Apply(req); return err }, retry.Delay(time.Second))

leader, _ := client.Status().Leader(ctx)
if leader == "" {
    return errors.New("no leader: defer variable writes until cluster is stable")
}

err := retry.Do(func() error {
    _, err := client.Variables().Apply(req, nil)
    if err != nil && strings.Contains(err.Error(), "raft apply failed") {
        return err // retryable
    }
    return nil
}, retry.Attempts(5), retry.DelayType(retry.BackOffDelay))

When it happens

Trigger: nomad var put / SDK Apply during leader election, quorum loss, Raft store errors, or when the FSM rejects the apply (e.g. CAS mismatch handled separately, but transport/commit failures surface here).

Common situations: Writing variables while servers are down; network partition between client and leader; Raft disk issues on the leader; retry storms during upgrades.

Related errors

• failed to reset heartbeat since server is not leader
• unsupported minimum common raft protocol version
• must provide peer id or address
• No cluster leader
• reconciliation of job summaries failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4eafa99f2b222789. Report an issue: GitHub.