{"record":{"id":"25ed374c7b33c374","repo":"hashicorp/nomad","slug":"index-update-failed-v","errorCode":null,"errorMessage":"index update failed: %v","messagePattern":"index update failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/fsm.go","lineNumber":892,"sourceCode":"\tif err := n.periodicDispatcher.Remove(namespace, jobID); err != nil {\n\t\treturn fmt.Errorf(\"periodicDispatcher.Remove failed: %w\", err)\n\t}\n\n\tif noShutdownDelay {\n\t\tws := memdb.NewWatchSet()\n\t\tallocs, err := n.state.AllocsByJob(ws, namespace, jobID, false)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\ttransition := &structs.DesiredTransition{NoShutdownDelay: new(true)}\n\t\tfor _, alloc := range allocs {\n\t\t\terr := n.state.UpdateAllocDesiredTransitionTxn(tx, index, alloc.ID, transition)\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\terr = tx.Insert(\"index\", &state.IndexEntry{Key: \"allocs\", Value: index})\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t\t\t}\n\t\t}\n\t}\n\n\tif purge {\n\t\tif err := n.state.DeleteJobTxn(index, namespace, jobID, tx); err != nil {\n\t\t\treturn fmt.Errorf(\"DeleteJob failed: %w\", err)\n\t\t}\n\n\t\t// We always delete from the periodic launch table because it is possible that\n\t\t// the job was updated to be non-periodic, thus checking if it is periodic\n\t\t// doesn't ensure we clean it up properly.\n\t\tn.state.DeletePeriodicLaunchTxn(index, namespace, jobID, tx)\n\t\treturn nil\n\t}\n\n\t// Get the current job and mark it as stopped and re-insert it.\n\tws := memdb.NewWatchSet()","sourceCodeStart":874,"sourceCodeEnd":910,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/fsm.go#L874-L910","documentation":"While applying alloc desired-transition updates inside handleJobDeregister (for stop-after-deregistration flows), the FSM inserts an IndexEntry (\"allocs\") into the state store txn so watches fire. If tx.Insert fails — typically a memdb/internal error — it is wrapped as \"index update failed\". This indicates the state store transaction is failing at a low level.","triggerScenarios":"handleJobDeregister iterates allocs with noShutdownDelay, UpdateAllocDesiredTransitionTxn succeeded but tx.Insert(\"index\", ...) into the memdb index table fails.","commonSituations":"Memory pressure / allocation failure on the Nomad server, corrupted state store, or transaction already aborted mid-apply.","solutions":["Inspect the wrapped %v error for the underlying cause (e.g. out-of-memory or memdb panic).","Restart the Nomad server agent to rebuild the in-memory state store; state will be re-hydrated from Raft snapshots.","Check server memory limits and host health if OOM-kill signatures appear in logs.","If persistent, restore the cluster from a known-good Raft snapshot."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := client.Jobs().Deregister(jobID, false, nil)\nif err != nil && strings.Contains(err.Error(), \"index update failed\") {\n    // server-side state issue: alert, check server health, retry after restart\n}","preventionTips":["Provision adequate RAM on Nomad servers","Watch for OOM-kill events in server host logs","Keep all servers on the same Nomad version","Take regular Raft snapshots for recovery"],"tags":["nomad","raft","fsm","state-store"],"backgroundTag":"state-store-index-update-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}