{"record":{"id":"3229876af2bec9be","repo":"hashicorp/nomad","slug":"incompatible-state-version-expected-q-but-found","errorCode":null,"errorMessage":"incompatible state version. expected %q but found %q","messagePattern":"incompatible state version\\. expected %q but found %q","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/state/upgrade.go","lineNumber":49,"sourceCode":"\n\t\tv := b.Get(metaVersionKey)\n\t\tif len(v) == 0 {\n\t\t\t// No version; upgrade\n\t\t\treturn nil\n\t\t}\n\n\t\tif bytes.Equal(v, []byte{'2'}) {\n\t\t\tupgradeTo09 = false\n\t\t\treturn nil\n\t\t}\n\t\tif bytes.Equal(v, metaVersion) {\n\t\t\tupgradeTo09 = false\n\t\t\tupgradeTo13 = false\n\t\t\treturn nil\n\t\t}\n\n\t\t// Version exists but does not match. Abort.\n\t\treturn fmt.Errorf(\"incompatible state version. expected %q but found %q\",\n\t\t\tmetaVersion, v)\n\n\t})\n\n\treturn\n}\n\n// addMeta adds version metadata to BoltDB to mark it as upgraded and\n// should be run at the end of the upgrade transaction.\nfunc addMeta(tx *bbolt.Tx) error {\n\t// Create the meta bucket if it doesn't exist\n\tbkt, err := tx.CreateBucketIfNotExists(metaBucketName)\n\tif err != nil {\n\t\treturn err\n\t}\n\treturn bkt.Put(metaVersionKey, metaVersion)\n}\n","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/state/upgrade.go#L31-L67","documentation":"Aborts state DB upgrade when the database's meta version does not match the version the current upgrade path expects. The upgrade code opens a transaction, reads the version bucket, and refuses to proceed on any mismatch — protecting against running an upgrade against a DB from the wrong Nomad version.","triggerScenarios":"During upgrade (e.g. to 0.9/1.3 schema) the version read from the meta bucket is not the expected metaVersion — e.g. DB was already upgraded, was written by a newer Nomad, or the version key is corrupted.","commonSituations":"Downgrading Nomad on a client whose state.db was upgraded by a newer version; a previously half-completed upgrade; manually edited or restored state.db with a stale version record.","solutions":["Check the wrapped/found version value: if it is newer than the agent's, you are downgrading — restore the newer Nomad version or restore a pre-upgrade state.db.backup.","If the DB was already upgraded, remove/rotate state.db and let the client rebuild state.","Never hand-edit the version bucket; restore from backup instead.","Pin Nomad versions across the cluster to avoid mixed-version state writes."],"exampleFix":"// before\nnomad 0.8 agent + state.db version \"2\" -> incompatible state version. expected \"1\" but found \"2\"\n// after\n# either upgrade the agent binary\nnomad agent -client  # run matching (newer) version\n# or restore compatible state\nmv /var/lib/nomad/state.db /var/lib/nomad/state.db.newer && systemctl start nomad","handlingStrategy":"validation","validationCode":"// check recorded version before running an older binary\nv, err := readStateDBVersion(stateDBPath)\nif err == nil && v > currentBinaryVersion {\n    return fmt.Errorf(\"state.db written by newer Nomad (%s); refusing to start\", v)\n}","typeGuard":"func versionMatches(found, expected string) bool { return found == expected }","tryCatchPattern":"if err := upgradeState(); err != nil {\n    if strings.Contains(err.Error(), \"incompatible state version\") {\n        // restore backup or run matching Nomad version\n        restoreFromBackup(stateDir)\n    }\n}","preventionTips":["Never downgrade Nomad clients without restoring a pre-upgrade state backup","Take state.db.backup before every version change","Keep cluster Nomad versions homogeneous","Capture the found version from logs to diagnose which direction the mismatch is"],"tags":["state-store","upgrade","version-mismatch","nomad"],"backgroundTag":"state-schema-version-mismatch","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}