hashicorp/nomad · critical

incompatible state version. expected %q but found %q

Error message

incompatible state version. expected %q but found %q

What it means

Aborts state DB upgrade when the database's meta version does not match the version the current upgrade path expects. The upgrade code opens a transaction, reads the version bucket, and refuses to proceed on any mismatch — protecting against running an upgrade against a DB from the wrong Nomad version.

Source

Thrown at client/state/upgrade.go:49

		v := b.Get(metaVersionKey)
		if len(v) == 0 {
			// No version; upgrade
			return nil
		}

		if bytes.Equal(v, []byte{'2'}) {
			upgradeTo09 = false
			return nil
		}
		if bytes.Equal(v, metaVersion) {
			upgradeTo09 = false
			upgradeTo13 = false
			return nil
		}

		// Version exists but does not match. Abort.
		return fmt.Errorf("incompatible state version. expected %q but found %q",
			metaVersion, v)

	})

	return
}

// addMeta adds version metadata to BoltDB to mark it as upgraded and
// should be run at the end of the upgrade transaction.
func addMeta(tx *bbolt.Tx) error {
	// Create the meta bucket if it doesn't exist
	bkt, err := tx.CreateBucketIfNotExists(metaBucketName)
	if err != nil {
		return err
	}
	return bkt.Put(metaVersionKey, metaVersion)
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped/found version value: if it is newer than the agent's, you are downgrading — restore the newer Nomad version or restore a pre-upgrade state.db.backup.
2. If the DB was already upgraded, remove/rotate state.db and let the client rebuild state.
3. Never hand-edit the version bucket; restore from backup instead.
4. Pin Nomad versions across the cluster to avoid mixed-version state writes.

Example fix

// before
nomad 0.8 agent + state.db version "2" -> incompatible state version. expected "1" but found "2"
// after
# either upgrade the agent binary
nomad agent -client  # run matching (newer) version
# or restore compatible state
mv /var/lib/nomad/state.db /var/lib/nomad/state.db.newer && systemctl start nomad

// check recorded version before running an older binary
v, err := readStateDBVersion(stateDBPath)
if err == nil && v > currentBinaryVersion {
    return fmt.Errorf("state.db written by newer Nomad (%s); refusing to start", v)
}

func versionMatches(found, expected string) bool { return found == expected }

if err := upgradeState(); err != nil {
    if strings.Contains(err.Error(), "incompatible state version") {
        // restore backup or run matching Nomad version
        restoreFromBackup(stateDir)
    }
}

When it happens

Trigger: During upgrade (e.g. to 0.9/1.3 schema) the version read from the meta bucket is not the expected metaVersion — e.g. DB was already upgraded, was written by a newer Nomad, or the version key is corrupted.

Common situations: Downgrading Nomad on a client whose state.db was upgraded by a newer version; a previously half-completed upgrade; manually edited or restored state.db with a stale version record.

Related errors

• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state
• eval not found
• deployment promotion cannot be undone

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3229876af2bec9be. Report an issue: GitHub.