cockroachdb/cockroach · critical

cannot set both COCKROACH_FORCE_DEV_VERSION and…

Error message

cannot set both COCKROACH_FORCE_DEV_VERSION and COCKROACH_TESTING_FORCE_RELEASE_BRANCH

What it means

clusterversion computes the dev version offset at package-init time (devOffsetKeyStart). The two env vars are contradictory: COCKROACH_FORCE_DEV_VERSION=1 forces dev offsetting on, COCKROACH_TESTING_FORCE_RELEASE_BRANCH=1 forces it off (used by upgrade tests simulating real releases). Setting both trips an assertion panic at init, crashing any cockroach binary before it does anything.

Solutions

1. Unset one of the two: `unset COCKROACH_FORCE_DEV_VERSION` or `unset COCKROACH_TESTING_FORCE_RELEASE_BRANCH`
2. Audit the CI job/docker env for both variables and remove the stale one
3. Keep COCKROACH_TESTING_FORCE_RELEASE_BRANCH exclusively inside upgrade tests that simulate real upgrades; keep FORCE_DEV only for running release binaries against dev clusters

Example fix

# before
export COCKROACH_FORCE_DEV_VERSION=1
export COCKROACH_TESTING_FORCE_RELEASE_BRANCH=1
cockroach start

# after
export COCKROACH_FORCE_DEV_VERSION=1  # offsetting ON
unset COCKROACH_TESTING_FORCE_RELEASE_BRANCH
cockroach start

#!/bin/sh
if [ -n "$COCKROACH_FORCE_DEV_VERSION" ] && [ -n "$COCKROACH_TESTING_FORCE_RELEASE_BRANCH" ]; then
  echo "conflicting env: unset one of COCKROACH_FORCE_DEV_VERSION / COCKROACH_TESTING_FORCE_RELEASE_BRANCH" >&2
  exit 1
fi
exec "$@"

When it happens

Trigger: Launching any cockroach binary (server, cli, test) with both COCKROACH_FORCE_DEV_VERSION and COCKROACH_TESTING_FORCE_RELEASE_BRANCH exported — typically upgrade-test leftovers in the shell plus a manually added force-dev variable in CI or a container env.

Common situations: CI pipelines that layer env blocks (base job sets one, an override sets the other); developer shells reused after running upgrade roachtests; Docker images baking in testing env vars.

Related errors

• user.Current
• cannot have two files with the same basename in…
• could not parse query (expected PKG.TEST)
• error while processing file
• failed parsing datum string as

AI-assisted analysis of cockroachdb/cockroach@8812064a01 (2026-08-15). Data as JSON: /api/errors/9732ad78660a13b7. Report an issue: GitHub.

Appendix: source

Thrown at pkg/clusterversion/dev_offset.go:53

// cluster that is on released version 3 with minimum supported version 2, a dev
// binary containing versions 1, 2, 3, and 4 started with this flag would
// renumber only 2-4 to be +1M. It would then step from 3 "up" to 1000002 -
// which conceptually is actually back down to 2 - then back to 1000003, then on
// to 1000004, etc.
//
// Version offsetting is controlled by the developmentBranch constant, but can
// be overridden with env vars:
//   - COCKROACH_TESTING_FORCE_RELEASE_BRANCH=1 disables version offsetting
//     unconditionally; it is used for certain upgrade tests that are trying to
//     simulate a "real" upgrade.
//   - COCKROACH_FORCE_DEV_VERSION=1 enables version offsetting unconditionally;
//     it is useful if we want to run a final release binary in a cluster that
//     was already created with a dev version.
var devOffsetKeyStart = func() Key {
	forceDev := envutil.EnvOrDefaultBool("COCKROACH_FORCE_DEV_VERSION", false)
	forceRelease := envutil.EnvOrDefaultBool("COCKROACH_TESTING_FORCE_RELEASE_BRANCH", false)
	if forceDev && forceRelease {
		panic(errors.AssertionFailedf("cannot set both COCKROACH_FORCE_DEV_VERSION and COCKROACH_TESTING_FORCE_RELEASE_BRANCH"))
	}
	isDev := (DevelopmentBranch || forceDev) && !forceRelease
	if !isDev {
		// No dev offsets.
		return numKeys + 1
	}
	// If COCKROACH_UPGRADE_TO_DEV_VERSION is set, we allow updating from the
	// minimum supported release, so we only apply the dev offset to subsequent
	// versions.
	allowUpgradeToDev := envutil.EnvOrDefaultBool("COCKROACH_UPGRADE_TO_DEV_VERSION", false)
	if allowUpgradeToDev {
		return MinSupported + 1
	}
	// Apply the dev offset to all versions (except VBootstrap versions, which
	// don't matter for offsetting logic).
	return VBootstrapMax + 1
}()

View on GitHub (pinned to 8812064a01)