cockroachdb/cockroach · critical

cannot set both COCKROACH_FORCE_DEV_VERSION and COCKROACH_TE

Error message

cannot set both COCKROACH_FORCE_DEV_VERSION and COCKROACH_TESTING_FORCE_RELEASE_BRANCH

What it means

clusterversion computes the dev version offset at package-init time (devOffsetKeyStart). The two env vars are contradictory: COCKROACH_FORCE_DEV_VERSION=1 forces dev offsetting on, COCKROACH_TESTING_FORCE_RELEASE_BRANCH=1 forces it off (used by upgrade tests simulating real releases). Setting both trips an assertion panic at init, crashing any cockroach binary before it does anything.

Source

Thrown at pkg/clusterversion/dev_offset.go:53

// cluster that is on released version 3 with minimum supported version 2, a dev
// binary containing versions 1, 2, 3, and 4 started with this flag would
// renumber only 2-4 to be +1M. It would then step from 3 "up" to 1000002 -
// which conceptually is actually back down to 2 - then back to 1000003, then on
// to 1000004, etc.
//
// Version offsetting is controlled by the developmentBranch constant, but can
// be overridden with env vars:
//   - COCKROACH_TESTING_FORCE_RELEASE_BRANCH=1 disables version offsetting
//     unconditionally; it is used for certain upgrade tests that are trying to
//     simulate a "real" upgrade.
//   - COCKROACH_FORCE_DEV_VERSION=1 enables version offsetting unconditionally;
//     it is useful if we want to run a final release binary in a cluster that
//     was already created with a dev version.
var devOffsetKeyStart = func() Key {
	forceDev := envutil.EnvOrDefaultBool("COCKROACH_FORCE_DEV_VERSION", false)
	forceRelease := envutil.EnvOrDefaultBool("COCKROACH_TESTING_FORCE_RELEASE_BRANCH", false)
	if forceDev && forceRelease {
		panic(errors.AssertionFailedf("cannot set both COCKROACH_FORCE_DEV_VERSION and COCKROACH_TESTING_FORCE_RELEASE_BRANCH"))
	}
	isDev := (DevelopmentBranch || forceDev) && !forceRelease
	if !isDev {
		// No dev offsets.
		return numKeys + 1
	}
	// If COCKROACH_UPGRADE_TO_DEV_VERSION is set, we allow updating from the
	// minimum supported release, so we only apply the dev offset to subsequent
	// versions.
	allowUpgradeToDev := envutil.EnvOrDefaultBool("COCKROACH_UPGRADE_TO_DEV_VERSION", false)
	if allowUpgradeToDev {
		return MinSupported + 1
	}
	// Apply the dev offset to all versions (except VBootstrap versions, which
	// don't matter for offsetting logic).
	return VBootstrapMax + 1
}()

View on GitHub (pinned to 8812064a01)

Solutions

1. Unset one of the two: `unset COCKROACH_FORCE_DEV_VERSION` or `unset COCKROACH_TESTING_FORCE_RELEASE_BRANCH`
2. Audit the CI job/docker env for both variables and remove the stale one
3. Keep COCKROACH_TESTING_FORCE_RELEASE_BRANCH exclusively inside upgrade tests that simulate real upgrades; keep FORCE_DEV only for running release binaries against dev clusters

Example fix

# before
export COCKROACH_FORCE_DEV_VERSION=1
export COCKROACH_TESTING_FORCE_RELEASE_BRANCH=1
cockroach start

# after
export COCKROACH_FORCE_DEV_VERSION=1  # offsetting ON
unset COCKROACH_TESTING_FORCE_RELEASE_BRANCH
cockroach start

#!/bin/sh
if [ -n "$COCKROACH_FORCE_DEV_VERSION" ] && [ -n "$COCKROACH_TESTING_FORCE_RELEASE_BRANCH" ]; then
  echo "conflicting env: unset one of COCKROACH_FORCE_DEV_VERSION / COCKROACH_TESTING_FORCE_RELEASE_BRANCH" >&2
  exit 1
fi
exec "$@"

When it happens

Trigger: Launching any cockroach binary (server, cli, test) with both COCKROACH_FORCE_DEV_VERSION and COCKROACH_TESTING_FORCE_RELEASE_BRANCH exported — typically upgrade-test leftovers in the shell plus a manually added force-dev variable in CI or a container env.

Common situations: CI pipelines that layer env blocks (base job sets one, an override sets the other); developer shells reused after running upgrade roachtests; Docker images baking in testing env vars.

Related errors

• user.Current: %s
• error while processing file %s
• failed parsing datum string as
• cannot have two files with the same basename in DISTDIR_FILE
• invalid trusted proxy CIDRs

AI-assisted analysis of cockroachdb/cockroach@8812064a01 (2026-08-15). Data as JSON: /api/errors/9732ad78660a13b7. Report an issue: GitHub.