tensorflow/models · error · ValueError

TPUStrategy should not run eagerly as it heavily relies on…

Error message

TPUStrategy should not run eagerly as it heavily relies on graph optimization for the distributed system.

What it means

Raised when run_eagerly=True is combined with a TPUStrategy (or experimental TPUStrategy) in run_customized_training_loop. TPU distribution depends on graph-mode compilation and clustering, so eager execution on TPU is unsupported and this guard rejects the configuration before training starts.

Solutions

1. Leave run_eagerly at its default False when running on TPU
2. Use tf.distribute.MirroredStrategy or another CPU/GPU strategy if you truly need eager execution for debugging

When it happens

Trigger: Thrown at official/legacy/bert/model_training_utils.py:251 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/df30dae2372d213b. Report an issue: GitHub.

Appendix: source

Thrown at official/legacy/bert/model_training_utils.py:251

      # default users to a more useful value.
      steps_per_loop = min(1000, steps_between_evals)
    else:
      steps_per_loop = 1
    logging.info('steps_per_loop not specified. Using steps_per_loop=%d',
                 steps_per_loop)
  if steps_per_loop > steps_between_evals:
    logging.warning(
        'steps_per_loop: %d is specified to be greater than '
        ' steps_between_evals: %d, we will use steps_between_evals as'
        ' steps_per_loop.', steps_per_loop, steps_between_evals)
    steps_per_loop = steps_between_evals
  assert tf.executing_eagerly()

  if run_eagerly:
    if isinstance(
        strategy,
        (tf.distribute.TPUStrategy, tf.distribute.experimental.TPUStrategy)):
      raise ValueError(
          'TPUStrategy should not run eagerly as it heavily relies on graph'
          ' optimization for the distributed system.')

  if eval_input_fn and eval_steps is None:
    raise ValueError(
        '`eval_step` is required when `eval_input_fn ` is not none.')
  if metric_fn and not callable(metric_fn):
    raise ValueError(
        'if `metric_fn` is specified, metric_fn must be a callable.')

  total_training_steps = steps_per_epoch * epochs  # pyrefly: ignore[unsupported-operation]
  train_iterator = _get_input_iterator(train_input_fn, strategy)
  eval_loss_metric = tf_keras.metrics.Mean('training_loss', dtype=tf.float32)

  with distribute_utils.get_strategy_scope(strategy):
    # To correctly place the model weights on accelerators,
    # model and optimizer should be created in scope.
    model, sub_model = model_fn()  # pyrefly: ignore[not-callable]

View on GitHub (pinned to e006f5f0d5)