tensorflow/models · error · ValueError

Distribution strategy is not specified.

Error message

Distribution strategy is not specified.

What it means

Guard in run_bert_pretrain: the strategy argument is None/falsy, so no distribution strategy exists for pre-training. It usually means main() did not build a strategy (e.g. distribution_utils.get_distribution_strategy returned None or was skipped) before calling the pre-training routine.

Solutions

1. Create the strategy in main: strategy = distribution_utils.get_distribution_strategy(FLAGS.strategy_type) and pass it into run_bert_pretrain
2. Verify the strategy_type/ tpu flags so a non-None strategy is produced

When it happens

Trigger: Thrown at official/legacy/bert/run_pretraining.py:157 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/08b9c154ec790a69. Report an issue: GitHub.

Appendix: source

Thrown at official/legacy/bert/run_pretraining.py:157

      steps_per_loop=steps_per_loop,
      epochs=epochs,
      sub_model_export_name='pretrained/bert_model',
      explicit_allreduce=explicit_allreduce,
      pre_allreduce_callbacks=pre_allreduce_callbacks,
      post_allreduce_callbacks=post_allreduce_callbacks,
      allreduce_bytes_per_pack=allreduce_bytes_per_pack,
      train_summary_interval=train_summary_interval,
      custom_callbacks=custom_callbacks)

  return trained_model


def run_bert_pretrain(strategy, custom_callbacks=None):
  """Runs BERT pre-training."""

  bert_config = configs.BertConfig.from_json_file(FLAGS.bert_config_file)
  if not strategy:
    raise ValueError('Distribution strategy is not specified.')

  # Runs customized training loop.
  logging.info('Training using customized training loop TF 2.0 with distributed'
               'strategy.')

  performance.set_mixed_precision_policy(common_flags.dtype())

  # Only when explicit_allreduce = True, post_allreduce_callbacks and
  # allreduce_bytes_per_pack will take effect. optimizer.apply_gradients() no
  # longer implicitly allreduce gradients, users manually allreduce gradient and
  # pass the allreduced grads_and_vars to apply_gradients().
  # With explicit_allreduce = True, clip_by_global_norm is moved to after
  # allreduce.
  return run_customized_training(
      strategy,
      bert_config,
      FLAGS.init_checkpoint,  # Used to initialize only the BERT submodel.
      FLAGS.max_seq_length,

View on GitHub (pinned to e006f5f0d5)