karpathy/nanochat · error · ValueError

No training horizon specified

Error message

No training horizon specified

What it means

scripts/base_train.py computes the training length (num_iterations) from CLI args in priority order: --num-iterations, then --target-flops, then --target-param-data-ratio. If all three are 0/unset, there is no way to size the run, so it raises ValueError('No training horizon specified'). Note line 339 already asserts at least one is positive, so in practice you would hit the assert first; the ValueError is the same guard restated.

Source

Thrown at scripts/base_train.py:353

# -----------------------------------------------------------------------------
# Calculate the number of iterations we will train for and set up the various schedulers

# num_iterations: either it is given, or from target flops, or from target data:param ratio (in that order)
assert args.num_iterations > 0 or args.target_param_data_ratio > 0 or args.target_flops > 0
if args.num_iterations > 0:
    # Override num_iterations to a specific value if given
    num_iterations = args.num_iterations
    print0(f"Using user-provided number of iterations: {num_iterations:,}")
elif args.target_flops > 0:
    # Calculate the number of iterations from the target flops (used in scaling laws analysis, e.g. runs/scaling_laws.sh)
    num_iterations = round(args.target_flops / (num_flops_per_token * total_batch_size))
    print0(f"Calculated number of iterations from target FLOPs: {num_iterations:,}")
elif args.target_param_data_ratio > 0:
    # Calculate the number of iterations from the target param data ratio (the most common use case)
    num_iterations = target_tokens // total_batch_size
    print0(f"Calculated number of iterations from target data:param ratio: {num_iterations:,}")
else:
    raise ValueError("No training horizon specified")
total_tokens = total_batch_size * num_iterations # the actual number of tokens we will train for
print0(f"Total number of training tokens: {total_tokens:,}")
print0(f"Tokens : Scaling params ratio: {total_batch_size * num_iterations / num_scaling_params:.2f}") # e.g. Chinchilla was ~20
print0(f"Total training FLOPs estimate: {num_flops_per_token * total_tokens:e}")

# Learning rate schedule (linear warmup, constant, linear warmdown)
def get_lr_multiplier(it):
    warmup_iters = args.warmup_steps
    warmdown_iters = round(args.warmdown_ratio * num_iterations)
    if it < warmup_iters:
        return (it + 1) / warmup_iters
    elif it <= num_iterations - warmdown_iters:
        return 1.0
    else:
        progress = (num_iterations - it) / warmdown_iters
        return progress * 1.0 + (1 - progress) * args.final_lr_frac

# Momentum scheduler for Muon optimizer (warms up to 0.97, warms down to 0.90 during LR warmdown)

View on GitHub (pinned to 92d63d4e8b)

Solutions

1. Pass --target-param-data-ratio (typical Chinchilla-style choice, e.g. 20) for standard runs.
2. Or pass --num-iterations N for an exact step count.
3. Or pass --target-flops F for scaling-law runs.
4. Check your run script still contains one of these three flags after edits.

Example fix

# before
torchrun --nproc_per_node=8 -m scripts.base_train

# after
torchrun --nproc_per_node=8 -m scripts.base_train --target-param-data-ratio 20

assert args.num_iterations > 0 or args.target_flops > 0 or args.target_param_data_ratio > 0, (
    "specify one of --num-iterations, --target-flops, --target-param-data-ratio"
)

When it happens

Trigger: Running base_train.py without any of --num-iterations, --target-flops, or --target-param-data-ratio (all default to 0), or explicitly passing 0/negative values for all three.

Common situations: A new launch command copied without the horizon flags; a runs/*.sh script edited and the ratio flag accidentally removed; overriding flags to 0 expecting a default horizon (there is none).

Related errors

• Unknown dataset tag: {dataset_tag}
• Only 'tensorwise' recipe is supported, got '{recipe_name}'.
• Unknown optimizer kind: {group['kind']}
• No checkpoints found in {checkpoints_dir}
• No checkpoints found in {checkpoint_dir}

AI-assisted analysis of karpathy/nanochat@92d63d4e8b (2026-08-15). Data as JSON: /api/errors/d6a356dcbc61bcd4. Report an issue: GitHub.