hiyouga/LlamaFactory · error · NotImplementedError

Stage does not supported: {stage}.

Error message

Stage does not supported: {stage}.

What it means

The HyperParallel workflow builds HyperParallelArguments from finetuning args (workflow.py:48); the integration lives in the separate `hyper_parallel` package, so the path first checks is_hyper_parallel_available() and raises ImportError with install instructions when the package is absent. This is the entry gate before any HP feature (CP, activation optimization) can be used.

Source

Thrown at scripts/stat_utils/cal_lr.py:79

            packing=packing,
            preprocessing_num_workers=16,
            output_dir="dummy_dir",
            overwrite_cache=True,
            do_train=True,
        )
    )
    tokenizer_module = load_tokenizer(model_args)
    tokenizer = tokenizer_module["tokenizer"]
    template = get_template_and_fix_tokenizer(tokenizer, data_args)
    trainset = get_dataset(template, model_args, data_args, training_args, stage, **tokenizer_module)["train_dataset"]
    if stage == "pt":
        data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
    elif stage == "sft":
        data_collator = MultiModalDataCollatorForSeq2Seq(
            template=template, tokenizer=tokenizer, label_pad_token_id=IGNORE_INDEX
        )
    else:
        raise NotImplementedError(f"Stage does not supported: {stage}.")

    dataloader = DataLoader(trainset, batch_size, shuffle=False, collate_fn=data_collator, pin_memory=True)
    valid_tokens, total_tokens = 0, 0
    for batch in tqdm(dataloader, desc="Collecting valid tokens"):
        valid_tokens += torch.sum(batch["labels"] != IGNORE_INDEX).item()
        total_tokens += torch.numel(batch["labels"])

    valid_ratio = valid_tokens / total_tokens
    token_batch_size = cutoff_len * batch_size * valid_ratio
    lr = BASE_LR * math.sqrt(token_batch_size / BASE_BS)  # lr ~ sqrt(batch_size)
    lr = lr / 6.0 if is_mistral_or_gemma else lr
    print(
        f"Optimal learning rate is {lr:.2e} for valid ratio% {valid_ratio * 100:.2f} "
        f"and effective token batch size {token_batch_size:.2f}"
    )


if __name__ == "__main__":

View on GitHub (pinned to f28afaf635)

Solutions

1. pip install hyper_parallel
2. If the import still fails after install, verify with `python -c "import hyper_parallel"` and check for conflicting packages (e.g. mismatched torch) before retrying training
3. If you don't need HP features, remove the HyperParallel args from the YAML

Example fix

# before
# YAML has hyper_parallel_cp_size: 4, package missing -> ImportError

# after
pip install hyper_parallel

from llamafactory.extras.packages import is_hyper_parallel_available
if uses_hyper_parallel(finetuning_args):
    assert is_hyper_parallel_available(), 'pip install hyper_parallel'

def hyper_parallel_ready() -> bool:
    try:
        import hyper_parallel  # noqa: F401
        return True
    except ImportError:
        return False

try:
    run_exp()
except ImportError as e:
    if 'hyper_parallel' in str(e):
        raise SystemExit('pip install hyper_parallel') from e
    raise

When it happens

Trigger: Setting HyperParallel-related finetuning args (e.g. hyper_parallel_cp_size > 1 or an activation_mode) in an environment where `import hyper_parallel` fails; _prepare_hp_args raises immediately during workflow setup.

Common situations: Enabling experimental HP features in a base LlamaFactory install; fresh clones where extras/requirements-hyper-parallel were not installed; version drift making the hyper_parallel import fail even though it is listed.

Related errors

• Qwen2VL requires 3D position ids for mrope.
• Pipeline parallel size should be smaller than the number of
• Only supports u/a/u/a/u...
• hyper_parallel is not installed. Please install it with `pip
• All 'dcp_path', 'hf_path', and 'config_path' are required.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/12ac2b91e1a05509. Report an issue: GitHub.