hiyouga/LlamaFactory · error · NotImplementedError

Batch infer is not implemented.

Error message

Batch infer is not implemented.

What it means

Raised by the v1 InferenceEngine base class when batch_infer() is called. The base class only implements streaming inference (infer); batch inference over a TorchDataset is declared but intentionally left unimplemented. Any backend that does not override batch_infer inherits this NotImplementedError.

Source

Thrown at src/llamafactory/v1/core/utils/inference_engine.py:121

                "max_new_tokens": self.args.max_new_tokens,
                "streamer": streamer,
            }
            thread = Thread(target=self.model.generate, kwargs=kwargs, daemon=True)
            thread.start()

            async for token in streamer:
                yield token

    async def batch_infer(self, dataset: TorchDataset) -> list[Sample]:
        """Batch infer samples.

        Args:
            dataset: Torch dataset.

        Returns:
            List of samples.
        """
        raise NotImplementedError("Batch infer is not implemented.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Use the streaming API instead: iterate engine.infer(sample) per sample.
2. If you own the engine subclass, implement batch_infer (e.g. loop the streaming path or use vLLM offline batching).
3. Fall back to v0 (unset USE_V1) which has mature batch inference.
4. File a feature request upstream if a built-in engine lacks it.

Example fix

// before
results = engine.batch_infer(dataset)

# after
results = [await engine.infer(sample) async for sample in ...]
# or implement in subclass:
async def batch_infer(self, dataset):
    return [await self.infer(ds[i]) for i in range(len(ds))]

import inspect

def supports_batch_infer(engine) -> bool:
    cls = type(engine).batch_infer
    base = __import__('llamafactory.v1.core.utils.inference_engine', fromlist=['InferenceEngine']).InferenceEngine.batch_infer
    return cls is not base and not getattr(cls, '__isabstractmethod__', False)

def supports_batch_infer(engine) -> bool:
    """True when the engine overrides batch_infer."""
    return type(engine).batch_infer is not InferenceEngine.batch_infer

try:
    results = await engine.batch_infer(dataset)
except NotImplementedError:
    results = [await engine.infer(s) for s in dataset]  # streaming fallback

When it happens

Trigger: Calling engine.batch_infer(dataset) on an InferenceEngine subclass that never overrides batch_infer (the abstract stream method is implemented, but batch_infer is not).

Common situations: Using the experimental v1 API (USE_V1=1) and assuming the batch-inference entry point works like v0's; a custom engine plugin that implements only the streaming path.

Related errors

• SGLang only supports n=1.
• SGLang engine does not support `get_scores`.
• vLLM engine does not support `get_scores`.
• Environment information is not implemented yet.
• Version information is not implemented yet.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/c8c896087b737406. Report an issue: GitHub.