{"record":{"id":"c8c896087b737406","repo":"hiyouga/LlamaFactory","slug":"batch-infer-is-not-implemented","errorCode":null,"errorMessage":"Batch infer is not implemented.","messagePattern":"Batch infer is not implemented\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/utils/inference_engine.py","lineNumber":121,"sourceCode":"                \"max_new_tokens\": self.args.max_new_tokens,\n                \"streamer\": streamer,\n            }\n            thread = Thread(target=self.model.generate, kwargs=kwargs, daemon=True)\n            thread.start()\n\n            async for token in streamer:\n                yield token\n\n    async def batch_infer(self, dataset: TorchDataset) -> list[Sample]:\n        \"\"\"Batch infer samples.\n\n        Args:\n            dataset: Torch dataset.\n\n        Returns:\n            List of samples.\n        \"\"\"\n        raise NotImplementedError(\"Batch infer is not implemented.\")\n","sourceCodeStart":103,"sourceCodeEnd":122,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/utils/inference_engine.py#L103-L122","documentation":"Raised by the v1 InferenceEngine base class when batch_infer() is called. The base class only implements streaming inference (infer); batch inference over a TorchDataset is declared but intentionally left unimplemented. Any backend that does not override batch_infer inherits this NotImplementedError.","triggerScenarios":"Calling engine.batch_infer(dataset) on an InferenceEngine subclass that never overrides batch_infer (the abstract stream method is implemented, but batch_infer is not).","commonSituations":"Using the experimental v1 API (USE_V1=1) and assuming the batch-inference entry point works like v0's; a custom engine plugin that implements only the streaming path.","solutions":["Use the streaming API instead: iterate engine.infer(sample) per sample.","If you own the engine subclass, implement batch_infer (e.g. loop the streaming path or use vLLM offline batching).","Fall back to v0 (unset USE_V1) which has mature batch inference.","File a feature request upstream if a built-in engine lacks it."],"exampleFix":"// before\nresults = engine.batch_infer(dataset)\n\n# after\nresults = [await engine.infer(sample) async for sample in ...]\n# or implement in subclass:\nasync def batch_infer(self, dataset):\n    return [await self.infer(ds[i]) for i in range(len(ds))]","handlingStrategy":"type-guard","validationCode":"import inspect\n\ndef supports_batch_infer(engine) -> bool:\n    cls = type(engine).batch_infer\n    base = __import__('llamafactory.v1.core.utils.inference_engine', fromlist=['InferenceEngine']).InferenceEngine.batch_infer\n    return cls is not base and not getattr(cls, '__isabstractmethod__', False)","typeGuard":"def supports_batch_infer(engine) -> bool:\n    \"\"\"True when the engine overrides batch_infer.\"\"\"\n    return type(engine).batch_infer is not InferenceEngine.batch_infer","tryCatchPattern":"try:\n    results = await engine.batch_infer(dataset)\nexcept NotImplementedError:\n    results = [await engine.infer(s) for s in dataset]  # streaming fallback","preventionTips":["Feature-detect batch_infer before calling it.","Prefer the streaming infer() API in v1.","Pin the architecture choice (v0 vs v1) explicitly in training scripts."],"tags":["inference","not-implemented","v1-api"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}