huggingface/transformers · error · HTTPException

Audio transcription requires sequential generation (not CB)

Error message

Audio transcription requires sequential generation (not CB)

What it means

HTTP 400 from the transcription handler when the model's registered manager in GenerationState is not a GenerateManager, i.e. the server is running the model in continuous-batching (CB) mode. The transcription path is only implemented on the sequential generation manager, so CB-served models cannot serve /v1/audio/transcriptions.

Source

Thrown at src/transformers/cli/serving/transcription.py:119

                "Missing python-multipart dependency for file uploads. Install with `pip install python-multipart`"
            )

        async with request.form() as form:
            self._validate_request(set(form.keys()))
            file_field = form["file"]
            if isinstance(file_field, str):
                raise HTTPException(status_code=422, detail="Expected file upload, got string")
            file_bytes = await file_field.read()
            model = form["model"]
            if not isinstance(model, str):
                raise HTTPException(status_code=422, detail="Expected model name as string")
            stream = str(form.get("stream", "false")).lower() == "true"

        model_id_and_revision = self.model_manager.process_model_name(model)
        audio_model, audio_processor = self.model_manager.load_model_and_processor(model_id_and_revision)
        base_manager = self.generation_state.get_manager(model_id_and_revision)
        if not isinstance(base_manager, GenerateManager):
            raise HTTPException(status_code=400, detail="Audio transcription requires sequential generation (not CB)")
        gen_manager = base_manager
        audio_inputs = self._prepare_audio_inputs(file_bytes, audio_processor, audio_model)

        if stream:
            return self._streaming(gen_manager, audio_model, audio_processor, audio_inputs)
        return await self._non_streaming(gen_manager, audio_model, audio_processor, audio_inputs)

    @staticmethod
    def _prepare_audio_inputs(
        file_bytes: bytes, audio_processor: "ProcessorMixin", audio_model: "PreTrainedModel"
    ) -> dict:
        """Load audio bytes and convert to model inputs."""
        import librosa

        sampling_rate = audio_processor.feature_extractor.sampling_rate
        audio_array, _ = librosa.load(io.BytesIO(file_bytes), sr=sampling_rate, mono=True)
        audio_inputs = audio_processor(audio_array, sampling_rate=sampling_rate, return_tensors="pt").to(
            audio_model.device

View on GitHub (pinned to a597f97485)

Solutions

1. Restart the serving CLI without the continuous batching flag so a sequential GenerateManager handles the model
2. Run a second server instance without CB dedicated to audio transcription
3. Route transcription traffic to a deployment configured for sequential generation

Example fix

# before
$ transformers serve --cb --model openai/whisper-large-v3
# after
$ transformers serve --model openai/whisper-large-v3

# before shipping transcription traffic, probe capability
health = requests.get(f'{base}/health').json()  # or config endpoint exposing cb mode
if health.get('cb_enabled'):
    route_transcription_to_sequential_server()

if resp.status_code == 400 and 'sequential generation' in resp.text:
        raise RuntimeError('transcription unsupported in CB mode; use a non-CB deployment') from None

When it happens

Trigger: Starting the server with continuous batching enabled (e.g. --cb / CB flags) and then POSTing to /v1/audio/transcriptions with a model handled by the CB engine.

Common situations: A single serving deployment serving both chat completions (CB for throughput) and audio transcriptions; enabling CB for benchmarking then forgetting it blocks transcription.

Related errors

• prompt must be a string.
• Unexpected fields in the request: {unexpected}
• Missing librosa dependency for audio transcription. Install
• CB worker is dead and cannot accept request {request_id}: {s
• CB worker died during request {request_id}: {result.error}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/5a333120f42ef4e0. Report an issue: GitHub.