docling-project/docling · error · RuntimeError

Neither processor.batch_decode nor tokenizer.batch_decode is

Error message

Neither processor.batch_decode nor tokenizer.batch_decode is available.

What it means

After generation, TransformersVlmEngine decodes the output token ids via processor.batch_decode, falling back to tokenizer.batch_decode. If neither object exposes batch_decode, decoding cannot proceed and a RuntimeError is raised — this indicates an exotic or incompatible processor/tokenizer pair rather than a usage mistake.

Source

Thrown at docling/models/inference_engines/vlm/transformers_engine.py:474

            gen_kwargs["do_sample"] = False

        if stopping_criteria_list:
            gen_kwargs["stopping_criteria"] = stopping_criteria_list

        start_time = time.time()
        with torch.inference_mode():
            generated_ids = self.vlm_model.generate(**gen_kwargs)  # type: ignore[union-attr,operator]
        generation_time = time.time() - start_time

        # Decode
        input_len = inputs["input_ids"].shape[1]
        trimmed_sequences = generated_ids[:, input_len:]

        decode_fn = getattr(self.processor, "batch_decode", None)
        if decode_fn is None and tokenizer is not None:
            decode_fn = getattr(tokenizer, "batch_decode", None)
        if decode_fn is None:
            raise RuntimeError(
                "Neither processor.batch_decode nor tokenizer.batch_decode is available."
            )

        decoded_texts = decode_fn(trimmed_sequences, **decoder_config)

        # Remove padding
        pad_token = getattr(tokenizer, "pad_token", None)
        if pad_token:
            decoded_texts = [text.rstrip(pad_token) for text in decoded_texts]

        if self.strip_stop_strings and first_input.stop_strings:
            from docling.utils.vlm_utils import strip_stop_strings

            decoded_texts = strip_stop_strings(decoded_texts, first_input.stop_strings)

        # Create outputs
        outputs = []
        for i, text in enumerate(decoded_texts):

View on GitHub (pinned to 61d76f1ff3)

Solutions

Switch to a known-good VLM repo (e.g. the Docling-supported SmolDocling, GraniteDocling, or Phi-4 with pinned transformers)
Update or pin transformers to a version compatible with the chosen model's processor
Inspect the processor: python -c "from transformers import AutoProcessor; p = AutoProcessor.from_pretrained('<repo>'); print(hasattr(p, 'batch_decode'))"

Example fix

# before
model_config = EngineModelConfig(repo_id='<exotic-community-vlm>')  # processor lacks batch_decode

# after
model_config = EngineModelConfig(repo_id='ds4sd/SmolDocling-256M-preview')  # supported processor

Defensive patterns

Strategy: validation

Validate before calling

from transformers import AutoProcessor

p = AutoProcessor.from_pretrained(repo_id)
has_decode = hasattr(p, 'batch_decode')
# also check a tokenizer if present
try:
    tok = p.tokenizer
except AttributeError:
    tok = None
assert has_decode or (tok is not None and hasattr(tok, 'batch_decode')), (
    f'{repo_id} processor exposes no batch_decode; use a supported VLM repo'
)

Try / catch

try:
    outputs = engine.predict_batch(inputs)
except RuntimeError as e:
    if 'batch_decode' in str(e):
        raise SystemExit(f'Processor for {engine.model_config.repo_id} lacks batch_decode; switch to a supported VLM repo or compatible transformers version') from e
    raise

Prevention

Stick to the model repos Docling lists as supported for the Transformers engine
Smoke-test one inference after loading any new/community model to catch decode incompatibilities early
Pin transformers to a version validated for your chosen model

When it happens

Trigger: Loading a repo whose processor class lacks batch_decode (e.g. some new/legacy processor types) and whose tokenizer is also None or lacks the method, then running predict_batch to completion of generation.

Common situations: Very new transformers releases with changed processor APIs; custom or community VLM repos with non-standard preprocessing classes; models loaded with a processor-only config where no tokenizer object is attached.

Related errors

AI-assisted analysis of docling-project/docling@61d76f1ff3 (2026-08-14). Data as JSON: /api/errors/8da15397afa52686. Report an issue: GitHub.