docling-project/docling · error · ValueError

Could not find assistant response in decoded text

Error message

Could not find assistant response in decoded text

What it means

The Granite chart-extraction model decodes the LLM output and expects a chat-format response containing an `<|assistant|>` tag. This ValueError is raised when the regex `<\|assistant\|>\s*(.*)` finds no match in the decoded text, meaning the model's raw output does not follow the expected chat template (e.g. it is empty, truncated, or a plain completion without chat markers). It almost always indicates a generation/pipeline issue rather than a document problem.

Source

Thrown at docling/models/stages/chart_extraction/granite_vision.py:319

            except Exception as e:
                _log.error(f"Failed to extract DataFrame for image {i}: {e}")
                chart_data.append(None)

        return chart_data

    def _extract_csv_to_dataframe(self, decoded_text: str) -> pd.DataFrame:
        """
        Extract CSV content from decoded text and convert to DataFrame.

        Handles:
        - Chat format with <|assistant|> tags
        - Nested code blocks (```csv ``` inside ```)
        - Various CSV formatting issues
        """
        # Extract the assistant's response
        assistant_match = re.search(r"<\|assistant\|>\s*(.*)", decoded_text, re.DOTALL)
        if not assistant_match:
            raise ValueError("Could not find assistant response in decoded text")

        assistant_response = assistant_match.group(1).strip()

        # Extract the first CSV code block (```csv ... ```)
        csv_match = re.search(r"```csv\s*\n(.*?)\n```", assistant_response, re.DOTALL)
        if csv_match:
            csv_content = csv_match.group(1).strip()
        else:
            # Fallback: take content up to first <|end_of_text|> and strip code block markers
            csv_content = assistant_response.split("<|end_of_text|>")[0].strip()
            csv_content = re.sub(r"^```+(?:csv)?\s*", "", csv_content)
            csv_content = re.sub(r"```+\s*$", "", csv_content)
            csv_content = csv_content.strip()

        try:
            dataframe = pd.read_csv(StringIO(csv_content), header=None)
            return dataframe
        except Exception as e:

View on GitHub (pinned to 61d76f1ff3)

Solutions

  1. Inspect the decoded text (log it) to see what the model actually returned — empty, truncated, or differently formatted.
  2. Verify the prompt was built with the model's chat template (apply_chat_template with add_generation_prompt=True) so the model emits `<|assistant|>`.
  3. Increase generation limits (max_new_tokens) so the assistant response is not truncated before it begins.
  4. Pin the granite-vision model revision known to work with this stage; if the checkpoint changed its template, update the regex/extraction to match.

Example fix

# before
assistant_match = re.search(r"<\|assistant\|>\s*(.*)", decoded_text, re.DOTALL)
if not assistant_match:
    raise ValueError("Could not find assistant response in decoded text")

# after — fall back to the whole decoded text when no chat marker is present
assistant_match = re.search(r"<\|assistant\|>\s*(.*)", decoded_text, re.DOTALL)
if assistant_match:
    assistant_response = assistant_match.group(1).strip()
else:
    _log.warning("No <|assistant|> tag in Granite output; using raw decoded text")
    assistant_response = decoded_text.strip()
Defensive patterns

Strategy: try-catch

Validate before calling

import re

def has_assistant_response(decoded_text: str) -> bool:
    return re.search(r"<\|assistant\|>\s*(.)", decoded_text, re.DOTALL) is not None

Try / catch

try:
    df = stage._extract_csv_to_dataframe(decoded)
except ValueError as err:
    if "assistant response" in str(err):
        logger.warning("Granite output lacked assistant tag; skipping chart: %s", decoded[:200])
        continue  # skip this chart rather than fail the document
    raise

Prevention

When it happens

Trigger: Calling the granite-vision chart-to-CSV stage when the underlying Granite VLM returns output without any `<|assistant|>` marker: truncated generation (max_new_tokens hit early), a model revision that changed the chat template, wrong processor/chat-template application, or empty decoder output.

Common situations: Using the chart extraction stage with a newer/older granite-vision checkpoint whose prompt format differs; low max token limits producing cut-off output; passing a non-chat-formatted prompt; decoding a batch where the assistant turn was dropped.

Related errors

AI-assisted analysis of docling-project/docling@61d76f1ff3 (2026-08-14). Data as JSON: /api/errors/08f08ed78b0b0c34. Report an issue: GitHub.