binary-husky/gpt_academic · warning · TimeoutError

文本分割超时（5秒）

Error message

文本分割超时（5秒）

What it means

This message represents the intended 5-second budget for breakdown_text_to_satisfy_token_limit. In this implementation the clock is checked only before a single synchronous call; a normal split that takes longer returns and breaks successfully. The error is visible only when control re-enters the loop, the global 30-second timer has already fired, or the clock changes. It is not a hard timeout for the tokenizer.

Source

Thrown at crazy_functions/Document_Conversation.py:153

                self.failed_files.append((fp, "文件内容为空"))
                mutable_status[2] = "内容为空"
                return fragments

            check_timeout()

            # 更新状态
            mutable_status[0] = "分割文本"
            mutable_status[1] = time.time()

            # 分割文本 - 添加超时检查
            split_start_time = time.time()
            try:
                while True:
                    check_timeout()  # 检查全局超时

                    # 检查分割过程是否超时（5秒）
                    if time.time() - split_start_time > 5:
                        raise TimeoutError("文本分割超时（5秒）")

                    paper_fragments = breakdown_text_to_satisfy_token_limit(
                        txt=content,
                        limit=self._get_token_limit(),
                        llm_model=self.llm_kwargs['llm_model']
                    )
                    break

            except Exception as e:
                self.failed_files.append((fp, f"文本分割失败：{str(e)}"))
                mutable_status[2] = "分割失败"
                return fragments

            # 处理片段
            rel_path = os.path.relpath(fp, project_folder)
            for i, frag in enumerate(paper_fragments):
                check_timeout()  # 每处理一个片段检查一次超时
                if frag.strip():

View on GitHub (pinned to d6bde0fa54)

Solutions

1. Check whether the global file timeout also fired and treat the split stage as the symptom rather than the root cause.
2. Measure breakdown_text_to_satisfy_token_limit on the extracted text in isolation.
3. Chunk very large extracted text before splitting or use a model with a larger max_token.
4. Remove the ineffective 5-second pre-call check or add a real subprocess timeout at the desired limit.
5. Handle RuntimeError('存在一行极长的文本！') from the splitter by force-breaking the offending line.

Example fix

# before
if time.time() - split_start_time > 5:
    raise TimeoutError("文本分割超时（5秒）")
paper_fragments = breakdown_text_to_satisfy_token_limit(...)

# after
paper_fragments = breakdown_text_to_satisfy_token_limit(...)
if time.time() - split_start_time > 5:
    raise TimeoutError("文本分割超时（5秒）")

if not content or not content.strip():
    raise ValueError("Cannot split empty extracted text")
if max((len(line) for line in content.splitlines()), default=0) > 1_000_000:
    content = "\n".join(line[i:i+100000] for line in content.splitlines() for i in range(0, len(line), 100000))

def is_splitable_text(content) -> bool:
    return isinstance(content, str) and bool(content.strip()) and max(map(len, content.splitlines()), default=0) < 1_000_000

try:
    fragments = breakdown_text_to_satisfy_token_limit(...)
except TimeoutError as e:
    record_failed_file(fp, f"splitting timeout: {e}")

When it happens

Trigger: check_timeout() finds the global worker timer set, or elapsed time exceeds five seconds before another split attempt. Huge content, a very long unbroken line, repeated tokenizer calls, or an already overloaded subprocess make the stage slow enough to reach a checkpoint late.

Common situations: A model with a small max_token creates many fragments; PDF text extraction produced one enormous line; tokenizer/model configuration is invalid; high CPU contention; tests monkey or alter time; the global 30-second timeout fired while splitting.

Related errors

• 处理文件 {os.path.basename(fp)} 超时（{TIMEOUT_SECONDS}秒）
• 文件内容提取超时（10秒）
• 等待用户输入超时
• 存在一行极长的文本！{remain_txt_to_cut}
• 存在一行极长的文本！{remain_txt_to_cut}

AI-assisted analysis of binary-husky/gpt_academic@d6bde0fa54 (2026-08-14). Data as JSON: /api/errors/f7bd31b5e9d2e585. Report an issue: GitHub.