{"record":{"id":"f7bd31b5e9d2e585","repo":"binary-husky/gpt_academic","slug":"5","errorCode":null,"errorMessage":"文本分割超时（5秒）","messagePattern":"文本分割超时（5秒）","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"warning","filePath":"crazy_functions/Document_Conversation.py","lineNumber":153,"sourceCode":"                self.failed_files.append((fp, \"文件内容为空\"))\n                mutable_status[2] = \"内容为空\"\n                return fragments\n\n            check_timeout()\n\n            # 更新状态\n            mutable_status[0] = \"分割文本\"\n            mutable_status[1] = time.time()\n\n            # 分割文本 - 添加超时检查\n            split_start_time = time.time()\n            try:\n                while True:\n                    check_timeout()  # 检查全局超时\n\n                    # 检查分割过程是否超时（5秒）\n                    if time.time() - split_start_time > 5:\n                        raise TimeoutError(\"文本分割超时（5秒）\")\n\n                    paper_fragments = breakdown_text_to_satisfy_token_limit(\n                        txt=content,\n                        limit=self._get_token_limit(),\n                        llm_model=self.llm_kwargs['llm_model']\n                    )\n                    break\n\n            except Exception as e:\n                self.failed_files.append((fp, f\"文本分割失败：{str(e)}\"))\n                mutable_status[2] = \"分割失败\"\n                return fragments\n\n            # 处理片段\n            rel_path = os.path.relpath(fp, project_folder)\n            for i, frag in enumerate(paper_fragments):\n                check_timeout()  # 每处理一个片段检查一次超时\n                if frag.strip():","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/binary-husky/gpt_academic/blob/d6bde0fa54373309bd05823a49bda8da019d2c77/crazy_functions/Document_Conversation.py#L135-L171","documentation":"This message represents the intended 5-second budget for breakdown_text_to_satisfy_token_limit. In this implementation the clock is checked only before a single synchronous call; a normal split that takes longer returns and breaks successfully. The error is visible only when control re-enters the loop, the global 30-second timer has already fired, or the clock changes. It is not a hard timeout for the tokenizer.","triggerScenarios":"check_timeout() finds the global worker timer set, or elapsed time exceeds five seconds before another split attempt. Huge content, a very long unbroken line, repeated tokenizer calls, or an already overloaded subprocess make the stage slow enough to reach a checkpoint late.","commonSituations":"A model with a small max_token creates many fragments; PDF text extraction produced one enormous line; tokenizer/model configuration is invalid; high CPU contention; tests monkey or alter time; the global 30-second timeout fired while splitting.","solutions":["Check whether the global file timeout also fired and treat the split stage as the symptom rather than the root cause.","Measure breakdown_text_to_satisfy_token_limit on the extracted text in isolation.","Chunk very large extracted text before splitting or use a model with a larger max_token.","Remove the ineffective 5-second pre-call check or add a real subprocess timeout at the desired limit.","Handle RuntimeError('存在一行极长的文本！') from the splitter by force-breaking the offending line."],"exampleFix":"# before\nif time.time() - split_start_time > 5:\n    raise TimeoutError(\"文本分割超时（5秒）\")\npaper_fragments = breakdown_text_to_satisfy_token_limit(...)\n\n# after\npaper_fragments = breakdown_text_to_satisfy_token_limit(...)\nif time.time() - split_start_time > 5:\n    raise TimeoutError(\"文本分割超时（5秒）\")\n","handlingStrategy":"validation","validationCode":"if not content or not content.strip():\n    raise ValueError(\"Cannot split empty extracted text\")\nif max((len(line) for line in content.splitlines()), default=0) > 1_000_000:\n    content = \"\\n\".join(line[i:i+100000] for line in content.splitlines() for i in range(0, len(line), 100000))\n","typeGuard":"def is_splitable_text(content) -> bool:\n    return isinstance(content, str) and bool(content.strip()) and max(map(len, content.splitlines()), default=0) < 1_000_000\n","tryCatchPattern":"try:\n    fragments = breakdown_text_to_satisfy_token_limit(...)\nexcept TimeoutError as e:\n    record_failed_file(fp, f\"splitting timeout: {e}\")\n","preventionTips":["Check model_info has a valid tokenizer and max_token before splitting.","Break million-character lines before calling the splitter.","Profile splitting on large documents rather than assuming the 5-second check enforces it.","Use a subprocess timeout when a hard limit is required."],"tags":["timeout","tokenizer","text-splitting","dead-code"],"backgroundTag":null,"analyzedSha":"d6bde0fa54373309bd05823a49bda8da019d2c77","analyzedAt":"2026-08-14T22:48:35.038Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}