{"record":{"id":"984d9ef643555102","repo":"binary-husky/gpt_academic","slug":"remain-txt-to-cut","errorCode":null,"errorMessage":"存在一行极长的文本！{remain_txt_to_cut}","messagePattern":"存在一行极长的文本！(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"crazy_functions/pdf_fns/breakdown_pdf_txt.py","lineNumber":74,"sourceCode":"            cnt = 0\n            for cnt in reversed(range(estimated_line_cut)):\n                if must_break_at_empty_line:\n                    # 首先尝试用双空行（\\n\\n）作为切分点\n                    if lines[cnt] != \"\":\n                        continue\n                prev = \"\\n\".join(lines[:cnt])\n                post = \"\\n\".join(lines[cnt:])\n                if get_token_fn(prev) < limit:\n                    break\n\n            if cnt == 0:\n                # 如果没有找到合适的切分点\n                if break_anyway:\n                    # 是否允许暴力切分\n                    prev, post = force_breakdown(remain_txt_to_cut, limit, get_token_fn)\n                else:\n                    # 不允许直接报错\n                    raise RuntimeError(f\"存在一行极长的文本！{remain_txt_to_cut}\")\n\n            # 追加列表\n            res.append(prev); fin_len+=len(prev)\n            # 准备下一次迭代\n            remain_txt_to_cut = post\n            remain_txt_to_cut, remain_txt_to_cut_storage = maintain_storage(remain_txt_to_cut, remain_txt_to_cut_storage)\n            process = fin_len/total_len\n            logger.info(f'正在文本切分 {int(process*100)}%')\n            if len(remain_txt_to_cut.strip()) == 0:\n                break\n    return res\n\n\ndef breakdown_text_to_satisfy_token_limit_(txt, limit, llm_model=\"gpt-3.5-turbo\"):\n    \"\"\" 使用多种方式尝试切分文本，以满足 token 限制\n    \"\"\"\n    from request_llms.bridge_all import model_info\n    enc = model_info[llm_model]['tokenizer']","sourceCodeStart":56,"sourceCodeEnd":92,"githubUrl":"https://github.com/binary-husky/gpt_academic/blob/d6bde0fa54373309bd05823a49bda8da019d2c77/crazy_functions/pdf_fns/breakdown_pdf_txt.py#L56-L92","documentation":"Raised by breakdown_txt_to_token_limit (breakdown_pdf_txt.py variant) when the text splitter cannot find a split point: it walks candidate line counts cnt and if even the first line alone exceeds the token limit (cnt == 0) and break_anyway=False, it refuses to cut mid-line and raises. The message embeds the offending (very long) text fragment.","triggerScenarios":"A PDF-extracted text blob with a single line longer than the token limit — common with minified/binary-ish PDF text extraction, tables, or base64 blobs; calling the splitter with break_anyway=False (default strict mode) and a small limit.","commonSituations":"Scanned/corrupt PDFs whose text layer contains one giant string; academic papers with long unbroken URL/DOI lines; token limit configured below the length of any single line.","solutions":["Call the function with break_anyway=True to allow force_breakdown to cut mid-line at the token limit","Pre-normalize the text: replace very long unbroken runs with newlines or spaces before splitting","Increase the limit so at least one line fits under it","Check whether the PDF text extraction produced garbage (one-line blob) and fix extraction instead"],"exampleFix":"# before\nchunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=False)\n\n# after\nimport re\ntxt = re.sub(r'([^\\n]{1000})', r'\\1\\n', txt)  # hard-wrap absurdly long lines\nchunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=True)","handlingStrategy":"validation","validationCode":"def longest_line_tokens(text, get_token_fn):\n    return max((get_token_fn(l) for l in text.split('\\n')), default=0)\n\nif longest_line_tokens(txt, get_token_fn) >= limit:\n    txt = re.sub(r'([^\\n]{500})', r'\\1\\n', txt)  # wrap monster lines first","typeGuard":null,"tryCatchPattern":"try:\n    chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=False)\nexcept RuntimeError as e:\n    if '存在一行极长的文本' in str(e):\n        chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=True)\n    else:\n        raise","preventionTips":["Sanitize extracted PDF text (wrap lines > 500-1000 chars) before chunking","Expose break_anyway in user-facing config for untrusted inputs","Set limit comfortably above the longest realistic line"],"tags":["text-splitting","pdf","token-limit","long-line"],"backgroundTag":null,"analyzedSha":"d6bde0fa54373309bd05823a49bda8da019d2c77","analyzedAt":"2026-08-14T22:48:35.038Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}