binary-husky/gpt_academic · error · RuntimeError

存在一行极长的文本！{remain_txt_to_cut}

Error message

存在一行极长的文本！{remain_txt_to_cut}

What it means

Raised by breakdown_txt_to_token_limit (breakdown_pdf_txt.py variant) when the text splitter cannot find a split point: it walks candidate line counts cnt and if even the first line alone exceeds the token limit (cnt == 0) and break_anyway=False, it refuses to cut mid-line and raises. The message embeds the offending (very long) text fragment.

Source

Thrown at crazy_functions/pdf_fns/breakdown_pdf_txt.py:74

            cnt = 0
            for cnt in reversed(range(estimated_line_cut)):
                if must_break_at_empty_line:
                    # 首先尝试用双空行（\n\n）作为切分点
                    if lines[cnt] != "":
                        continue
                prev = "\n".join(lines[:cnt])
                post = "\n".join(lines[cnt:])
                if get_token_fn(prev) < limit:
                    break

            if cnt == 0:
                # 如果没有找到合适的切分点
                if break_anyway:
                    # 是否允许暴力切分
                    prev, post = force_breakdown(remain_txt_to_cut, limit, get_token_fn)
                else:
                    # 不允许直接报错
                    raise RuntimeError(f"存在一行极长的文本！{remain_txt_to_cut}")

            # 追加列表
            res.append(prev); fin_len+=len(prev)
            # 准备下一次迭代
            remain_txt_to_cut = post
            remain_txt_to_cut, remain_txt_to_cut_storage = maintain_storage(remain_txt_to_cut, remain_txt_to_cut_storage)
            process = fin_len/total_len
            logger.info(f'正在文本切分 {int(process*100)}%')
            if len(remain_txt_to_cut.strip()) == 0:
                break
    return res


def breakdown_text_to_satisfy_token_limit_(txt, limit, llm_model="gpt-3.5-turbo"):
    """ 使用多种方式尝试切分文本，以满足 token 限制
    """
    from request_llms.bridge_all import model_info
    enc = model_info[llm_model]['tokenizer']

View on GitHub (pinned to d6bde0fa54)

Solutions

1. Call the function with break_anyway=True to allow force_breakdown to cut mid-line at the token limit
2. Pre-normalize the text: replace very long unbroken runs with newlines or spaces before splitting
3. Increase the limit so at least one line fits under it
4. Check whether the PDF text extraction produced garbage (one-line blob) and fix extraction instead

Example fix

# before
chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=False)

# after
import re
txt = re.sub(r'([^\n]{1000})', r'\1\n', txt)  # hard-wrap absurdly long lines
chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=True)

def longest_line_tokens(text, get_token_fn):
    return max((get_token_fn(l) for l in text.split('\n')), default=0)

if longest_line_tokens(txt, get_token_fn) >= limit:
    txt = re.sub(r'([^\n]{500})', r'\1\n', txt)  # wrap monster lines first

try:
    chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=False)
except RuntimeError as e:
    if '存在一行极长的文本' in str(e):
        chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=True)
    else:
        raise

When it happens

Trigger: A PDF-extracted text blob with a single line longer than the token limit — common with minified/binary-ish PDF text extraction, tables, or base64 blobs; calling the splitter with break_anyway=False (default strict mode) and a small limit.

Common situations: Scanned/corrupt PDFs whose text layer contains one giant string; academic papers with long unbroken URL/DOI lines; token limit configured below the length of any single line.

Related errors

• 存在一行极长的文本！{remain_txt_to_cut}
• 文本分割超时（5秒）
• 抱歉, 我们暂时无法解析此PDF文档: {fp}。
• 文件不存在: {path}
• 不是一个文件: {path}

AI-assisted analysis of binary-husky/gpt_academic@d6bde0fa54 (2026-08-14). Data as JSON: /api/errors/984d9ef643555102. Report an issue: GitHub.