{"record":{"id":"3e517d801011da34","repo":"hankcs/HanLP","slug":"failed-to-load-tsv-file-path-sent","errorCode":null,"errorMessage":"Failed to load {tsv_file_path}: {sent}","messagePattern":"Failed to load (.+?): (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"hanlp/utils/io_util.py","lineNumber":477,"sourceCode":"        if max_seq_length:\n            offset = 0\n            # try to split the sequence to make it fit into max_seq_length\n            for shorter_words in split_long_sentence_into(words, max_seq_length, sent_delimiter, char_level,\n                                                          hard_constraint):\n                if gold:\n                    shorter_tags = [cells[1] for cells in sent[offset:offset + len(shorter_words)]]\n                    offset += len(shorter_words)\n                else:\n                    shorter_tags = None\n                if lower:\n                    shorter_words = [word.lower() for word in shorter_words]\n                yield shorter_words, shorter_tags\n        else:\n            if gold:\n                try:\n                    tags = [cells[1] for cells in sent]\n                except:\n                    raise ValueError(f'Failed to load {tsv_file_path}: {sent}')\n            else:\n                tags = None\n            if lower:\n                words = [word.lower() for word in words]\n            yield words, tags\n\n\ndef split_file(filepath, train=0.8, dev=0.1, test=0.1, names=None, shuffle=False):\n    num_samples = 0\n    if filepath.endswith('.tsv'):\n        for sent in read_tsv_as_sents(filepath):\n            num_samples += 1\n    else:\n        with open(filepath, encoding='utf-8') as src:\n            for sample in src:\n                num_samples += 1\n    splits = {'train': train, 'dev': dev, 'test': test}\n    splits = dict((k, v) for k, v in splits.items() if v)","sourceCodeStart":459,"sourceCodeEnd":495,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/utils/io_util.py#L459-L495","documentation":"While reading a TSV corpus with gold tags (gold=True), the loader tried to take cells[1] from tokens that have no second column; the bare except re-raises as ValueError showing the offending sentence, so the TSV structure does not match the expected word<TAB>tag format.","triggerScenarios":"Calling load_file/file_to_inputs on a TSV where some line has only one column (untagged corpus fed with gold=True), uses spaces instead of tabs, or has a header/malformed row.","commonSituations":"Forgetting to pass gold=False for inference data; corpora exported with ' ' delimiter; stray blank-with-space lines or a header row inside sentences.","solutions":["Re-check the printed sentence: identify tokens lacking a tag column and fix the corpus (tab-separate word and tag).","For untagged prediction data, call with gold=False so no tags are parsed.","Normalize the file: strip trailing whitespace, remove headers, ensure every non-blank line is 'word\\ttag'."],"exampleFix":"# before\nload_file('raw.tsv', gold=True)   # raw.tsv has only words\n# after\nload_file('raw.tsv', gold=False)","handlingStrategy":"validation","validationCode":"with open(tsv) as f:\n    for i, line in enumerate(f):\n        if line.strip():\n            assert '\\t' in line, f'line {i+1} missing tab'\n            assert len(line.rstrip('\\n').split('\\t')) >= 2, f'line {i+1} lacks tag column'","typeGuard":"def is_tagged_tsv(path):\n    return all(not l.strip() or len(l.rstrip('\\n').split('\\t')) >= 2 for l in open(path))","tryCatchPattern":"try:\n    load_file(p, gold=True)\nexcept ValueError as e:\n    if 'Failed to load' in str(e):\n        load_file(p, gold=False)  # data is untagged\n    else:\n        raise","preventionTips":["Pass gold=False for inference data.","Standardize corpus exports as word\\ttag with blank lines between sentences."],"tags":["tsv","data-format","corpus","validation"],"backgroundTag":"malformed-tabular-data","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}