hankcs/HanLP · error · ValueError

Failed to load {tsv_file_path}: {sent}

Error message

Failed to load {tsv_file_path}: {sent}

What it means

While reading a TSV corpus with gold tags (gold=True), the loader tried to take cells[1] from tokens that have no second column; the bare except re-raises as ValueError showing the offending sentence, so the TSV structure does not match the expected word<TAB>tag format.

Source

Thrown at hanlp/utils/io_util.py:477

        if max_seq_length:
            offset = 0
            # try to split the sequence to make it fit into max_seq_length
            for shorter_words in split_long_sentence_into(words, max_seq_length, sent_delimiter, char_level,
                                                          hard_constraint):
                if gold:
                    shorter_tags = [cells[1] for cells in sent[offset:offset + len(shorter_words)]]
                    offset += len(shorter_words)
                else:
                    shorter_tags = None
                if lower:
                    shorter_words = [word.lower() for word in shorter_words]
                yield shorter_words, shorter_tags
        else:
            if gold:
                try:
                    tags = [cells[1] for cells in sent]
                except:
                    raise ValueError(f'Failed to load {tsv_file_path}: {sent}')
            else:
                tags = None
            if lower:
                words = [word.lower() for word in words]
            yield words, tags


def split_file(filepath, train=0.8, dev=0.1, test=0.1, names=None, shuffle=False):
    num_samples = 0
    if filepath.endswith('.tsv'):
        for sent in read_tsv_as_sents(filepath):
            num_samples += 1
    else:
        with open(filepath, encoding='utf-8') as src:
            for sample in src:
                num_samples += 1
    splits = {'train': train, 'dev': dev, 'test': test}
    splits = dict((k, v) for k, v in splits.items() if v)

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Re-check the printed sentence: identify tokens lacking a tag column and fix the corpus (tab-separate word and tag).
2. For untagged prediction data, call with gold=False so no tags are parsed.
3. Normalize the file: strip trailing whitespace, remove headers, ensure every non-blank line is 'word\ttag'.

Example fix

# before
load_file('raw.tsv', gold=True)   # raw.tsv has only words
# after
load_file('raw.tsv', gold=False)

with open(tsv) as f:
    for i, line in enumerate(f):
        if line.strip():
            assert '\t' in line, f'line {i+1} missing tab'
            assert len(line.rstrip('\n').split('\t')) >= 2, f'line {i+1} lacks tag column'

def is_tagged_tsv(path):
    return all(not l.strip() or len(l.rstrip('\n').split('\t')) >= 2 for l in open(path))

try:
    load_file(p, gold=True)
except ValueError as e:
    if 'Failed to load' in str(e):
        load_file(p, gold=False)  # data is untagged
    else:
        raise

When it happens

Trigger: Calling load_file/file_to_inputs on a TSV where some line has only one column (untagged corpus fed with gold=True), uses spaces instead of tabs, or has a header/malformed row.

Common situations: Forgetting to pass gold=False for inference data; corpora exported with ' ' delimiter; stray blank-with-space lines or a header row inside sentences.

Related errors

• Unknown data arrangement
• the first two dimensions of emissions and tags must match, g
• Unrecognized label encoding {self.label_encoding}
• Expect X to be 2 or 3 elements but got {repr(X)}
• {} contains None or zero-length word {}

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/3e517d801011da34. Report an issue: GitHub.