hankcs/HanLP · error · ValueError

Negative offset occurred, maybe doc_level_offset=False

Error message

Negative offset occurred, maybe doc_level_offset=False

What it means

When loading CoNLL-2012 SRL data, offsets can be document-level or sentence-level. If doc_level_offset=True the loader subtracts num_tokens_in_doc; if the resulting predicate/argument offsets go negative, the offsets were actually sentence-level (or misaligned), so it raises ValueError suggesting doc_level_offset=False.

Source

Thrown at hanlp/datasets/srl/loaders/conll2012.py:126

        Args:
            filepath: ``.jsonlines`` CoNLL12 corpus.
        """
        filename = os.path.basename(filepath)
        reader = TimingFileIterator(filepath)
        num_docs, num_sentences = 0, 0
        for line in reader:
            doc = json.loads(line)
            num_docs += 1
            num_tokens_in_doc = 0
            for sid, (sentence, srl) in enumerate(zip(doc['sentences'], doc['srl'])):
                if self.doc_level_offset:
                    srl = [(x[0] - num_tokens_in_doc, x[1] - num_tokens_in_doc, x[2] - num_tokens_in_doc, x[3]) for x in
                           srl]
                else:
                    srl = [(x[0], x[1], x[2], x[3]) for x in srl]
                for x in srl:
                    if any([o < 0 for o in x[:3]]):
                        raise ValueError(f'Negative offset occurred, maybe doc_level_offset=False')
                    if any([o >= len(sentence) for o in x[:3]]):
                        raise ValueError('Offset exceeds sentence length, maybe doc_level_offset=True')
                deduplicated_srl = set()
                pa_set = set()
                for p, b, e, l in srl:
                    pa = (p, b, e)
                    if pa in pa_set:
                        continue
                    pa_set.add(pa)
                    deduplicated_srl.add((p, b, e, l))
                yield self.build_sample(sentence, deduplicated_srl, doc, sid)
                num_sentences += 1
                num_tokens_in_doc += len(sentence)
            reader.log(
                f'{filename} {num_docs} documents, {num_sentences} sentences [blink][yellow]...[/yellow][/blink]')
        reader.erase()

    # noinspection PyMethodMayBeStatic

View on GitHub (pinned to ddb1299bdd)

Solutions

Re-load with doc_level_offset=False if your offsets are sentence-relative
Re-download or regenerate the _conll files from the standard make_gold_conll pipeline so offsets match the expected convention
Inspect one file: if offsets restart near 0 each sentence, they are sentence-level → use False

Example fix

# before
docs = load_file(path, doc_level_offset=True)
# after
docs = load_file(path, doc_level_offset=False)

Defensive patterns

Strategy: validation

Validate before calling

def detect_offset_level(path):
    # peek: sentence-level offsets reset near 0 each sentence
    prev_max, saw_reset = 0, False
    for line in open(path):
        parts = line.split()
        if len(parts) > 1 and parts[0] == '#':
            prev_max = 0
        # heuristic left to caller; fallback: try doc_level_offset=False on a sample
import itertools
def offsets_look_sentence_level(load_file, path):
    try:
        load_file(path, doc_level_offset=True, max_samples=10)
        return False  # doc-level worked
    except ValueError:
        return True
# prefer simple probing:
try:
    docs = load_file(path, doc_level_offset=False, n=5)
except ValueError:
    pass  # not sentence-level

Try / catch

try:
    docs = load_file(path, doc_level_offset=False)
except ValueError as e:
    if 'doc_level_offset=True' in str(e):
        docs = load_file(path, doc_level_offset=True)
    else:
        raise

Prevention

Probe a handful of samples with each flag before loading the full corpus
Keep a record of which pipeline produced your _conll files and its offset convention

When it happens

Trigger: Calling load_file with doc_level_offset=True on files whose SRL offsets are sentence-relative (common in pre-split or re-generated _conll files), producing negative indices after subtraction.

Common situations: Re-processing OntoNotes with different sentence segmentation than the original; mixing files from different preprocessing pipelines; guessing the doc_level_offset flag.

Related errors

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/015b7a0d11f87351. Report an issue: GitHub.