hankcs/HanLP · error · ValueError

Offset exceeds sentence length, maybe doc_level_offset=True

Error message

Offset exceeds sentence length, maybe doc_level_offset=True

What it means

The sibling check to the negative-offset case: when SRL offsets (treated as sentence-level, i.e. doc_level_offset=False) are greater than or equal to the sentence length, they exceed the sentence boundary, which indicates they were document-level offsets. The loader raises ValueError suggesting doc_level_offset=True so the doc-token prefix is subtracted.

Source

Thrown at hanlp/datasets/srl/loaders/conll2012.py:128

        """
        filename = os.path.basename(filepath)
        reader = TimingFileIterator(filepath)
        num_docs, num_sentences = 0, 0
        for line in reader:
            doc = json.loads(line)
            num_docs += 1
            num_tokens_in_doc = 0
            for sid, (sentence, srl) in enumerate(zip(doc['sentences'], doc['srl'])):
                if self.doc_level_offset:
                    srl = [(x[0] - num_tokens_in_doc, x[1] - num_tokens_in_doc, x[2] - num_tokens_in_doc, x[3]) for x in
                           srl]
                else:
                    srl = [(x[0], x[1], x[2], x[3]) for x in srl]
                for x in srl:
                    if any([o < 0 for o in x[:3]]):
                        raise ValueError(f'Negative offset occurred, maybe doc_level_offset=False')
                    if any([o >= len(sentence) for o in x[:3]]):
                        raise ValueError('Offset exceeds sentence length, maybe doc_level_offset=True')
                deduplicated_srl = set()
                pa_set = set()
                for p, b, e, l in srl:
                    pa = (p, b, e)
                    if pa in pa_set:
                        continue
                    pa_set.add(pa)
                    deduplicated_srl.add((p, b, e, l))
                yield self.build_sample(sentence, deduplicated_srl, doc, sid)
                num_sentences += 1
                num_tokens_in_doc += len(sentence)
            reader.log(
                f'{filename} {num_docs} documents, {num_sentences} sentences [blink][yellow]...[/yellow][/blink]')
        reader.erase()

    # noinspection PyMethodMayBeStatic
    def build_sample(self, sentence, deduplicated_srl, doc, sid):
        return {

View on GitHub (pinned to ddb1299bdd)

Solutions

  1. Re-load with doc_level_offset=True
  2. If offsets grow monotonically across sentences in a document, they are document-level → use True
  3. Ensure you use the loader variant matching how the files were produced (original OntoNotes vs HanLP-processed)

Example fix

# before
docs = load_file(path, doc_level_offset=False)
# after
docs = load_file(path, doc_level_offset=True)
Defensive patterns

Strategy: try-catch

Validate before calling

# probe a small slice with one flag; fall back to the other
try:
    sample = load_file(path, doc_level_offset=False, max_docs=5)
    level = False
except ValueError:
    level = True

Try / catch

try:
    docs = load_file(path, doc_level_offset=False)
except ValueError as e:
    if 'maybe doc_level_offset=True' in str(e):
        docs = load_file(path, doc_level_offset=True)
    else:
        raise

Prevention

When it happens

Trigger: Calling load_file with doc_level_offset=False on files whose offsets are cumulative from document start; the first sentence of each doc may pass, but later sentences have offsets >= len(sentence).

Common situations: Loading standard OntoNotes _conll files produced by the official pipeline with the wrong flag; default flag not matching the data variant being used.

Related errors

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/24c6b7d9ca15f453. Report an issue: GitHub.