{"record":{"id":"24c6b7d9ca15f453","repo":"hankcs/HanLP","slug":"offset-exceeds-sentence-length-maybe-doc-level-of","errorCode":null,"errorMessage":"Offset exceeds sentence length, maybe doc_level_offset=True","messagePattern":"Offset exceeds sentence length, maybe doc_level_offset=True","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"hanlp/datasets/srl/loaders/conll2012.py","lineNumber":128,"sourceCode":"        \"\"\"\n        filename = os.path.basename(filepath)\n        reader = TimingFileIterator(filepath)\n        num_docs, num_sentences = 0, 0\n        for line in reader:\n            doc = json.loads(line)\n            num_docs += 1\n            num_tokens_in_doc = 0\n            for sid, (sentence, srl) in enumerate(zip(doc['sentences'], doc['srl'])):\n                if self.doc_level_offset:\n                    srl = [(x[0] - num_tokens_in_doc, x[1] - num_tokens_in_doc, x[2] - num_tokens_in_doc, x[3]) for x in\n                           srl]\n                else:\n                    srl = [(x[0], x[1], x[2], x[3]) for x in srl]\n                for x in srl:\n                    if any([o < 0 for o in x[:3]]):\n                        raise ValueError(f'Negative offset occurred, maybe doc_level_offset=False')\n                    if any([o >= len(sentence) for o in x[:3]]):\n                        raise ValueError('Offset exceeds sentence length, maybe doc_level_offset=True')\n                deduplicated_srl = set()\n                pa_set = set()\n                for p, b, e, l in srl:\n                    pa = (p, b, e)\n                    if pa in pa_set:\n                        continue\n                    pa_set.add(pa)\n                    deduplicated_srl.add((p, b, e, l))\n                yield self.build_sample(sentence, deduplicated_srl, doc, sid)\n                num_sentences += 1\n                num_tokens_in_doc += len(sentence)\n            reader.log(\n                f'{filename} {num_docs} documents, {num_sentences} sentences [blink][yellow]...[/yellow][/blink]')\n        reader.erase()\n\n    # noinspection PyMethodMayBeStatic\n    def build_sample(self, sentence, deduplicated_srl, doc, sid):\n        return {","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/datasets/srl/loaders/conll2012.py#L110-L146","documentation":"The sibling check to the negative-offset case: when SRL offsets (treated as sentence-level, i.e. doc_level_offset=False) are greater than or equal to the sentence length, they exceed the sentence boundary, which indicates they were document-level offsets. The loader raises ValueError suggesting doc_level_offset=True so the doc-token prefix is subtracted.","triggerScenarios":"Calling load_file with doc_level_offset=False on files whose offsets are cumulative from document start; the first sentence of each doc may pass, but later sentences have offsets >= len(sentence).","commonSituations":"Loading standard OntoNotes _conll files produced by the official pipeline with the wrong flag; default flag not matching the data variant being used.","solutions":["Re-load with doc_level_offset=True","If offsets grow monotonically across sentences in a document, they are document-level → use True","Ensure you use the loader variant matching how the files were produced (original OntoNotes vs HanLP-processed)"],"exampleFix":"# before\ndocs = load_file(path, doc_level_offset=False)\n# after\ndocs = load_file(path, doc_level_offset=True)","handlingStrategy":"try-catch","validationCode":"# probe a small slice with one flag; fall back to the other\ntry:\n    sample = load_file(path, doc_level_offset=False, max_docs=5)\n    level = False\nexcept ValueError:\n    level = True","typeGuard":null,"tryCatchPattern":"try:\n    docs = load_file(path, doc_level_offset=False)\nexcept ValueError as e:\n    if 'maybe doc_level_offset=True' in str(e):\n        docs = load_file(path, doc_level_offset=True)\n    else:\n        raise","preventionTips":["Auto-detect offset level on a sample before full loads"],"tags":["python","srl","ontonotes","conll2012","data-loading"],"backgroundTag":"offset-misalignment","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}