{"record":{"id":"015b7a0d11f87351","repo":"hankcs/HanLP","slug":"negative-offset-occurred-maybe-doc-level-offset-f","errorCode":null,"errorMessage":"Negative offset occurred, maybe doc_level_offset=False","messagePattern":"Negative offset occurred, maybe doc_level_offset=False","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"hanlp/datasets/srl/loaders/conll2012.py","lineNumber":126,"sourceCode":"        Args:\n            filepath: ``.jsonlines`` CoNLL12 corpus.\n        \"\"\"\n        filename = os.path.basename(filepath)\n        reader = TimingFileIterator(filepath)\n        num_docs, num_sentences = 0, 0\n        for line in reader:\n            doc = json.loads(line)\n            num_docs += 1\n            num_tokens_in_doc = 0\n            for sid, (sentence, srl) in enumerate(zip(doc['sentences'], doc['srl'])):\n                if self.doc_level_offset:\n                    srl = [(x[0] - num_tokens_in_doc, x[1] - num_tokens_in_doc, x[2] - num_tokens_in_doc, x[3]) for x in\n                           srl]\n                else:\n                    srl = [(x[0], x[1], x[2], x[3]) for x in srl]\n                for x in srl:\n                    if any([o < 0 for o in x[:3]]):\n                        raise ValueError(f'Negative offset occurred, maybe doc_level_offset=False')\n                    if any([o >= len(sentence) for o in x[:3]]):\n                        raise ValueError('Offset exceeds sentence length, maybe doc_level_offset=True')\n                deduplicated_srl = set()\n                pa_set = set()\n                for p, b, e, l in srl:\n                    pa = (p, b, e)\n                    if pa in pa_set:\n                        continue\n                    pa_set.add(pa)\n                    deduplicated_srl.add((p, b, e, l))\n                yield self.build_sample(sentence, deduplicated_srl, doc, sid)\n                num_sentences += 1\n                num_tokens_in_doc += len(sentence)\n            reader.log(\n                f'{filename} {num_docs} documents, {num_sentences} sentences [blink][yellow]...[/yellow][/blink]')\n        reader.erase()\n\n    # noinspection PyMethodMayBeStatic","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/datasets/srl/loaders/conll2012.py#L108-L144","documentation":"When loading CoNLL-2012 SRL data, offsets can be document-level or sentence-level. If doc_level_offset=True the loader subtracts num_tokens_in_doc; if the resulting predicate/argument offsets go negative, the offsets were actually sentence-level (or misaligned), so it raises ValueError suggesting doc_level_offset=False.","triggerScenarios":"Calling load_file with doc_level_offset=True on files whose SRL offsets are sentence-relative (common in pre-split or re-generated _conll files), producing negative indices after subtraction.","commonSituations":"Re-processing OntoNotes with different sentence segmentation than the original; mixing files from different preprocessing pipelines; guessing the doc_level_offset flag.","solutions":["Re-load with doc_level_offset=False if your offsets are sentence-relative","Re-download or regenerate the _conll files from the standard make_gold_conll pipeline so offsets match the expected convention","Inspect one file: if offsets restart near 0 each sentence, they are sentence-level → use False"],"exampleFix":"# before\ndocs = load_file(path, doc_level_offset=True)\n# after\ndocs = load_file(path, doc_level_offset=False)","handlingStrategy":"validation","validationCode":"def detect_offset_level(path):\n    # peek: sentence-level offsets reset near 0 each sentence\n    prev_max, saw_reset = 0, False\n    for line in open(path):\n        parts = line.split()\n        if len(parts) > 1 and parts[0] == '#':\n            prev_max = 0\n        # heuristic left to caller; fallback: try doc_level_offset=False on a sample\nimport itertools\ndef offsets_look_sentence_level(load_file, path):\n    try:\n        load_file(path, doc_level_offset=True, max_samples=10)\n        return False  # doc-level worked\n    except ValueError:\n        return True\n# prefer simple probing:\ntry:\n    docs = load_file(path, doc_level_offset=False, n=5)\nexcept ValueError:\n    pass  # not sentence-level","typeGuard":null,"tryCatchPattern":"try:\n    docs = load_file(path, doc_level_offset=False)\nexcept ValueError as e:\n    if 'doc_level_offset=True' in str(e):\n        docs = load_file(path, doc_level_offset=True)\n    else:\n        raise","preventionTips":["Probe a handful of samples with each flag before loading the full corpus","Keep a record of which pipeline produced your _conll files and its offset convention"],"tags":["python","srl","ontonotes","conll2012","data-loading"],"backgroundTag":"offset-misalignment","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}