{"record":{"id":"8c50bac30956313c","repo":"hankcs/HanLP","slug":"mask-of-the-first-timestep-must-all-be-on","errorCode":null,"errorMessage":"mask of the first timestep must all be on","messagePattern":"mask of the first timestep must all be on","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"hanlp/layers/crf/crf.py","lineNumber":186,"sourceCode":"            raise ValueError(\n                f'expected last dimension of emissions is {self.num_tags}, '\n                f'got {emissions.size(2)}')\n\n        if tags is not None:\n            if emissions.shape[:2] != tags.shape:\n                raise ValueError(\n                    'the first two dimensions of emissions and tags must match, '\n                    f'got {tuple(emissions.shape[:2])} and {tuple(tags.shape)}')\n\n        if mask is not None:\n            if emissions.shape[:2] != mask.shape:\n                raise ValueError(\n                    'the first two dimensions of emissions and mask must match, '\n                    f'got {tuple(emissions.shape[:2])} and {tuple(mask.shape)}')\n            no_empty_seq = not self.batch_first and mask[0].all()\n            no_empty_seq_bf = self.batch_first and mask[:, 0].all()\n            if not no_empty_seq and not no_empty_seq_bf:\n                raise ValueError('mask of the first timestep must all be on')\n\n    def _compute_score(\n            self, emissions: torch.Tensor, tags: torch.LongTensor,\n            mask: torch.ByteTensor) -> torch.Tensor:\n        # emissions: (seq_length, batch_size, num_tags)\n        # tags: (seq_length, batch_size)\n        # mask: (seq_length, batch_size)\n        assert emissions.dim() == 3 and tags.dim() == 2\n        assert emissions.shape[:2] == tags.shape\n        assert emissions.size(2) == self.num_tags\n        assert mask.shape == tags.shape\n        assert mask[0].all()\n\n        seq_length, batch_size = tags.shape\n        mask = mask.type_as(emissions)\n\n        # Start transition score and first emission\n        # shape: (batch_size,)","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/layers/crf/crf.py#L168-L204","documentation":"HanLP's TorchCRF requires that the first timestep of every sequence is valid (mask all-on at timestep 0), because CRF scoring assumes each sequence starts at the first emission. _validate checks mask[0].all() (or mask[:,0].all() for batch_first) and rejects masks whose first step has any zeros. This is inherited from torchcrf's semantics where empty prefixes are not representable.","triggerScenarios":"Passing a mask where any sequence has mask[..., 0] == 0, e.g. mask built with an off-by-one roll, sorted-by-length batches misaligned, or a mask that marks pad positions starting at index 0 for shorter sequences placed after longer ones without batch_first alignment.","commonSituations":"Left-padding sequences instead of right-padding; constructing mask from lengths with reversed or transposed axes; feeding a mask of all zeros for some sample.","solutions":["Use right-padding so each sequence's first timestep is real and mask[:,0]==1 for all batches","Double-check mask orientation matches batch_first (mask[:,0] vs mask[0])","Rebuild mask as arange(T) < length per sequence"],"exampleFix":"# before\nmask = (torch.arange(T)[None, :] >= lengths[:, None])  # inverted -> first step off\n# after\nmask = (torch.arange(T)[None, :] < lengths[:, None]).to(torch.uint8)\nassert mask[:, 0].all()","handlingStrategy":"validation","validationCode":"assert (mask[:, 0] if batch_first else mask[0]).all(), 'first timestep must be unmasked'","typeGuard":"def first_step_on(mask: torch.Tensor, batch_first: bool) -> bool:\n    return bool((mask[:, 0] if batch_first else mask[0]).all())","tryCatchPattern":null,"preventionTips":["Always right-pad variable-length sequences","Never feed all-zero masks","Sanity-check mask[:,0].sum() == batch_size in tests"],"tags":["hanlp","crf","pytorch","mask","sequence-padding"],"backgroundTag":"invalid-mask-shape-or-values","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}