hankcs/HanLP · error · ValueError

the first two dimensions of emissions and mask must match, g

Error message

the first two dimensions of emissions and mask must match, got {tuple(emissions.shape[:2])} and {tuple(mask.shape)}

What it means

Raised by HanLP's TorchCRF._validate during forward/decode when mask shape does not equal emissions.shape[:2]. The mask marks valid timesteps per sequence and must have exactly the same (seq, batch) or (batch, seq) dimensions as emissions. This matches torchcrf's contract so masking in Viterbi/score computations is valid.

Source

Thrown at hanlp/layers/crf/crf.py:180

            emissions: torch.Tensor,
            tags: Optional[torch.LongTensor] = None,
            mask: Optional[torch.ByteTensor] = None) -> None:
        if emissions.dim() != 3:
            raise ValueError(f'emissions must have dimension of 3, got {emissions.dim()}')
        if emissions.size(2) != self.num_tags:
            raise ValueError(
                f'expected last dimension of emissions is {self.num_tags}, '
                f'got {emissions.size(2)}')

        if tags is not None:
            if emissions.shape[:2] != tags.shape:
                raise ValueError(
                    'the first two dimensions of emissions and tags must match, '
                    f'got {tuple(emissions.shape[:2])} and {tuple(tags.shape)}')

        if mask is not None:
            if emissions.shape[:2] != mask.shape:
                raise ValueError(
                    'the first two dimensions of emissions and mask must match, '
                    f'got {tuple(emissions.shape[:2])} and {tuple(mask.shape)}')
            no_empty_seq = not self.batch_first and mask[0].all()
            no_empty_seq_bf = self.batch_first and mask[:, 0].all()
            if not no_empty_seq and not no_empty_seq_bf:
                raise ValueError('mask of the first timestep must all be on')

    def _compute_score(
            self, emissions: torch.Tensor, tags: torch.LongTensor,
            mask: torch.ByteTensor) -> torch.Tensor:
        # emissions: (seq_length, batch_size, num_tags)
        # tags: (seq_length, batch_size)
        # mask: (seq_length, batch_size)
        assert emissions.dim() == 3 and tags.dim() == 2
        assert emissions.shape[:2] == tags.shape
        assert emissions.size(2) == self.num_tags
        assert mask.shape == tags.shape
        assert mask[0].all()

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Generate mask from the same lengths used for padding: mask = torch.arange(T)[None,:] < lengths[:,None] (batch_first) sized to emissions.shape[:2]
2. Confirm the CRF batch_first flag matches the layout of emissions and mask
3. Verify your padding collator produces emissions and mask from identical metadata

Example fix

# before
mask = torch.ones(tags.shape, dtype=torch.uint8)  # wrong shape
# after
mask = (torch.arange(emissions.shape[1])[None, :] < lengths[:, None]).to(emissions.device)
# mask.shape == emissions.shape[:2] when batch_first=True

assert tuple(emissions.shape[:2]) == tuple(mask.shape), (emissions.shape, mask.shape)

def mask_ok(emissions: torch.Tensor, mask: torch.Tensor) -> bool:
    return tuple(emissions.shape[:2]) == tuple(mask.shape)

try:
    out = crf.decode(emissions, mask=mask)
except ValueError as e:
    if 'emissions and mask' in str(e):
        mask = build_mask(lengths, emissions.shape[:2]); out = crf.decode(emissions, mask=mask)
    else:
        raise

When it happens

Trigger: Passing a mask of different length/batch size than emissions, e.g. mask built from tags lengths while emissions were padded to another length, or a transposed mask when batch_first is False.

Common situations: Building mask from lengths with a different max_len than the padded emissions; batch_first layout mismatch between mask and CRF config; reusing a mask from a previous batch with different padding.

Related errors

• the first two dimensions of emissions and tags must match, g
• mask of the first timestep must all be on
• The `{mask_name}` should be specified for {len(self.layers)}
• Unsupported argument type: {item}
• Attention weights should be of size {(bsz * self.num_heads,

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/760022d5b45cb58c. Report an issue: GitHub.