huggingface/tokenizers · error

Cannot build Piece from string

Error message

Cannot build Piece from string "{s}"

What it means

This error comes from `TryFrom<String> for TemplateProcessing`'s `Piece` in the tokenizers library. A `Piece` in a post-processor template must be a `<id>` like `$A`, `$B`, or a special token like `[CLS]`, optionally with `:<type_id>`. When the `id:type_id` form is given, the part after the colon must parse as a `u32`; if it does not, the library rejects the whole string with `Cannot build Piece from string "{s}"`.

Solutions

1. Make the type id a plain non-negative integer with no spaces or quotes inside the segment: `$A:0 $B:1`.
2. If you intended only a sequence id, drop the colon entirely (`$A`) instead of supplying a non-numeric type id.
3. Validate the template before feeding it to `TemplateProcessing::new`/config loading by splitting on whitespace and `:` and checking `u32::from_str` on the second part.
4. Check that the template values correspond to declared `single`/`pair` sequence ids (`A`, `B`) and special tokens in your tokenizer.

Example fix

// before
TemplateProcessing::builder().single("$A:first $B:1")

// after
TemplateProcessing::builder().single("$A:0 $B:1")

import re
PIECE_RE = re.compile(r"^(\$[A-Za-z]|\[[^\]]+\])(?::(\d+))?$")
def valid_piece(piece: str) -> bool:
    return bool(PIECE_RE.match(piece))

try:
    proc = TemplateProcessing.builder().single(template).build()
except Exception as e:
    raise ValueError(f"bad template piece in {template!r}: {e}") from e

When it happens

Trigger: Calling `Piece::try_from` (directly or while building a `TemplateProcessing` template string like `"$A:0 $B:1"`) where the segment after `:` is not a valid unsigned integer, e.g. `"$A:first"`, `"$B:-1"`, `"$A:1.5"`, or `"$A: "`.

Common situations: Common when writing post-processor JSON configs by hand — typos in the type id, using a 1-based index when 0-based is expected plus a non-numeric value, pasting templates from other tokenizers (e.g. sentencepiece style), or quoting/whitespace sneaking into the type id.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• NormalizedString bad split
• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/400457eb38b10c61. Report an issue: GitHub.

Appendix: source

Thrown at tokenizers/src/processors/template.rs:156

    fn with_type_id(self, type_id: u32) -> Self {
        match self {
            Self::Sequence { id, .. } => Self::Sequence { id, type_id },
            Self::SpecialToken { id, .. } => Self::SpecialToken { id, type_id },
        }
    }
}

impl TryFrom<String> for Piece {
    type Error = String;

    fn try_from(s: String) -> StdResult<Self, Self::Error> {
        let parts = s.split(':').collect::<Vec<_>>();

        let err = || format!("Cannot build Piece from string \"{s}\"");
        match parts.as_slice() {
            [id, type_id] => {
                let type_id: u32 = type_id.parse().map_err(|_| err())?;
                let piece = Self::extract_id(id).ok_or_else(err)?;
                Ok(piece.with_type_id(type_id))
            }
            [id] => Self::extract_id(id).ok_or_else(err),
            _ => Err(err()),
        }
    }
}

impl TryFrom<&str> for Piece {
    type Error = String;

    fn try_from(s: &str) -> StdResult<Self, Self::Error> {
        Piece::try_from(s.to_owned())
    }
}

/// Represents a bunch of tokens to be used in a template.

View on GitHub (pinned to 6cfd9d385c)