rohitg00/ai-engineering-from-scratch · error · ValueError

max_len must be >= 3 to fit INST, RESP, and one response tok

Error message

max_len must be >= 3 to fit INST, RESP, and one response token

What it means

Error "max_len must be >= 3 to fit INST, RESP, and one response token" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/39-instruction-tuning-sft/code/main.py:51

# Tokeniser
# ---------------------------------------------------------------------------


class InstructionTokenizer:
    """Byte-level tokenizer with INST, RESP, PAD specials."""

    INST_ID = 256
    RESP_ID = 257
    PAD_ID = 258
    VOCAB = 260
    IGNORE_INDEX = -100

    def encode_pair(self, instruction: str, response: str, max_len: int) -> Tuple[List[int], int]:
        """Return (token_ids, response_start_index). Truncates to max_len if
        needed but always keeps the RESP marker plus at least one response
        token so SFT collation never produces fully-masked labels."""
        if max_len < 3:
            raise ValueError("max_len must be >= 3 to fit INST, RESP, and one response token")
        inst_bytes = list(instruction.encode("utf-8", errors="ignore"))
        resp_bytes = list(response.encode("utf-8", errors="ignore"))
        # Reserve 2 control tokens + at least 1 response byte.
        max_inst = max_len - 3
        inst_bytes = inst_bytes[:max_inst]
        ids = [self.INST_ID] + inst_bytes + [self.RESP_ID]
        resp_start = len(ids)
        ids.extend(resp_bytes[: max_len - len(ids)])
        return ids, resp_start

    def encode_prefix(self, instruction: str, max_len: int) -> List[int]:
        """Encode just the instruction prefix for generation. Always keeps the
        RESP marker so the model sees the same boundary as during training."""
        if max_len < 2:
            raise ValueError("max_len must be >= 2 to fit INST and RESP")
        inst_bytes = list(instruction.encode("utf-8", errors="ignore"))[: max_len - 2]
        ids = [self.INST_ID] + inst_bytes + [self.RESP_ID]
        return ids

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/39-instruction-tuning-sft/code/main.py:51 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/edb275dfd6ea52dc. Report an issue: GitHub.