lancedb/lancedb · error · ValueError

pack_sequences requires a list-typed token column;

Error message

pack_sequences requires a list-typed token column; {columns[0]} has type {field.type}

What it means

After resolving the single column with `columns[0]`, the constructor checks the table schema: the column must be a list, large_list, or fixed_size_list of tokens. A scalar or struct column cannot be packed into blocks of sequences, so a ValueError with the actual Arrow type is raised.

Solutions

1. Check `table.schema.field('<name>').type` and pass the column that is actually a list of integers.
2. Re-encode the data so the target column is list-typed (e.g. list<int64> of token ids) before packing.
3. Fix any typo in `columns[0]` so it refers to the intended token column.

Example fix

// before
loader = DataLoader(table, pack_sequences=True, columns=['text'])  # string column
// after
loader = DataLoader(table, pack_sequences=True, columns=['input_ids'])  # list<int64>

import pyarrow as pa
f = table.schema.field('tokens')
assert f is not None and (pa.types.is_list(f.type) or pa.types.is_large_list(f.type) or pa.types.is_fixed_size_list(f.type)), f'tokens type: {f.type}'

def is_list_column(field):
    return field is not None and any(check(field.type) for check in (pa.types.is_list, pa.types.is_large_list, pa.types.is_fixed_size_list))

try:
    loader = DataLoader(table, pack_sequences=True, columns=['tokens'])
except ValueError as e:
    logger.error('pack_sequences column invalid: %s', e)
    raise

When it happens

Trigger: DataLoader with `pack_sequences=True, columns=['text']` where `text` is a string/struct/scalar column rather than a list column; pointing at a column that was flattened or converted to binary at write time.

Common situations: Column-name typo resolving to a different typed column; schema changed between dataset versions (column migrated from list<int> to string); selecting an embedding (fixed_size_list of float) thinking it is token ids.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• pack_sequences requires a token column with integer values;
• blocks_per_epoch='auto' cannot estimate an empty dataset
• blocks_per_epoch requires pack_sequences
• Cannot create table from empty list without a schema
• Expected a Dictionary type to have an `dictionary` property

AI-assisted analysis of lancedb/lancedb@c7b051aff7 (2026-09-08). Data as JSON: /api/errors/f98079465e5dfbd0. Report an issue: GitHub.

Appendix: source

Thrown at python/python/lancedb/streaming.py:471

                if blocks_per_epoch % num_splits != 0:
                    raise ValueError(
                        f"blocks_per_epoch ({blocks_per_epoch}) must be divisible by "
                        f"num_splits ({num_splits})"
                    )
            if transform is not None:
                raise ValueError("transform cannot be combined with pack_sequences")
            if columns is None or len(columns) != 1:
                raise ValueError(
                    "pack_sequences requires columns to name exactly one "
                    "list-typed column of token ids"
                )
            field = table.schema.field(columns[0])
            if not (
                pa.types.is_list(field.type)
                or pa.types.is_large_list(field.type)
                or pa.types.is_fixed_size_list(field.type)
            ):
                raise ValueError(
                    f"pack_sequences requires a list-typed token column; "
                    f"{columns[0]} has type {field.type}"
                )
            if not pa.types.is_integer(field.type.value_type):
                raise ValueError(
                    "pack_sequences requires a token column with integer values; "
                    f"{columns[0]} has value type {field.type.value_type}"
                )
        elif blocks_per_epoch is not None:
            raise ValueError("blocks_per_epoch requires pack_sequences")
        if on_transform_error not in ("raise", "skip", "warn") and not callable(
            on_transform_error
        ):
            raise ValueError(
                "on_transform_error must be 'raise', 'skip', 'warn', or a "
                f"callable, got {on_transform_error!r}"
            )
        if transform_queue_depth is not None and transform_queue_depth <= 0:

View on GitHub (pinned to c7b051aff7)