lancedb/lancedb · error · ValueError

pack_sequences does not support null token lists or values

Error message

pack_sequences does not support null token lists or values

What it means

When pack_sequences is configured, the dataset treats each batch's first column as a list of token lists and packs them into fixed-size blocks. Nulls anywhere in that column (a null list entry or a null token inside a list) would corrupt packing arithmetic, so arrow_tokens raises ValueError instead. The library requires fully materialized token lists for sequence packing.

Solutions

Filter out rows with null token lists before iteration (e.g. filter the underlying dataset on the token column not being null).
Backfill nulls with empty token lists during preprocessing (fill_null([])) so packing sees valid lists.
Fix the upstream tokenization step to always emit a list (possibly empty) rather than null.
If nulls are acceptable, disable pack_sequences or drop null rows inside an earlier transform that raises for bad rows with on_transform_error='skip'.

Example fix

# before
ds = dataset.to_streaming_dataset(pack_sequences=(eos_id, block_size))
# after
ds = dataset.filter(~ds["tokens"].is_null()).to_streaming_dataset(pack_sequences=(eos_id, block_size))

Defensive patterns

Strategy: validation

Validate before calling

import pyarrow as pa
col = batch.column(0)
assert col.null_count == 0 and col.flatten().null_count == 0, 'nulls in token column'
# or upfront:
# ds = ds.filter(~ds['tokens'].is_null())

Type guard

def has_null_tokens(batch: pa.RecordBatch) -> bool:
    col = batch.column(0)
    return col.null_count > 0 or col.flatten().null_count > 0

Try / catch

try:
    for packed in dataset_iter:
        ...
except ValueError as e:
    if 'pack_sequences does not support null' in str(e):
        dataset = dataset.filter(~dataset['tokens'].is_null())
        # rebuild and retry

Prevention

Backfill null token lists with empty lists at preprocessing time
Filter null rows before enabling pack_sequences
Make tokenizers always emit a list, never null

When it happens

Trigger: Querying a column whose token-list entries are null (e.g. an empty document was stored as NULL, or a variable-length list column with null elements) while pack_sequences is set on the StreamingDataset.

Common situations: Preprocessed corpora where documents that failed tokenization were stored as null instead of an empty list; schema evolution left null list values; joining token data where missing rows produce nulls.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of lancedb/lancedb@c7b051aff7 (2026-09-08). Data as JSON: /api/errors/50401bc0e4f706f1. Report an issue: GitHub.

Appendix: source

Thrown at python/python/lancedb/streaming.py:785

        # Permutation position each split has consumed through (absolute,
        # i.e. counted from the start of the unskipped split).  Runs ahead of
        # initial + local_consumed when rows are skipped.
        pos_consumed = list(initial_positions)

        batch_size = self._read_batch_size
        io_queue_depth = self._io_queue_depth
        transform_workers = (
            self._transform_parallelism
            if self._transform_parallelism is not None
            else (os.cpu_count() or 1)
        )
        final_transform: Callable[[pa.RecordBatch], Any]
        if self._pack_sequences is not None:
            # Packing consumes raw token lists, one per document.
            def arrow_tokens(batch: pa.RecordBatch) -> list[list[int]]:
                token_column = batch.column(0)
                if token_column.null_count or token_column.flatten().null_count:
                    raise ValueError(
                        "pack_sequences does not support null token lists or values"
                    )
                return cast(list[list[int]], token_column.to_pylist())

            final_transform = arrow_tokens
        else:
            final_transform = (
                self._transform
                if self._transform is not None
                else Transforms.arrow2python
            )
        # None means no limit; otherwise cap rows per split to
        # transform_queue_depth batches worth (including in-flight transforms).
        max_cooked_rows = (
            self._transform_queue_depth * batch_size
            if self._transform_queue_depth is not None
            else None
        )

View on GitHub (pinned to c7b051aff7)