{"record":{"id":"f98079465e5dfbd0","repo":"lancedb/lancedb","slug":"pack-sequences-requires-a-list-typed-token-column","errorCode":null,"errorMessage":"pack_sequences requires a list-typed token column; {columns[0]} has type {field.type}","messagePattern":"pack_sequences requires a list-typed token column; (.+?) has type (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/python/lancedb/streaming.py","lineNumber":471,"sourceCode":"                if blocks_per_epoch % num_splits != 0:\n                    raise ValueError(\n                        f\"blocks_per_epoch ({blocks_per_epoch}) must be divisible by \"\n                        f\"num_splits ({num_splits})\"\n                    )\n            if transform is not None:\n                raise ValueError(\"transform cannot be combined with pack_sequences\")\n            if columns is None or len(columns) != 1:\n                raise ValueError(\n                    \"pack_sequences requires columns to name exactly one \"\n                    \"list-typed column of token ids\"\n                )\n            field = table.schema.field(columns[0])\n            if not (\n                pa.types.is_list(field.type)\n                or pa.types.is_large_list(field.type)\n                or pa.types.is_fixed_size_list(field.type)\n            ):\n                raise ValueError(\n                    f\"pack_sequences requires a list-typed token column; \"\n                    f\"{columns[0]} has type {field.type}\"\n                )\n            if not pa.types.is_integer(field.type.value_type):\n                raise ValueError(\n                    \"pack_sequences requires a token column with integer values; \"\n                    f\"{columns[0]} has value type {field.type.value_type}\"\n                )\n        elif blocks_per_epoch is not None:\n            raise ValueError(\"blocks_per_epoch requires pack_sequences\")\n        if on_transform_error not in (\"raise\", \"skip\", \"warn\") and not callable(\n            on_transform_error\n        ):\n            raise ValueError(\n                \"on_transform_error must be 'raise', 'skip', 'warn', or a \"\n                f\"callable, got {on_transform_error!r}\"\n            )\n        if transform_queue_depth is not None and transform_queue_depth <= 0:","sourceCodeStart":453,"sourceCodeEnd":489,"githubUrl":"https://github.com/lancedb/lancedb/blob/c7b051aff7039333a3f61b79217246c27676806a/python/python/lancedb/streaming.py#L453-L489","documentation":"After resolving the single column with `columns[0]`, the constructor checks the table schema: the column must be a list, large_list, or fixed_size_list of tokens. A scalar or struct column cannot be packed into blocks of sequences, so a ValueError with the actual Arrow type is raised.","triggerScenarios":"DataLoader with `pack_sequences=True, columns=['text']` where `text` is a string/struct/scalar column rather than a list column; pointing at a column that was flattened or converted to binary at write time.","commonSituations":"Column-name typo resolving to a different typed column; schema changed between dataset versions (column migrated from list<int> to string); selecting an embedding (fixed_size_list of float) thinking it is token ids.","solutions":["Check `table.schema.field('<name>').type` and pass the column that is actually a list of integers.","Re-encode the data so the target column is list-typed (e.g. list<int64> of token ids) before packing.","Fix any typo in `columns[0]` so it refers to the intended token column."],"exampleFix":"// before\nloader = DataLoader(table, pack_sequences=True, columns=['text'])  # string column\n// after\nloader = DataLoader(table, pack_sequences=True, columns=['input_ids'])  # list<int64>","handlingStrategy":"validation","validationCode":"import pyarrow as pa\nf = table.schema.field('tokens')\nassert f is not None and (pa.types.is_list(f.type) or pa.types.is_large_list(f.type) or pa.types.is_fixed_size_list(f.type)), f'tokens type: {f.type}'","typeGuard":"def is_list_column(field):\n    return field is not None and any(check(field.type) for check in (pa.types.is_list, pa.types.is_large_list, pa.types.is_fixed_size_list))","tryCatchPattern":"try:\n    loader = DataLoader(table, pack_sequences=True, columns=['tokens'])\nexcept ValueError as e:\n    logger.error('pack_sequences column invalid: %s', e)\n    raise","preventionTips":["Print table.schema before configuring loaders.","Fix the token column name and type at dataset write time."],"tags":["python","data-loader","schema","arrow","validation"],"backgroundTag":"schema-validation-failed","analyzedSha":"c7b051aff7039333a3f61b79217246c27676806a","analyzedAt":"2026-09-08T23:42:37.579Z","contentChangedAt":"2026-09-08T23:42:37.579Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}