{"record":{"id":"50401bc0e4f706f1","repo":"lancedb/lancedb","slug":"pack-sequences-does-not-support-null-token-lists-o","errorCode":null,"errorMessage":"pack_sequences does not support null token lists or values","messagePattern":"pack_sequences does not support null token lists or values","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/python/lancedb/streaming.py","lineNumber":785,"sourceCode":"        # Permutation position each split has consumed through (absolute,\n        # i.e. counted from the start of the unskipped split).  Runs ahead of\n        # initial + local_consumed when rows are skipped.\n        pos_consumed = list(initial_positions)\n\n        batch_size = self._read_batch_size\n        io_queue_depth = self._io_queue_depth\n        transform_workers = (\n            self._transform_parallelism\n            if self._transform_parallelism is not None\n            else (os.cpu_count() or 1)\n        )\n        final_transform: Callable[[pa.RecordBatch], Any]\n        if self._pack_sequences is not None:\n            # Packing consumes raw token lists, one per document.\n            def arrow_tokens(batch: pa.RecordBatch) -> list[list[int]]:\n                token_column = batch.column(0)\n                if token_column.null_count or token_column.flatten().null_count:\n                    raise ValueError(\n                        \"pack_sequences does not support null token lists or values\"\n                    )\n                return cast(list[list[int]], token_column.to_pylist())\n\n            final_transform = arrow_tokens\n        else:\n            final_transform = (\n                self._transform\n                if self._transform is not None\n                else Transforms.arrow2python\n            )\n        # None means no limit; otherwise cap rows per split to\n        # transform_queue_depth batches worth (including in-flight transforms).\n        max_cooked_rows = (\n            self._transform_queue_depth * batch_size\n            if self._transform_queue_depth is not None\n            else None\n        )","sourceCodeStart":767,"sourceCodeEnd":803,"githubUrl":"https://github.com/lancedb/lancedb/blob/c7b051aff7039333a3f61b79217246c27676806a/python/python/lancedb/streaming.py#L767-L803","documentation":"When pack_sequences is configured, the dataset treats each batch's first column as a list of token lists and packs them into fixed-size blocks. Nulls anywhere in that column (a null list entry or a null token inside a list) would corrupt packing arithmetic, so arrow_tokens raises ValueError instead. The library requires fully materialized token lists for sequence packing.","triggerScenarios":"Querying a column whose token-list entries are null (e.g. an empty document was stored as NULL, or a variable-length list column with null elements) while pack_sequences is set on the StreamingDataset.","commonSituations":"Preprocessed corpora where documents that failed tokenization were stored as null instead of an empty list; schema evolution left null list values; joining token data where missing rows produce nulls.","solutions":["Filter out rows with null token lists before iteration (e.g. filter the underlying dataset on the token column not being null).","Backfill nulls with empty token lists during preprocessing (fill_null([])) so packing sees valid lists.","Fix the upstream tokenization step to always emit a list (possibly empty) rather than null.","If nulls are acceptable, disable pack_sequences or drop null rows inside an earlier transform that raises for bad rows with on_transform_error='skip'."],"exampleFix":"# before\nds = dataset.to_streaming_dataset(pack_sequences=(eos_id, block_size))\n# after\nds = dataset.filter(~ds[\"tokens\"].is_null()).to_streaming_dataset(pack_sequences=(eos_id, block_size))","handlingStrategy":"validation","validationCode":"import pyarrow as pa\ncol = batch.column(0)\nassert col.null_count == 0 and col.flatten().null_count == 0, 'nulls in token column'\n# or upfront:\n# ds = ds.filter(~ds['tokens'].is_null())","typeGuard":"def has_null_tokens(batch: pa.RecordBatch) -> bool:\n    col = batch.column(0)\n    return col.null_count > 0 or col.flatten().null_count > 0","tryCatchPattern":"try:\n    for packed in dataset_iter:\n        ...\nexcept ValueError as e:\n    if 'pack_sequences does not support null' in str(e):\n        dataset = dataset.filter(~dataset['tokens'].is_null())\n        # rebuild and retry","preventionTips":["Backfill null token lists with empty lists at preprocessing time","Filter null rows before enabling pack_sequences","Make tokenizers always emit a list, never null"],"tags":["python","streaming","null-values","sequence-packing"],"backgroundTag":"schema-validation-failed","analyzedSha":"c7b051aff7039333a3f61b79217246c27676806a","analyzedAt":"2026-09-08T23:42:37.579Z","contentChangedAt":"2026-09-08T23:42:37.579Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}