xai-org/x-algorithm · error · ValueError

All dumps are None

Error message

All dumps are None

What it means

layer_stack_block gathers per-layer dumps across a stacked/unstacked layer loop. For one dump slot across layers it replaces None entries with zeros_like of a non-None dump; if every layer produced None for that slot, there is no template to copy and it raises ValueError('All dumps are None').

Source

Thrown at phoenix/xrex/models/transformer.py:572

        segment_ids_k=segment_ids_k,
        name=f"{name_prefix}_0",
        global_layer_index=global_layer_index + layer_index_offset,
        seqpack_layout=seqpack_layout,
    )
    h = d.output
    if debug_tensor_dump_output_folder is not None:
        layer_dumps.append(d.layer_dumps)

    if debug_tensor_dump_output_folder is not None:
        num_elements = len(layer_dumps[0])

        concatenated_dumps = []
        for i in range(num_elements):
            dumps = [dump[i] for dump in layer_dumps]
            if any(dump is None for dump in dumps):
                non_none_dump = next((dump for dump in dumps if dump is not None), None)
                if non_none_dump is None:
                    raise ValueError("All dumps are None")
                dummy_tensor = jnp.zeros_like(non_none_dump)
                dumps = [dump if dump is not None else dummy_tensor for dump in dumps]
            concatenated_dumps.append(jnp.stack(dumps, axis=0))
        layer_dumps = tuple(concatenated_dumps)

    return h, DecoderOutput(
        output=jnp.zeros(()),
        layer_dumps=layer_dumps,
    )


@dataclass
class Transformer(hk.Module):
    config: TransformerConfig
    sharding_context: ShardingContext
    name: Optional[str] = None

    summarizer_prefix: str = ""

View on GitHub (pinned to 24c60942c5)

Solutions

1. Ensure at least one layer in the stack returns a real (non-None) dump for each slot, e.g. re-enable dumping in at least one layer
2. Skip concatenation for slots that are all-None instead of stacking (patch layer_stack_block to append None)

Example fix

# before
dumps = [dump[i] for dump in layer_dumps]
# ...raises if all None
# after
if all(dump is None for dump in dumps):
    concatenated_dumps.append(None)
    continue

assert any(dump[i] is not None for dump in layer_dumps), f'all dumps None at slot {i}'

def has_any_real_dump(dumps) -> bool:
    return any(d is not None for d in dumps)

try:
    out = layer_stack_block(...)
except ValueError as e:
    if 'All dumps are None' in str(e):
        # rerun with dumping enabled in at least one layer
        ...

When it happens

Trigger: Running with debug/dump collection enabled (or model variant where layers return no dumps) such that every layer's dump tuple has None at position i for some i, e.g. all layers returning None for an activation dump that the code still tries to stack.

Common situations: Disabling intermediate dumping in all layers but leaving dump-collection logic active; a model refactor where layers stopped returning dumps; mixed layer types where none implement dumps.

Related errors

• async_emb axis {axis!r} is not a mesh axis of {mesh}
• async_emb requires token shards to vary across the communica
• async_emb requires exactly one token shard per communicator
• async_emb tokens_per_batch={tokens_per_batch} does not shard
• async_emb emb_width={emb_width} does not shard evenly over t

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/63418bf4f5f407ae. Report an issue: GitHub.