xai-org/x-algorithm · error · ValueError

All dumps are None

Error message

All dumps are None

What it means

layer_stack_block gathers per-layer dumps across a stacked/unstacked layer loop. For one dump slot across layers it replaces None entries with zeros_like of a non-None dump; if every layer produced None for that slot, there is no template to copy and it raises ValueError('All dumps are None').

Solutions

1. Ensure at least one layer in the stack returns a real (non-None) dump for each slot, e.g. re-enable dumping in at least one layer
2. Skip concatenation for slots that are all-None instead of stacking (patch layer_stack_block to append None)

Example fix

# before
dumps = [dump[i] for dump in layer_dumps]
# ...raises if all None
# after
if all(dump is None for dump in dumps):
    concatenated_dumps.append(None)
    continue

assert any(dump[i] is not None for dump in layer_dumps), f'all dumps None at slot {i}'

def has_any_real_dump(dumps) -> bool:
    return any(d is not None for d in dumps)

try:
    out = layer_stack_block(...)
except ValueError as e:
    if 'All dumps are None' in str(e):
        # rerun with dumping enabled in at least one layer
        ...

When it happens

Trigger: Running with debug/dump collection enabled (or model variant where layers return no dumps) such that every layer's dump tuple has None at position i for some i, e.g. all layers returning None for an activation dump that the code still tries to stack.

Common situations: Disabling intermediate dumping in all layers but leaving dump-collection logic active; a model refactor where layers stopped returning dumps; mixed layer types where none implement dumps.

Related errors

• AOT is not implemented for non-XLA lowerings.
• async_emb axis is not a mesh axis of
• async_emb emb_width=
• async_emb requires exactly one token shard per communicator…
• async_emb requires token shards to vary across the…

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/63418bf4f5f407ae. Report an issue: GitHub.

Appendix: source

Thrown at phoenix/xrex/models/transformer.py:572

        segment_ids_k=segment_ids_k,
        name=f"{name_prefix}_0",
        global_layer_index=global_layer_index + layer_index_offset,
        seqpack_layout=seqpack_layout,
    )
    h = d.output
    if debug_tensor_dump_output_folder is not None:
        layer_dumps.append(d.layer_dumps)

    if debug_tensor_dump_output_folder is not None:
        num_elements = len(layer_dumps[0])

        concatenated_dumps = []
        for i in range(num_elements):
            dumps = [dump[i] for dump in layer_dumps]
            if any(dump is None for dump in dumps):
                non_none_dump = next((dump for dump in dumps if dump is not None), None)
                if non_none_dump is None:
                    raise ValueError("All dumps are None")
                dummy_tensor = jnp.zeros_like(non_none_dump)
                dumps = [dump if dump is not None else dummy_tensor for dump in dumps]
            concatenated_dumps.append(jnp.stack(dumps, axis=0))
        layer_dumps = tuple(concatenated_dumps)

    return h, DecoderOutput(
        output=jnp.zeros(()),
        layer_dumps=layer_dumps,
    )


@dataclass
class Transformer(hk.Module):
    config: TransformerConfig
    sharding_context: ShardingContext
    name: Optional[str] = None

    summarizer_prefix: str = ""

View on GitHub (pinned to 24c60942c5)