hpcaitech/Open-Sora · error · NotImplementedError

ConvPixelUnshuffle downsample is not supported for video

Error message

ConvPixelUnshuffle downsample is not supported for video

What it means

The ConvPixelUnshuffle downsampling layer is only implemented for spatial (4D) inputs; when the autoencoder is constructed in video mode (is_video=True) this path raises NotImplementedError during build_downsample_block. Video tensors (B,C,T,H,W) are incompatible with the pixel-unshuffle conv layout used by this layer.

Source

Thrown at opensora/models/dc_ae/models/dc_ae.py:198

            if temporal_downsample:
                stride = (2, 2, 2)
            else:
                stride = (1, 2, 2)
        else:
            stride = 2
        block = ConvLayer(
            in_channels=in_channels,
            out_channels=out_channels,
            kernel_size=3,
            stride=stride,
            use_bias=True,
            norm=None,
            act_func=None,
            is_video=is_video,
        )
    elif block_type == "ConvPixelUnshuffle":
        if is_video:
            raise NotImplementedError("ConvPixelUnshuffle downsample is not supported for video")
        block = ConvPixelUnshuffleDownSampleLayer(
            in_channels=in_channels, out_channels=out_channels, kernel_size=3, factor=2
        )
    else:
        raise ValueError(f"block_type {block_type} is not supported for downsampling")
    if shortcut is None:
        pass
    elif shortcut == "averaging":
        shortcut_block = PixelUnshuffleChannelAveragingDownSampleLayer(
            in_channels=in_channels, out_channels=out_channels, factor=2, temporal_downsample=temporal_downsample
        )
        block = ResidualBlock(block, shortcut_block)
    else:
        raise ValueError(f"shortcut {shortcut} is not supported for downsample")
    return block


def build_upsample_block(

View on GitHub (pinned to 7ad6a96a13)

Solutions

Switch downsample_block_type to a video-capable layer (e.g. the conv-based downsample used by video configs such as dc-ae-videodata)
Use a video-specific config shipped with the repo
Keep is_video=False if you are only processing images

Example fix

# before
downsample_block_type: ConvPixelUnshuffle  # with is_video: true
# after
downsample_block_type: Conv            # video-capable layer

Defensive patterns

Strategy: validation

Validate before calling

if is_video:
    assert downsample_block_type != 'ConvPixelUnshuffle', 'ConvPixelUnshuffle downsample is image-only; use the conv path'

Prevention

Keep separate image and video config templates
Add a lint rule forbidding pixel-unshuffle blocks when is_video is true

When it happens

Trigger: Building a DCAE with is_video=True (e.g. for video encoding) whose config sets downsample_block_type='ConvPixelUnshuffle'; triggered from build_encoder_project_in_block or Encoder __init__.

Common situations: Reusing an image-model dc_ae config (e.g. dc-ae-512 or similar image checkpoints) to run on video; setting is_video in a config that was authored for image models.

Related errors

AI-assisted analysis of hpcaitech/Open-Sora@7ad6a96a13 (2026-08-28). Data as JSON: /api/errors/6223e7f4769cde67. Report an issue: GitHub.