Thank you for your feedback. We appreciate your recognition of the effectiveness of our method and the writing of our paper. Here are our responses to your comments.

Main Comments

Q1: Lack of text-to-image results.

A1: Thank you for your insightful suggestions. Given the limited time during the rebuttal phase, we follow U-ViT [1] and REPA [2] to conduct small-scale text-to-image generation experiments. Specifically, we adopt the same experimental setup as [1,2]: training and evaluating models from scratch on the MS-COCO dataset, using CLIP as the text encoder with a token length of 77.

We investigate two different approaches for incorporating textual features into DiCo.

1. The first uses the widely adopted cross-attention mechanism, integrated into the DiCo architecture to fuse text and visual features.
2. The second transforms CLIP text embeddings into dynamic depthwise convolution (DWC) kernels. We pad the text embeddings to a length of 81, feed them through a learnable MLP, and reshape the output into a 9×9 kernel. This kernel dynamically modulates DiCo's features via depthwise convolution. In this way, we can construct a fully convolutional text-to-image DiCo model without relying on any self-attention or cross-attention operations. We provide its PyTorch implementation below:

Algorithm A: PyTorch code of text conditional depthwise convolution

import torch
import torch.nn.functional as F

def text_conditional_dwconv(x, context):
    # x: (B, C, H, W) input feature maps
    # context: (B, 77, C) CLIP text embeddings after an MLP
    # output: (B, C, H, W) output after depthwise convolution
    B, C, H, W = x.shape
    context_pad = torch.cat([context, context[:, -1:].expand(-1, 4, -1)], dim=1)  # (B, 81, C)
    kernels = context_pad.reshape(B, 9, 9, C).permute(0, 3, 1, 2).reshape(B * C, 1, 9, 9)
    x_flat = x.view(1, B * C, H, W)
    output = F.conv2d(x_flat, kernels, padding=4, groups=B * C).view(B, C, H, W)
    return output

Both feature fusion modules are inserted after the Conv Module within each DiCo block. Following U-ViT, we match the model size and use a CFG scale of 2.0. All FID results of other methods reported below are taken from REPA [2].

Table: Comparison on MS-COCO for text-to-image generation

Our DiCo outperforms prior methods in both efficiency and generation quality. Notably, using text conditional DWC in place of cross-attention further improves throughput while maintaining competitive performance. This suggests that the fully convolutional DiCo has the potential to serve as the backbone for large-scale text-to-image diffusion models. We will include these results in the final version of our paper.

Q2: The necessity of the long residuals. UNet often converges faster with an inferior final performance. The U-Net inductive bias is not crucial (see introduction in DiT), and the U-shape limits further scaling (see Fig.4 in SD3).

A2: Thank you for this insightful observation. We would like to clarify the following points:

1. While the inductive bias of U-Net may become less crucial when ample training resources are available, it remains valuable for reducing training costs in resource-constrained scenarios.
2. Our proposed method is not limited to the U-Net structure. It also achieves strong performance within isotropic architectures.

Below, we provide a more detailed explanation of these points.

For point 1: We acknowledge that recent works such as DiT and SD3 have argued that the U-Net inductive bias is less critical when models are trained at large scales with ample computational resources. However, in many practical settings where compute is constrained, the U-Net-style long residual connections remain valuable for accelerating convergence and improving training efficiency.

As shown in the table below, DiCo with a U-Shape architecture achieves better performance than DiT while requiring significantly fewer training iterations. This highlights that the U-Shape structure enables faster convergence and lower training cost, which is especially important in resource-constrained settings.

Table: Comparison on ImageNet 256x256 for different iterations

Furthermore, we respectfully clarify that Fig. 4 in SD3 does not claim U-Net inherently limits scalability (see the FID and CLIP score curves). In fact, the SD3 paper explicitly notes that "Vanilla DiT underperforms UViT" (see Sec. 5.2.3, page 9). A similar observation is made in DreamOmni [3] (see Fig. 3 and Sec. 4, page 6). In practice, several large-scale diffusion models continue to use long residuals successfully. For example, DreamOmni [3] (U-Shape) and Vidu [4] (Isotropic with skip connections) both adopt architectures with long residuals at scale.

For point 2: Equally important, we emphasize that the U-Shape design is not our core architectural contribution. As shown in Table 3 of the paper, DiCo consistently outperforms DiT in both performance and efficiency across various architectural settings (isotropic, isotropic with skip connections, and U-shape). Moreover, we demonstrate that our method can be effectively applied to build isotropic diffusion models, achieving superior performance (FID of 1.33 on ImageNet 256x256) compared to modern DiTs (refer to A3). This provides strong evidence that our method generalizes well beyond U-Net designs and is not inherently tied to long residuals.

We will include the above discussion in the final version of the paper.

Q3: The claim of being SOTA is unjustified, as DiCo's performance lags behind modern DiTs such as LightningDiT.

A3: We appreciate your feedback regarding our SOTA claim. Given the rapid advancements in modern DiTs, we agree that the SOTA statement in our paper should be revised. We also recognize the importance of comparing our method with recent DiTs such as LightningDiT. We find that after aligning with their experimental settings, our method achieves competitive performance along with a clear efficiency advantage. The details are shown as follows.

In our paper, we adopt the original DiT experimental setup (e.g., using SD-VAE, standard DDPM training, fixed hyperparameters) to ensure a fair and controlled comparison with DiT, DiG, and DiC. However, LightningDiT differs from DiT's setup in several aspects—for example, it replaces the SD-VAE with VA-VAE, adopts the Rectified Flow training objective, and modifies training hyperparameters, etc.

To fairly assess DiCo’s competitiveness under this enhanced setup, we construct LightningDiCo, which replaces LightningDiT’s self-attention with our Conv Module, while keeping all other settings identical. This version uses an isotropic backbone (no U-Net structure), aligning directly with LightningDiT.

Table: Comparison with modern DiTs on ImageNet 256x256

LightningDiCo achieves an FID score of 1.33 and offers a ~2.8× throughput gain over modern DiTs. This demonstrates that our convolutional backbone can match or exceed the generative quality of modern DiTs while being significantly more efficient. We will include this comparison in the final version of the paper and revise the SOTA claim.

Q4: Missing comparison with FlowDCN.

A4: Thank you for pointing this out. We agree that deformable convolution is a strong and widely used convolutional variant, and FlowDCN represents an important baseline in this space.

To fairly compare with FlowDCN, we retain its original isotropic architecture and training setup, and replace its MultiScale DCN with our proposed Conv Module. The results are shown below:

Table: Comparison on ImageNet 256x256 (400K iterations, Small models)

Our Conv Module achieves better generative performance and ~2.1× higher throughput compared to MultiScale DCN, despite being structurally simpler and easier to implement. We will include this comparison in the final version of the paper and add a discussion of FlowDCN in the related work.

Thank you again for helping us strengthen our work. We hope that our response has addressed your concerns. We look forward to further communication with you!

[1] All Are Worth Words: A ViT Backbone for Diffusion Models. In CVPR, 2023.

[2] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. In ICLR, 2025.

[3] DreamOmni: Unified Image Generation and Editing. In CVPR, 2025.

[4] Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233.

Thank you for your feedback. We appreciate your recognition of the motivation and effectiveness of our method. Here are our responses to your comments.

Main Comments

Q1: Lack of experiments for text-to-image generation.

A1: Thank you for your insightful suggestions. Given the limited time during the rebuttal phase, we follow U-ViT [1] and REPA [2] to conduct small-scale text-to-image generation experiments. Specifically, we adopt the same experimental setup as [1,2]: training and evaluating models from scratch on the MS-COCO dataset, using CLIP as the text encoder with a token length of 77.

We investigate two different approaches for incorporating textual features into DiCo.

1. The first uses the widely adopted cross-attention mechanism, integrated into the DiCo architecture to fuse text and visual features.
2. The second transforms CLIP text embeddings into dynamic depthwise convolution (DWC) kernels. We pad the text embeddings to a length of 81, feed them through a learnable MLP, and reshape the output into a 9×9 kernel. This kernel dynamically modulates DiCo's features via depthwise convolution. In this way, we can construct a fully convolutional text-to-image DiCo model without relying on any self-attention or cross-attention operations. We provide its PyTorch implementation below:

Algorithm: PyTorch code of text conditional depthwise convolution

import torch
import torch.nn.functional as F

def text_conditional_dwconv(x, context):
    # x: (B, C, H, W) input feature maps
    # context: (B, 77, C) CLIP text embeddings after an MLP
    # output: (B, C, H, W) output after depthwise convolution
    B, C, H, W = x.shape
    context_pad = torch.cat([context, context[:, -1:].expand(-1, 4, -1)], dim=1)  # (B, 81, C)
    kernels = context_pad.reshape(B, 9, 9, C).permute(0, 3, 1, 2).reshape(B * C, 1, 9, 9)
    x_flat = x.view(1, B * C, H, W)
    output = F.conv2d(x_flat, kernels, padding=4, groups=B * C).view(B, C, H, W)
    return output

Both feature fusion modules are inserted after the Conv Module within each DiCo block. Following U-ViT, we match the model size and use a CFG scale of 2.0. All FID results of other methods reported below are taken from REPA [2].

Table: Comparison on MS-COCO for text-to-image generation

Our DiCo outperforms prior methods in both efficiency and generation quality. Notably, using text conditional DWC in place of cross-attention further improves throughput while maintaining competitive performance. This suggests that the fully convolutional DiCo has the potential to serve as the backbone for large-scale text-to-image diffusion models. We will include these results in the final version of our paper.

Q2: Functional overlap between CCA and existing channel attention mechanisms (e.g., channel self-attention).

A2: Thank you for raising this insightful point. We acknowledge that CCA and channel self-attention both aim to modulate channel-wise feature importance. However, our proposed CCA is specifically designed to be more efficient, particularly at large model scales, while retaining comparable generative performance.

Unlike standard channel self-attention, which introduces quadratic complexity with respect to channel numbers, CCA adopts a simplified, structured attention mechanism that avoids heavy pairwise computations. This design choice enables substantial computational savings without sacrificing effectiveness. The comparison between CCA and channel self-attention at the XL model scale is shown below:

Table: CCA vs. channel self-attention on ImageNet 256x256 (400K iterations, XL models)

CCA achieves a ~1.4× speedup with slightly better FID and IS, demonstrating its advantage as a lightweight and scalable alternative to conventional channel attention. We will include this analysis and additional clarification on CCA’s design motivation in the final version of the paper.

Q3: Discussion of addressing long-range dependencies and generalizability.

A3: Thank you for highlighting this important point. We acknowledge that the local receptive field of convolutional networks may limit their ability to capture long-range dependencies, especially in high-resolution settings.

To address this, we explore two complementary solutions in DiCo:

1. Enlarging convolutional kernel sizes: A straightforward yet effective way to expand the receptive field is to use larger convolution kernels. As shown in Table 4 of our paper, increasing the kernel size leads to consistent improvements in generation quality, albeit with a trade-off in computational efficiency. This approach enables DiCo to capture broader context while maintaining the simplicity of convolution.
2. Employing high-compression autoencoders: An alternative solution is to leverage modern latent diffusion frameworks with highly compressed latent spaces, which reduces the burden on the backbone to model long-range spatial dependencies directly. Recent works such as VA-VAE [3] and DC-AE [4] propose autoencoders with downsampling ratios of up to 16× or 32×, which effectively shrink the spatial dimensions.

We conduct experiments with VA-VAE on ImageNet 256×256 and DC-AE on ImageNet 512×512 following their experimental setups. The results are shown below:

Table: Comparison on ImageNet 256x256 using different Autoencoders

Table: Comparison on ImageNet 512x512 using different Autoencoders

The use of high-compression autoencoders mitigates the locality limitation of convolution and enables DiCo to model long-range dependencies more effectively. Notably, DiCo achieves better FID scores of 1.33 on ImageNet 256×256 and 1.78 on ImageNet 512×512 with these autoencoders. We hope to further explore the application of DiCo in more complex scenarios in future work. We will include the discussion in the final version of our paper.

Q4: Table 4 consistently shows FID values below 54 across various configurations, whereas baseline models (DiT-S/2, DiC-S, DiG-S/2) in Table 1 report FID scores exceeding 58.

A4: Thank you for your careful review. The difference in FID values between Table 1 and Table 4 primarily stems from architecture settings. In Table 1, both DiCo and DiC are based on the U-Shape architecture, while DiT and DiG adopt isotropic architectures. In the ablation studies presented in Table 4, we keep the U-Shape architecture fixed while ablating different modules within DiCo. Moreover, in Table 3, we show that DiCo consistently achieves superior performance across different architectural settings. To make this distinction more explicit, we summarize the fair comparisons below by isolating the effect of architecture:

Table: Comparison on ImageNet 256x256 (Isotropic, 400K iterations, Small models)

Table: Comparison on ImageNet 256x256 (U-Shape, 400K iterations, Small models)

Under a fair comparison with identical architectures, our DiCo consistently outperforms the baseline methods. This indicates that the performance and efficiency gains of DiCo stem from our proposed Conv Module and CCA. We will clarify this point more explicitly in the final version of the paper.

Thank you again for helping us strengthen our work. We hope that our response has addressed your concerns. We look forward to further communication with you!

[1] All Are Worth Words: A ViT Backbone for Diffusion Models. In CVPR, 2023.

[2] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. In ICLR, 2025.

[3] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models. In CVPR, 2025.

[4] Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models. In ICLR, 2025.

Thank you for your feedback. We appreciate your recognition of our method's innovation and effectiveness. Here are our responses to your comments.

Main Comments

Q1: No text-to-image generation results, even small-scale.

A1: Thank you for your insightful suggestions. Given the limited time during the rebuttal phase, we follow U-ViT [1] and REPA [2] to conduct small-scale text-to-image generation experiments. Specifically, we adopt the same experimental setup as [1,2]: training and evaluating models from scratch on the MS-COCO dataset, using CLIP as the text encoder with a token length of 77.

We investigate two different approaches for incorporating textual features into DiCo.

1. The first uses the widely adopted cross-attention mechanism, integrated into the DiCo architecture to fuse text and visual features.
2. The second transforms CLIP text embeddings into dynamic depthwise convolution (DWC) kernels. We pad the text embeddings to a length of 81, feed them through a learnable MLP, and reshape the output into a 9×9 kernel. This kernel dynamically modulates DiCo's features via depthwise convolution. In this way, we can construct a fully convolutional text-to-image DiCo model without relying on any self-attention or cross-attention operations. We provide its PyTorch implementation below:

Algorithm: PyTorch code of text conditional depthwise convolution

import torch
import torch.nn.functional as F

def text_conditional_dwconv(x, context):
    # x: (B, C, H, W) input feature maps
    # context: (B, 77, C) CLIP text embeddings after an MLP
    # output: (B, C, H, W) output after depthwise convolution
    B, C, H, W = x.shape
    context_pad = torch.cat([context, context[:, -1:].expand(-1, 4, -1)], dim=1)  # (B, 81, C)
    kernels = context_pad.reshape(B, 9, 9, C).permute(0, 3, 1, 2).reshape(B * C, 1, 9, 9)
    x_flat = x.view(1, B * C, H, W)
    output = F.conv2d(x_flat, kernels, padding=4, groups=B * C).view(B, C, H, W)
    return output

Both feature fusion modules are inserted after the Conv Module within each DiCo block. Following U-ViT, we match the model size and use a CFG scale of 2.0. All FID results of other methods reported below are taken from REPA [2].

Table: Comparison on MS-COCO for text-to-image generation

Our DiCo outperforms prior methods in both efficiency and generation quality. Notably, using text conditional DWC in place of cross-attention further improves throughput while maintaining competitive performance. This suggests that the fully convolutional DiCo has the potential to serve as the backbone for large-scale text-to-image diffusion models. We will include these results in the final version of our paper.

Q2: Applicability to pixel-level diffusion.

A2: Thank you for your thoughtful question. Yes, the effectiveness of our method also holds in the pixel-level diffusion setting.

Following U-ViT’s setup, we conduct experiments on the ImageNet 64×64 dataset using pixel-level diffusion. We adjust DiCo’s width and depth to match the overall model size of U-ViT-L/4 for a fair comparison. As shown below, DiCo achieves significantly better performance and efficiency:

Table: Pixel-level diffusion on ImageNet 64×64

These results confirm that DiCo generalizes well to pixel-level diffusion. The benefits of convolutional locality and lightweight channel-wise modulation (via CCA) remain consistent in this setting. We will include these results in the final version of the paper.

Q3: Are the baselines in Table 1 with CFG or not?

A3: Thank you for the careful observation. In Table 1, all models except those trained for 400K iterations use CFG during evaluation. The 400K-iteration models are intended for early-stage comparison and are reported without CFG, consistent with prior works (e.g., DiT, DiG).

All baseline metrics are taken directly from their respective published papers to ensure fairness. We will clarify the use of CFG in Table 1 and the main text in the final version of the paper.

Q4: Comparison with DiT at a larger resolution like 1024x1024.

A4: Thank you for the valuable suggestion. We conduct experiments on ImageNet 1024×1024. Since the original DiT paper does not report results at this resolution, we train both DiT and DiCo from scratch under the same settings for a fair comparison. Due to DiT’s significantly slower training speed at high resolutions, we use small-scale models and train them for 400K iterations. The results are shown below:

Table: Comparison on ImageNet 1024x1024 (400K iterations, Small models)

These results clearly show that DiCo scales much more efficiently to high resolutions—achieving a 5× speedup with better generation quality. We will include these findings in the final version of the paper.

Thank you again for helping us strengthen our work. We hope that our response has addressed your concerns. We look forward to further communication with you!

[1] All Are Worth Words: A ViT Backbone for Diffusion Models. In CVPR, 2023.

[2] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. In ICLR, 2025.

Thank you for your feedback. We appreciate your recognition of the value of visual insights and the adequacy of experiments. Here are our responses to your comments.

Main Comments

Q1: Insufficient innovation in the model architecture.

A1: Thank you for your valuable comments. From a broader perspective, our work challenges a popular assumption in today’s community: that Vision Transformers (e.g., DiT) are inherently superior to convolutional architectures for generative modeling. Our proposed DiCo demonstrates that a purely convolutional design can not only match but outperform ViT-based diffusion models like DiT in terms of both quality and efficiency. This offers a fresh and timely perspective to the research community, which in itself constitutes an important form of innovation.

Moreover, we should know that the technical innovation should be consistent with the research background and motivation. In terms of model design, our innovations are reflected in both the overall architecture and key components:

1. Overall architecture: DiT is a pioneering diffusion model that applies the ViT as its backbone. Building on this line of research, DiCo introduces a fully convolutional backbone for diffusion models, inspired by the localized nature of attention in DiT. This design remains largely unexplored in the diffusion-based generative modeling literature. Furthermore, we extend DiCo to the text-to-image generation setting (refer to A2), resulting in a fully convolutional text-to-image model. This is a significant departure from existing text-to-image architectures, which predominantly rely on different attention mechanisms.
2. Component-level design: We identify a specific challenge in convolutional designs—channel redundancy—and propose a simple yet effective solution: the CCA module. While CCA shares some similarities with SE modules, it removes the FC layers and RELU non-linearities, resulting in a lighter and more efficient design. As shown in Table 4, CCA improves both generation quality and throughput compared to SE, validating its suitability in generative settings.

Overall, our model offers fresh insights to the research community and introduces a novel design perspective, through which it showcases meaningful technical innovation.

Q2: Lacking comparison with Stable Diffusion's UNet and FLUX's MM-DiT.

A2: Thank you for raising this important point. We agree that comparing DiCo with Stable Diffusion's UNet and FLUX's MM-DiT is essential for evaluating its effectiveness.

First, regarding FLUX's MM-DiT: its architecture is a variant of DiT, with the primary difference lying in how textual features are handled. When replacing the text prompt encoder with a class embedding module (as in class-conditional ImageNet generation), the architecture effectively reduces to DiT. Therefore, we consider MM-DiT a meaningful representative of DiT-style backbones in the text-to-image setting.

Second, we acknowledge the importance of comparing against strong baselines such as Stable Diffusion's UNet and FLUX's MM-DiT. Given the limited time during the rebuttal phase, we follow prior works (U-ViT [1], REPA [2]) and conduct a fair small-scale text-to-image evaluation using Stable Diffusion’s U-Net and FLUX's MM-DiT as baselines. Specifically, we adopt the same experimental setup as [1,2]: training and evaluating models from scratch on the MS-COCO dataset, using CLIP as the text encoder with a token length of 77.

We investigate two different approaches for incorporating textual features into DiCo.

1. The first uses the widely adopted cross-attention mechanism, integrated into the DiCo architecture to fuse text and visual features.
2. The second transforms CLIP text embeddings into dynamic depthwise convolution (DWC) kernels. We pad the text embeddings to a length of 81, feed them through a learnable MLP, and reshape the output into a 9×9 kernel. This kernel dynamically modulates DiCo's features via depthwise convolution. In this way, we can construct a fully convolutional text-to-image DiCo model without relying on any self-attention or cross-attention operations. We provide its PyTorch implementation below:

Algorithm: PyTorch code of text conditional depthwise convolution

import torch
import torch.nn.functional as F

def text_conditional_dwconv(x, context):
    # x: (B, C, H, W) input feature maps
    # context: (B, 77, C) CLIP text embeddings after an MLP
    # output: (B, C, H, W) output after depthwise convolution
    B, C, H, W = x.shape
    context_pad = torch.cat([context, context[:, -1:].expand(-1, 4, -1)], dim=1)  # (B, 81, C)
    kernels = context_pad.reshape(B, 9, 9, C).permute(0, 3, 1, 2).reshape(B * C, 1, 9, 9)
    x_flat = x.view(1, B * C, H, W)
    output = F.conv2d(x_flat, kernels, padding=4, groups=B * C).view(B, C, H, W)
    return output

Both fusion modules are inserted after the Conv Module in each DiCo block. Following U-ViT, we match model size and use a CFG scale of 2.0. The FID result for Stable Diffusion’s U-Net is taken from U-ViT [1], and that for FLUX's MM-DiT is from REPA [2].

Table: Comparison on MS-COCO for text-to-image generation

DiCo outperforms both Stable Diffusion's UNet and FLUX's MM-DiT in terms of efficiency and generation quality. Notably, using text conditional DWC in place of cross-attention further improves throughput while maintaining competitive performance. This suggests that the fully convolutional DiCo has the potential to serve as the backbone for large-scale text-to-image diffusion models. We will include these results in the final version of our paper.

Q3: Inappropriate abbreviation.

A3: Thank you for pointing this out. We understand the concern that the abbreviation "ConvNet" could be confused with "ConvNeXt," and we agree that "CNN" is the more widely recognized and unambiguous term in the deep learning community. While “ConvNet” has been used in several prior works (e.g., ConvNeXt [3], RepVGG [4]), we acknowledge that using "CNN" would provide greater clarity for readers. We will revise the manuscript to replace "ConvNet" with "CNN" throughout, in order to avoid confusion and maintain consistency with common terminology.

Thank you again for helping us strengthen our work. We hope that our response has addressed your concerns. We look forward to further communication with you!

[1] All Are Worth Words: A ViT Backbone for Diffusion Models. In CVPR, 2023.

[2] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. In ICLR, 2025.

[3] A ConvNet for the 2020s. In CVPR, 2022.

[4] RepVGG: Making VGG-style ConvNets Great Again. In CVPR, 2021.