Exploring DCN-like architecture for fast image generation with arbitrary resolution

Presentation issues. Thanks for your suggestions. We will re-organize the structure of our paper in the final version. We will add a preliminary section to introduce the background on DCN and Flow Matching. We will also rewrite the related work section to comprehensively discuss the relevant literature. The final version will be carefully revised according to your comments.

Ablation issues. Thanks for your suggestion. We will conduct ablation studies on ImageNet256 to offer more comprehensive results into the model's behavior. Due to limited time, we cannot provide the detailed results before the rebuttal ddl.

ImageNet $512\times512$ Experiments. We try to provide the ImageNet512 results of FlowDCN-XL/2 to show the generation power on high-resolution images. Due to the time constraint on the rebuttal, training FlowDCN-XL/2 from scratch on ImageNet at $512\times512$ resolution is not feasible. Instead, we fine-tune our pre-trained FlowDCN-XL/2 model, which was trained on ImageNet at $256\times256$ resolution of 1.5M steps, for just 100k steps on ImageNet 512. We adopt the same training pipeline as the original $256\times256$ resolution setting, without incorporating advanced techniques such as lognorm sampling, aspect ratio augmentation, and random cropping. Notably, our approach achieves a remarkable 2.76 FID score with the ODE solver in 50 steps, and 2.44 FID with the SDE solver in 250 steps.

Various Aspect Ratios training Experiments. Although FlowDCN, trained on fixed-resolution images, can generate images of arbitrary resolution within a reasonable aspect ratio range, the quality can be further enhanced by employing variable aspect ratio (VAR) training instead of a fixed $256\times256$ resolution. Therefore, we train a FlowDCN-B/2 model from scratch using VAR techniques to ensure a fair comparison with FiT, which inherently adopts VAR. We follow the same evaluation setting of FiT, using the same reference batch, without CFG. Note that the generated resolutions are the same with the FiT.

Which part contributes most to arbitrary resolution generation? Sorry for the confusion. Basically, the fundamental contribution to the ability of arbitrary resolution generation is our MultiScale DCN Block, which equips our model with the flexibility to handle different resolutions. So, the sentence in Line 245 is miss-leading and will be revised in the final version. Smax adjustment is another technique to further improve the global semantic consistency of generated images, as observed in Fig. 4. In rebuttal experiment, we further find aspect ratio augmentation during training is helpful to improve its performance on arbitary resolution generation. This result is easy to understand as data augmentation typically contributes to a better model with higher performance.

ImageNet $256\times256$ Long Iterations Experiments. Notably, our FlowDCN-XL/2 model, trained for only 1.5 million steps, achieves comparable results to SiT, which was trained for 7 million steps. This raises the question of whether FlowDCN-XL/2 is inherently more powerful or simply converges faster. To clarify this, we extended the training of our FlowDCN-XL/2 model by an additional 400k steps. The results show that our model attains a FID score of 2.01 with the ODE solver and 2.00 with the SDE solver, significantly outperforming SiT.

Various Aspect Ratios training Experiments. While FlowDCN, trained on fixed-resolution images, is capable of generating images of arbitrary resolution within a reasonable aspect ratio range, its performance can be improved by adopting variable aspect ratio (VAR) training instead of a fixed 256x256 resolution. To ensure a fair comparison with FiT, which inherently uses VAR, we train a FlowDCN-B/2 model from scratch using VAR techniques. We evaluate our model using the same pipeline and reference batch as FiT, without CFG.

The Evaluation of Arbitrary Resolution We follow the same evaluation pipeline of FiT, using the same reference statistics of ImageNet 256x256, without CFG.

The relationship between DCN and common CNN. As Eq.3 states, DCN introduces a deformable field $\Delta p(x)$ and a dynamic weight $w(x)$. When all channels share the same static weights instead of dynamic ones, and deformable field $\Delta p(x)$ degrades to zeros, DCN degenerates to common CNN. Therefore, in general, DCN-like architectures are more flexible and powerful than common CNNs. It should be noted that, the fixed $p_k$ in Tab. 2 (c) indicates that we freeze the $p_k$ (not the deformable field $\Delta p(x)$) and initialize it with a predefined grid. This setting is not equivalent to the common CNN.

Why not try a normal convolution. In many computer vision tasks, traditional CNNs have been outperformed by transformers, so we opted to explore a modern and advanced variant of CNN: Deformable Convolutional Networks (DCN). Additionally, we conducted a small experiment on ImageNet 256, where we replaced the DCN block in FlowDCN with a standard 3x3 group-wise convolution block. Due to limited time, the experiments with larger kernel of 5*5 is still on the way.

Different annotations for Total Training GFLOPs and 1-NFE FLOPs in Tab3. & 4. We appreciate your insightful feedback and concur with your opinions. We will clarify this point finally.

Lower IS metric. In Tab. 3, our FlowDCN consistently achieves superior performance in terms of the IS metric compared to its counterparts. However, the final results reported in Tab. 4 surprisingly yield relatively lower IS scores compared to DiT and SiT. Notably, we observe a trend of IS improvement when extending the training iteration by an additional 400k steps, with the score increasing from 257.5 at 1.5M training steps to 263.26 at 1.9M training steps. This suggests that the limited number of training steps may be the primary cause of the lower IS metric.

ImageNet $512\times512$ Experiments. Due to the time constraint on the rebuttal, training FlowDCN-XL/2 from scratch on ImageNet at $512\times512$ resolution is not feasible. Instead, we fine-tune our pre-trained FlowDCN-XL/2 model, which was trained on ImageNet at $256\times256$ resolution for 1.5M steps, for just 100k steps on ImageNet 512. We adopt the same training pipeline as the original $256\times256$ resolution setting, without incorporating advanced techniques such as lognorm sampling, aspect ratio augmentation, and random cropping. Notably, our approach achieves a remarkable 2.76 FID score with the ODE solver in 50 steps, and 2.44 FID with the SDE solver in 250 steps.

Deeply sorry, that the FlowCNN experiments referenced earlier utilized models with substantially larger parameter counts, attributable to a misconfigured number of groups and channels.

We have conducted a small experiment on ImageNet 256, where we replaced the DCN block in FlowDCN with a standard 3x3/5x5 group-wise convolution block, donated as FlowCNN. The results are as follows.

FID in Table 5 seems too high. Yes, we follow SiT, FiT and DiT to train our FlowDCN under 400K budgets. We follow the evaluation pipeline of FiT to obtain the metrics of arbitrary resolution generation without using CFG.

ImageNet $512\times512$ Experiments. Given the time constraint on the rebuttal, it is infeasible to train FlowDCN-XL/2 from scratch on ImageNet at a resolution of 512x512. As an alternative, we fine-tune our pre-trained FlowDCN-XL/2 model, which was initially trained on ImageNet at 256x256 resolution for 1.5 million steps, for an additional 100,000 steps on ImageNet 512. We utilize the same training pipeline as the original 256x256 resolution setting. Notably, our approach achieves a remarkable FID score of 2.76 with the ODE solver in just 50 steps, and 2.44 with the SDE solver in 250 steps.