Native-Resolution Image Synthesis

Thank you sincerely for your valuable suggestions and thorough review. We have thoughtfully addressed each of your points, as detailed in the responses below:

1. Key distinctions between FiT and NiT

Summary. NiT replaces FiT’s padding-centric regime with a packing strategy that eliminates dummy tokens, learns directly on native resolutions, and introduces packed attention and packed AdaLN modules that respect variable sequence boundaries. NiT learns the intrinsic spatial hierarchy of original inputs, capable of generating a more diverse range of resolutions and aspect ratios. This generalization is fundamentally different from FiT's reliance on RoPE extrapolation techniques for generalization.

2. Fair comparison between FiT and NiT

Both using DC-AE, we train the FiT-B/1 (i.e., 131M model size with the patch size of 1) and NiT-B/1 model with the same training configurations (131,072 token budget with 100K training steps, learning rate 1e-4). We provide the FID of FiT and NiT among diverse resolutions as follows:

The above table reveals the superiority of our NiT compared to FiT. NiT surpasses the performance of FiT on both 256x256 and 512x512 benchmarks. Besides, NiT demonstrates more stable and robust resolution generalization capability compared with FiT.

3. Additional related work

We apologize for the lack of related work on GAN-based native-resolution studies. We sincerely thank the reviewer for the constructive suggestion and would add these papers to the related work in revised manuscript.

The author address my concerns well but after seeing other concerns from other author, I decide to raise the score to 4 only.

We truly thank the reviewer for your detailed review and thoughtful comments in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:

1. Detailed experimental setup

• Hyper-parameters. As batch size is unsuitable in our packed image processing, we use the token budget (i.e., the summation of total tokens in all training iterations) to represent the training costs. For each iteration, we use $131,072$ tokens, which corresponds to a $512$ batch size on $256\times256$ generation of DiT models. We train our NiT-XL model for $1000K$ steps, with AdamW optimizer and a constant learning rate of $1\times{10}^{-4}$. Mixed-precision training is adopted with $`torch.bfloat16`$.
• Data pre-processing. We use DC-AE to convert an image with the shape of $H\times W$ into the latent tokens with the shape of $\frac{H}{32}\times\frac{W}{32}$. All the images are processed without the cropping operation to avoid its negative effects.
• Training procedure. To support packed training, we pre-counted the number of latent codes corresponding to different images, given a preset maximum length, we flattened the different latent tokens and packed them into one sequence. We use the longest-pack-first histogram packing algorithm to pack the images into different sequences. In our setting, the maximum sequence length is similar to the batch size in normal training. The instance number among different packed sequences is dynamic because diverse image resolutions lead to diverse latent token numbers. In the training stage, we directly train NiT with these packed sequences, as our NiT architecture (packed full-attention and packed AdaLN modules) supports efficient packed processing.
• Reproducibility. We would open-source all the code, scripts, datasets, and model checkpoints to ensure reproducibility.

2. Limitation

2.1 Resolution scalability To further evaluate the resolution scalability of NiT, we study the resolution extrapolation law of NiT. First, we counted the average sequence length of the whole training ImageNet dataset $L_\text{avg}$. We find that NiT takes an average sequence length of $L_{\text{avg}}=171$ during training. Then we calculate the sequence length and the FID score of these extrapolated resolutions, detailed in the following table:

Thus, we can quantify the resolution scalability of NiT by an extrapolation ratio $r = L/L_{\text{avg}}$, where $L$ is the longer side of the test image and $L_{\text{avg}}$ is the average longer side in the training set. Empirically, we observe a remarkably tight logarithmic relationship between this ratio $r$and FID metrics, captured by$\text{FID}(r) = 2.234\cdot\ln r + 1.0224$with an$R^{2}=0.9641$, indicating that the formula explains more than 96 % of the variance across resolutions. The slope 2.234, therefore, quantifies the resolution scalability of NiT, effectively describing the high-resolution extrapolation quality of NiT. We will incorporate these findings and discuss potential strategies to flatten this slope and further improve high-resolution fidelity in the revised manuscript.

2.2 Aspect ratio scalability. NiT-XL is trained on native-resolution images and therefore transfers smoothly to the common photographic aspect ratios—9 : 16, 3 : 4, 4 : 3, and 16 : 9—without any special handling. The vast majority of ImageNet training images fall within the continuous band [9/16, 16/9], and within this range, the model preserves object shapes and spatial coherence. Once the aspect ratio drifts outside that interval, however, objects start to appear unnaturally stretched or compressed. The most pronounced degradation occurs at the extremes of 1 : 3 and 3 : 1, ratios that are virtually absent from the training distribution. As a result, we treat [1/3, 3] as the current practical upper bound for NiT’s aspect-ratio generalization; supporting ratios beyond this would likely require targeted data augmentation or architectural adjustments.

2.3 Rationale for the current limitation and future work

• The chief bottleneck is the narrow resolution distribution in ImageNet itself. As shown in Fig. 2(a) of the manuscript, most ImageNet images fall between 200 and 600 pixels on their longer side, with very few exceeding 800 pixels. Consequently, the average sequence length is only 171 tokens, providing limited signal for learning truly scale-invariant representations.
• Looking ahead, we plan to (i) design architectures and training schemes that explicitly target resolution and aspect-ratio robustness, and (ii) study a dedicated “aspect-ratio scaling law” for NiT, which we expect to be more nuanced than the resolution law reported here. These steps should extend the model’s effective operating range beyond the present [1/3, 3] aspect-ratio band and improve high-resolution fidelity.

3. Societal impacts of this work

3.1 Positive societal impacts

• Enhanced Image Generation for Broad Applications: By accommodating various resolutions and aspect ratios within a single diffusion model, this approach can streamline workflows across industries (e.g., graphic design, healthcare imaging, education) where high-fidelity images at diverse scales are crucial.
• Reduced Computational Overhead: Training one unified model rather than multiple resolution-specific models may lower overall computational costs and carbon footprint, benefiting sustainability efforts.
• Democratization of High-Quality Image Synthesis: The ability to handle images at unseen scales may make advanced generative capabilities more accessible to smaller organizations or researchers with limited resources, fostering innovation.

3.2 Negative societal impacts

• Potential for Misinformation and Deepfakes: As generative models improve in fidelity, they can be misused to create highly convincing fake images or videos, posing risks to social trust and security.
• Impact on Creative Industries: Widespread adoption of automated image generation tools could disrupt commercial art, photography, or design sectors, raising questions about job displacement and the value of human creativity.

4. Comparisons with more SOTA baselines

We compare our NiT with more recent SOTA baselines: DiG [1] (CVPR2025), DeepFlow [2], GigaTok [3] (ICCV2025), RandAR [4] (ICML2025), and UCGM [5]. The results are summarized as follows:

Our NiT model demonstrates superior performance compared to these baselines on both ImageNet-$256\times256$ and ImageNet-$512\times$ benchmarks. Notably, we use a single model to compete on the two benchmarks.

Reference

[1] Zhu, Lianghui, Zilong Huang, Bencheng Liao, Jun Hao Liew, Hanshu Yan, Jiashi Feng, and Xinggang Wang. "Dig: Scalable and efficient diffusion models with gated linear attention." CVPR, 2025.

[2] Shin, Inkyu, Chenglin Yang, and Liang-Chieh Chen. "Deeply supervised flow-based generative models." arXiv preprint arXiv:2503.14494 (2025).

[3] Xiong, Tianwei, Jun Hao Liew, Zilong Huang, Jiashi Feng, and Xihui Liu. "Gigatok: Scaling visual tokenizers to 3 billion parameters for autoregressive image generation." arXiv preprint arXiv:2504.08736 (2025).

[4] Pang, Ziqi, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T. Freeman, and Yu-Xiong Wang. "Randar: Decoder-only autoregressive visual generation in random orders." CVPR, 2025.

[5] Sun, Peng, Yi Jiang, and Tao Lin. "Unified Continuous Generative Models." arXiv preprint arXiv:2505.07447 (2025).

Thank you very much for your insightful feedback and careful review, which has greatly helped improve the manuscript. In response to your comments, we have carefully prepared point-to-point clarifications as follows:

1. Qualitative analysis

• We thank the reviewer for suggesting more qualitative results in the main text, and we will add a detailed qualitative analysis in the following revision. Currently, we have provided the qualitative results of our model and the comparison with baselines EDM and FlowDCN in the supplementary material.
• We regret that our submission did not include visual examples of failure cases. In follow-up tests at very high resolutions (e.g., 2048 × 2048), we indeed observed spatial artefacts: certain objects are rendered with anatomically implausible features or with parts arranged unnaturally, compromising spatial coherence. Because rebuttal guidelines prohibit anonymous external links, we cannot attach the images here; however, we will include representative visualizations, together with an analysis of their causes and possible mitigations, in the revised manuscript.

2. The limitation of this method

2.1 Resolution scalability To further evaluate the resolution scalability of NiT, we study the resolution extrapolation law of NiT. First, we counted the average sequence length of the whole training ImageNet dataset $L_\text{avg}$. We find that NiT takes an average sequence length of $L_{\text{avg}}=171$ during training. Then we calculate the sequence length and the FID score of these extrapolated resolutions, detailed in the following table:

Thus, we can quantify the resolution scalability of NiT by an extrapolation ratio $r = L/L_{\text{avg}}$, where $L$ is the longer side of the test image and $L_{\text{avg}}$ is the average longer side in the training set. Empirically, we observe a remarkably tight logarithmic relationship between this ratio $r$and FID metrics, captured by$\text{FID}(r) = 2.234\cdot\ln r + 1.0224$with an$R^{2}=0.9641$, indicating that the formula explains more than 96 % of the variance across resolutions. The slope 2.234, therefore, quantifies the resolution scalability of NiT, effectively describing the high-resolution extrapolation quality of NiT. We will incorporate these findings and discuss potential strategies to flatten this slope and further improve high-resolution fidelity in the revised manuscript.

2.2 Aspect ratio scalability. NiT-XL is trained on native-resolution images and therefore transfers smoothly to the common photographic aspect ratios—9 : 16, 3 : 4, 4 : 3, and 16 : 9—without any special handling. The vast majority of ImageNet training images fall within the continuous band [9/16, 16/9], and within this range, the model preserves object shapes and spatial coherence. Once the aspect ratio drifts outside that interval, however, objects start to appear unnaturally stretched or compressed. The most pronounced degradation occurs at the extremes of 1 : 3 and 3 : 1, ratios that are virtually absent from the training distribution. As a result, we treat [1/3, 3] as the current practical upper bound for NiT’s aspect-ratio generalization; supporting ratios beyond this would likely require targeted data augmentation or architectural adjustments.

2.3 Rationale for the current limitation and future work

• The chief bottleneck is the narrow resolution distribution in ImageNet itself. As shown in Fig. 2(a) of the manuscript, most ImageNet images fall between 200 and 600 pixels on their longer side, with very few exceeding 800 pixels. Consequently, the average sequence length is only 171 tokens, providing limited signal for learning truly scale-invariant representations.
• Looking ahead, we plan to (i) design architectures and training schemes that explicitly target resolution and aspect-ratio robustness, and (ii) study a dedicated “aspect-ratio scaling law” for NiT, which we expect to be more nuanced than the resolution law reported here. These steps should extend the model’s effective operating range beyond the present [1/3, 3] aspect-ratio band and improve high-resolution fidelity.

3. Impact of specific resolution and aspect-ratio distributions

We conduct 4 groups of experiments with different dataset distributions to explore this effect. (a) we keep the aspect ratio of image data and resize its height and width smaller than $256$; (b) we keep the aspect ratio of image data and resize its height and width smaller than $512$; (c) we keep the aspect ratio of image data and resize its height and width smaller than $768$; (d) we use native-resolution image data for training without resizing them under certain resolution. We report the FID of each setting and resolution as in the following table.

We find that the training resolution distribution significantly impacts the generalization ability. When training images are capped at $256$ px on the long side (Config a), the model excels at $256 \times 256$ but degrades sharply at $512 \times 512$ and $768 \times 768$. Raising the cap to $512$ px (Config b) or $768$ px (Config c) postpones—but does not eliminate—the drop-off: once the test resolution exceeds the highest resolution encountered in training, FID climbs rapidly. By contrast, training at native image resolution with no upper bound (Config d) yields consistently strong performance across the full range of resolutions we evaluated. These results underline a simple principle: the broader the resolution spectrum in the training set, the better NiT generalizes, highlighting the necessity of resolution diversity.

We sincerely appreciate your constructive comments and efforts in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:

1. Effectiveness in large-scale settings

NiT remains both efficient and effective, even in large-scale text-to-image scenarios. As mentioned in line 222 of our manuscript, we have included detailed setup information and experimental results for the text-to-image generation task in the supplementary materials. Specifically, we trained a model containing $673M$ parameters on the SAM [1] dataset (comprising $10M$ images), utilizing descriptive captions generated by MiniCPM-V. Importantly, NiT-T2I achieves a strong CLIP score of $0.345$ and notably outperforms SD-v1.5 and SDXL-Turbo in both FID and CLIP scores, despite having a significantly smaller model size and lower training costs.

2. Training with diverse resolutions

2.1 Concrete details on experimental setup. We provide the details on multi-GPU parallel in our native-resolution training setting.

• Preparing a packed distributed sampler. We first construct the metafile, restoring the index and information of each image instance. Given a preset maximum length, we pack different image instances into one sequence, using the longest-pack-first histogram packing algorithm [2]. We record the index of each image instance in a sequence. Then, we use these indices to load the corresponding image instance and pack them into a sequence for each training iteration. For example, we first obtain the meta file of each image file with a piece as follows:

  {"index": 0, "image_file": "n01601694/n01601694_11629.JPEG", "image_h": 384, "image_w": 576,}
  {"index": 1, "image_file": "n01601694/n01601694_11799.JPEG", "image_h": 320, "image_w": 480}
    ......
  {"index": 1281166, "image_file":......}
  

After setting the index of each image instance, we build a packed distributed sampler: [ [846047, 838285, 837472, 836430, 828275, 807897], [816704, 796202, 793598, 792986, 781934, 769695], ......., ] Each list records the indices of image instances to be packed into one sequence. Note that the list may have different numbers of image instances, as we adhere to the native token number of each image instance.

• Given $N$ GPUs, with the above packed distributed sampler, we load the processed image latent codes according to the index provided by the sampler. Then we pack these codes into one sequence independently on each device. On the local devices, the model uses these packed sequences to calculate and then gather the loss to calculate the gradient. Then the gradient back-propagates to update the model parameters.

2.2 Comparison with bucket sampling. The comparison of bucket sampling and native-resolution strategy can be summarized as follows:

To further validate our claims, we use the same dataset (SAM [1] and JourneyDB [3]) as PixArt-$\alpha$ [4] and train our NiT-XL model to compare the effectiveness of our native-resolution strategy. PixArt-$\alpha$ divides the image size into $40$ buckets with different aspect ratios, each with varying aspect ratios ranging from $0.25$ to $4$. In contrast, our NiT directly learns the native-resolution of these image datasets. We use FID and CLIP Score on the COCO-30K [5] benchmark to evaluate the models. The results are summarized as follows:

NiT demonstrates more stable performance on a diverse resolution range than PixArt-$\alpha$. On $512\times512$ resolution, PixArt-$\alpha$ fails to generate meaningful samples, but NiT can generalize to this resolution. Besides, NiT demonstrates superior FID and CLIP Score on other resolutions, validating the effectiveness of our native-resolution training strategy.

2.3 Reliance on VAE. We follow the dominant latent diffusion model (LDM) paradigm to train our models, necessitating the usage of an image VAE. We use DC-AE with a $32$ downsampling scale to support super high-resolution (up to $2048$ resolution) image synthesis. For $2048\times2048$ resolution, it leads to $4096$ tokens with DC-AE and $65536$ tokens with SD-VAE [6], which has an $8$ downsampling ratio. In the real-world scenario, the former setting is more practical as it produces fewer tokens, facilitating the device requirement. We recognize that a $32$ downsampling scale may limit the granularity of resolutions, but it is compatible with most normal resolutions (such as $256, 512, 1024, 2048$, etc.) and is more feasible in super high-resolution generation scenarios.

3. Performance gain

We rigorously conduct experiments to validate the effects of each component. We conduct four groups for this component analysis: (a) we use fixed $512\times512$-resolution data and absolute positional embedding (sin-cos PE) as our positional embedding, corresponding to the SiT [7] baseline. (b) We use fixed $512\times512$-resolution data and RoPE. (c) We use native-resolution data and absolute positional embedding. (d) We use native-resolution data and RoPE, corresponding to our NiT. The following table shows the FID score of each setting and resolution.

The above table demonstrates the performance gain of each component. (1) NiT' s strong generalization to unseen resolutions and aspect ratios is primarily enabled by the training resolution diversity. (a) and (b) learn to overfit the fixed-resolution $512\times512$, thus losing the capability to generalize to other resolutions. When trained with native-resolution data, both using absolute PE (Config (c)) and using RoPE (Config (d)) can demonstrate strong generalization capability across multiple resolutions. (2) RoPE can accelerate the convergence speed compared to absolute PE. For both fixed-resolution settings and native-resolution settings, RoPE can improve model performance across multiple resolutions.

Reference

[1] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao et al. "Segment anything." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015-4026. 2023.

[2] Krell, Mario Michael, Matej Kosec, Sergio P. Perez, and Andrew Fitzgibbon. "Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance." arXiv preprint arXiv:2107.02027 (2021).

[3] Sun, Keqiang, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang et al. "Journeydb: A benchmark for generative image understanding." Advances in neural information processing systems 36 (2023): 49659-49678.

[4] Chen, Junsong, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang et al. "Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis." arXiv preprint arXiv:2310.00426 (2023).

[5] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.

[6] Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695. 2022.

[7] anye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740, 2024.