BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities

• I now agree with the philosophy of the authors that this is about fundamental research.
• The authors have presented text-to-image generation with good performance. At the final version, I recommend to add results with further training for the best.
• I appreciate for correcting me about LlamaGen.
• Considering all revision conducted on this paper, I would like to raise my score to '6: marginally above the acceptance threshold'.

Thank you for your valuable reviews! We address the concerns below:

W1: Text-to-image generation

The reviewer’s main concern is that BiGR is trained only for class-conditional generation and not for text-to-image generation.

We strongly agree that text-to-image generation is an important application to image generation models, while we also believe that studying the core class-conditional image generation model is of fundamental importance. The success of existing text-conditional image generation models is largely built on their class-conditional counterparts. This remains a crucial area of research, as evidenced by recent works such as SiT [1], VAR [2], and MAR [3].

Meanwhile, to comprehensively answer the reviewer’s query, we also enable BiGR to perform text-to-image generation by training it on a large-scale image-caption dataset. We train our XL-d24 model using 4M JourneyDB dataset [4] on 32 A800s for 62 hours. We reported our text-to-image generation results in Appendix B of the revised paper.

The results reveal the strong potential of BiGR in text-to-image generation.

Note that, due to time constraints, our training data (4M), model size (< 800M), and training duration (< 3 days) are relatively limited, and we only train a 256$\times$256 T2I model, which has a relatively low image resolution. We believe that scaling up training data and model size, along with exploring advanced techniques such as incorporating higher resolutions, fine-tuning autoencoders, and applying multi-stage training, can further enhance BiGR's T2I performance. We will explore this in future research.

W2: Effectiveness of binary latent code

The success of text-to-image generation demonstrates that the binary latent code can encode sufficient image information without malfunction.

We agree with the reviewer’s viewpoint that the image tokenizer can introduce information loss, which is inevitable for all image tokenizers due to image compression.

The reconstruction FID (rFID) metric is commonly used to evaluate the performance of image tokenizers. We compare our binary autoencoder with the widely used VQVAE employed in LlamaGen. The results are shown below:

These comparable rFID results also demonstrate that the binary code can effectively encode sufficient image information.

The rFIDs of B-AE have been plotted in Figure 4 (left) in the paper.

Besides, autoencoders that produce binary codes have also proven to be effective in works such as Binary Latent Diffusion [5] and MAGVIT-v2 [6].

W3: Fair comparison with LlamaGen

We would like to respectfully clarify some of the reviewer's descriptions about LlamaGen:

1. LlamaGen is an image generation model that handles two modality conditions: class and text. It can perform class-conditional and text-conditional generation, with these models trained separately.

2. Since LlamaGen is solely an image generation model, it cannot perform image discrimination tasks through prompting.

We kindly ask the reviewer to refer to the original LlamaGen paper [7] for more details, which may help verify the points discussed above and address the reviewer’s concerns.

To this end, the comparison between BiGR and LlamaGen is fair. We list the tasks that LlamaGen and BiGR can do:

• LlamaGen: class-conditional image generation, text-to-image generation
• BiGR: class-conditional image generation, text-to-image generation, improved discrimination, zero-shot applications (e.g., inpainting, outpainting, editing, interpolation, and enrichment).

Thus, BiGR can perform more tasks than LlamaGen.

Conclusion

All of the above evidence sufficiently demonstrates that the binary code works well across a broad scope.

We greatly appreciate the reviewer's suggestions, which help strengthen our paper with the addition of the text-to-image generation results.

If the reviewer has additional concerns, we would be happy to discuss them!

[1] Ma et al. SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers. ECCV 2024.
[2] Tian et al. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS 2024.
[3] Li et al. Autoregressive Image Generation without Vector Quantization. NeurIPS 2024.
[4] Sun et al. JourneyDB: A benchmark for generative image understanding. NeurIPS 2023.
[5] Wang et al. Binary Latent Diffusion. CVPR 2023.
[6] Yu et al. Language Model Beats Diffusion: Tokenizer is key to visual generation. ICLR 2024.
[7] Sun et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. https://arxiv.org/abs/2406.06525

Thank you for your review! We address the concerns below:

W1: Benchmark

The benchmarks used in this paper (image generation and linear probing on the ImageNet-1K 256$\times$256 validation split) are widely adopted in the fields of generation and representation learning. The evaluation metrics we report, including FID, Inception Score (IS), sFID, Precision (Pre.), Recall (Rec.), and linear-probe accuracy, provide a comprehensive assessment of the model performance.

We comprehensively compare BiGR with the state-of-the-art generative models in Tables 5 and 9, and the state-of-the-art discriminative models in Tables 6 and 10. We also added the results of SiT [1] in Tables 5 and 9 of the revised paper.

Despite this, we would like to re-emphasize that the goal of this work is to propose a uniform conditional generative model that can produce high-quality generations while maintaining strong representation capabilities. Therefore, surpassing state-of-the-art models across all metrics is not within the scope of this research.

W2: NaN issue with logarithmic operation

We use the alternative method $2 \times |p_k - 0.5|$ to compute confidence in sampling, as described in Appendix A (L750-754). This method functions identically to the original logarithmic operation but avoids the NaN issue. It is robust, and we did not encounter any problems in any of our experiments.

W3: Model exploration

Hyperparameters or architectural variations. We have explored different binary transcoders in Table 2, compared different sampling orders in Table 3, and tested varying sampling iterations and varying diffusion timesteps in Figure 3.

Please see more discussions on hyperparameters in the response to Reviewer jehB (W1). Due to the high training cost, it is infeasible to fully explore different architectural variations.

Adaptability to different tasks or datasets. We have shown our model can perform multiple vision tasks, including inpainting, outpainting, editing, interpolation, and enrichment, as presented in Figure 6. We further adapt our model for text-to-image generation by training it on a large-scale image-caption dataset. We included the results in Appendix B of the revised paper. These results demonstrate the adaptability of our model.

W4: Evaluation metrics

1. Scalability: We have discussed the metrics of FID-50K and linear-probe ACC across different model scales in Figure 4, validating the scalability of our model.
2. Robustness: The FID metric is evaluated with 50K randomly generated images, which demonstrates the robustness of the model. Additionally, our model can handle various zero-shot generalized applications, as shown in Figure 6, further indicating its robustness across different tasks.
3. Efficiency: We have compared models’ inference times in Tables 1 & 3 and Figure 3, which highlights the BiGR’s efficiency.

W5: Application and limitation

BiGR unifies conditional generation and discrimination within the same framework, a scenario not previously explored. BiGR is the first to demonstrate strong uniformity in both generation and representation capabilities for a conditional image generation model.

Additionally, BiGR supports zero-shot applications, such as inpainting, outpainting, editing, interpolation, and enrichment, as demonstrated in Figure 6.

Furthermore, we enable BiGR to perform text-to-image generation, a significant application for generative models.

We have discussed our limitations, including hyperparameter complexity and fixed sequence length, in L537-539. Additional discussions of our limitations can be found in our response to Reviewer jehB (W1&W2).

W6: Theoretical justification

A detailed theoretical justification of the Bernoulli diffusion forward and denoising processes (in our binary transcoder) is provided in [2]. The other components of our model are designed to be intuitive and straightforward. Hence, we do not provide theoretical justifications for them.

We conjecture that the non-deterministic binary transcoder achieves better performance because it enhances the diversity of generated images, leading to improved metrics.

We are happy to discuss any further concerns the reviewer may have!

[1] Ma et al. SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers. ECCV 2024.
[2] Wang et al. Binary Latent Diffusion. CVPR 2023.

W3: More explanation of bidirectional attention

Technically, replacing causal attention with bidirectional attention is straightforward: simply use all-one masks instead of causal masks.

No fine-tuning involved. We would like to clarify that we use bidirectional attention, and train BiGR with masked modeling from scratch. Therefore, there is no fine-tuning stage in training. During training, we simply compute losses for masked positions. The detailed description of how we train our model can be found in Sec. 3.2 in the main paper.

Inference. There are two inference scenarios: generation and representation extraction.

1. For generation, we design entropy-ordered sampling to iteratively unmask tokens from a full mask sequence, as detailed in Sec. 3.3 in the main paper.
2. For representation extraction, we input the full image into the model without any masks and use the intermediate features as the image representation. Please see details in L251-257.

Both inference processes work smoothly with bidirectional attention.

Note that, in both training and inference, the model uses bidirectional attention. Therefore, no special modifications are needed when switching between stages.

W4: Results in LlamaGen paper

We report the results from Table 8 in the Appendix of the LlamaGen paper. These results are conducted under a strict 256$\times$256 resolution setting, which is a fair comparison to our setting and is commonly used by most related papers.

The results in Table 6 of LlamaGen's main paper, which the reviewer may refer to, are based on a setting where the generated images have a resolution of 384$\times$384 and are resized to 256$\times$256 for metric evaluation. LlamaGen does not emphasize this detail in their main paper. More detailed results for the same setting are provided in Tables 9 and 10 in the Appendix of the LlamaGen paper. The results match those reported in Table 6. However, this setting is not a fair comparison, as our model directly generates 256$\times$256 images without resizing.

We kindly encourage the reviewer to cross-check Tables 6, 8, 9 and 10 in the LlamaGen paper.

Thus, the LlamaGen's results reported in our paper provide a fair comparison to ours. We clarified this point in L322-323 of the revised paper.

Q1: SiT

Thank you for pointing out this great work, SiT!

We added this citation at L53 and L140. We included the SiT results in Table 5 and Table 9, and highlighted the state-of-the-art results achieved by SiT among diffusion-based models in L467-468. All changes are reflected in the revised paper.

Q2: Citation format error

Thanks! We fixed the citation format at L196.

We deeply appreciate the reviewer's efforts in helping make our paper more clear and stronger.

If there are any further questions, we are happy to dicuss them!

Thank you for your review! We address the concerns below:

W1: Reasons behind the improved representation capabilities

We attribute the improvement largely to three key design elements of our model:

1. Masked modeling: The masking mechanism uses the bidirectional attention, allowing all patch tokens to communicate with each other. This greatly benefits the integration of the global visual information, which enhances global feature representations across all tokens.
2. Binary diffusion objective: Unlike LLM-style implementations, such as LlamaGen, which project the transformer output into categorical logits, our binary transcoder predicts Bernoulli distributions conditioned on the intermediate transformer feature ($h$). This approach allows the feature to reconstruct element-wise binary codes, rather than being constrained to learn from a fixed codebook. As a result, the feature has the potential to capture richer information by avoiding the limitations of learning within a categorical space.

These two designs are empirically demonstrated as effective in Table 1, and pointed out in L344-346 in the main paper.

3. Intuitively, binary latent codes provide more compact feature space, which can better discriminate visual features. Many works have studied this topic and demonstrate supportive evidence [1,2,3,4].

W2: Improvement with Bernoulli diffusion process and binary codes

The improvement of overall results comes from two perspectives: representation capabilities which have been discussed above, and generation performance.

Regarding generation performance, the Bernoulli diffusion process was first proposed in [5] for image generation, demonstrating that it is one of the most effective methods for modeling binary codes. Using binary codes provides a distinct advantage over encoding images as continuous values, as in VAE, or as discrete indices, as in VQVAE. Traditional image autoencoders face longstanding issues, such as "posterior collapse" in VAE and "low codebook utilization" in VQVAE. Binary autoencoders (B-AE) eliminates the need for codebooks, offering a compact yet expressive representation of images in a binary latent space. Our model is built upon these binary codes.

We provide a reconstruction FID (rFID) comparison between B-AE and VQVAE (used in LlamaGen) in our response to Reviewer FMho (W2). B-AE with code dimensions greater than 20 achieves lower rFID than VQVAE, which highlights the advantages of B-AE.

[1] Cakir et al. Hashing with mutual information. TPAMI 2019.
[2] Jiang et al. Asymmetric deep supervised hashing. AAAI 2018.
[3] Wei et al. A^2-NET: Learning attribute-aware hash codes for large-scale fine-grained image retrieval. NeurIPS 2021.
[4] Wu et al. Deep incremental hashing network for efficient image retrieval. CVPR 2019.
[5] Wang et al. Binary Latent Diffusion. CVPR 2023.

Thank you for your effort and recognition of our work! We address the concerns below:

As the reviewer acknowledged, we have discussed the two limitations in the paper. We would like to elaborate further to provide more insights.

W1: Hyperparameter complexity

BiGR is a novel model, so there is no prior experience to guide hyperparameter tuning. We have conducted extensive experiments to identify optimal hyperparameters. For each hyperparameter, we observe the following patterns:

1. CFG Scale: A larger CFG scale produces smoother images with clearer class features, while a smaller CFG scale enhances fine-grained details.
2. Gumbel temperature: A higher Gumbel temperature increases generation diversity but reduces quality, and vice versa.
3. Number of sampling iterations: 20–30 iterations generally perform well. Using 10 iterations speeds up generation but slightly reduces quality, while more than 30 iterations has minimal impact but slows down generation.
4. Number of diffusion timesteps: 100 steps typically yield good results. The broad range of 10–200 steps has only a marginal impact on performance.

We added these observations in L707-718 in Appendix A of the revised paper. We believe that with our work and continued efforts from the community, BiGR’s hyperparameters can be further optimized.

W2: Fixed Sequence Length

This limitation mainly arises from the need for the binary autoencoder to be retrained for different sequence lengths, requiring BiGR to be retrained accordingly. However, since the transformer architecture can handle varying sequence lengths, we can initialize the training for longer sequences (e.g., 32×32=1024) using a BiGR model pre-trained on shorter sequences (e.g., 16×16=256). This makes retraining more flexible.

W3: Clarification for better flow and easy understanding

Thank you for this valuable suggestion!

We added the clarification that the binary transcoder component is responsible for noise denoising in L207-208 of Bernoulli diffusion paragraph in Sec. 3.1 of the revised paper.

Q1: Quantization artifact

The quantization artifact inherently comes with image autoencoders. It is also introduced by VQVAE/VQGAN.

BiGR does not specially handle quantization artifacts but just follows and predicts the binary latent codes produced by the binary autoencoder. The potential issue of quantization artifacts is mitigated by training on large-scale data, as demonstrated by the generated results.

We provide the reconstruction FID (rFID) of our binary autoencoders, compared to the rFID of VQVAE used in LlamaGen, in the response to Reviewer FMho (W2). This reveals that our binary auto-encoder with code dimensions greater than 20 has lower quantization artifacts compared to VQVAE.

Q2: Priority of the entropy order

Interesting question! To explore this further, we visualize the generated results at different iterations during entropy-ordered sampling. We added this experiment in Appendix C of the revised paper, where the entropy-ordered sampling process is visualized in Figure 9.

We observe that early iterations capture class-level characteristics, while subsequent iterations generate finer object-related details. In the final stages, visual quality steadily improves.

Q3: Failure cases

We added examples of failure cases in Appendix D of the revised paper.

If the reviewer has any further questions, feel free to discuss them with us!