I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models

Reviewer fnpq

We thank reviewer fnpq for the insightful comments and suggestions. We address major concerns as follows and will add additional literature in the reference.

Q1: Ablation on different VLMs.

We use Qwen2-VL-7B which supports interleaved image and text inputs as the LVLM. The table below further ablates ThinkDiff-LVLM's performance with different LVLMs. InternVL2.5-8B achieves a worse performance compared to Qwen2-VL-7B, indicating that a stronger LVLM can improve the alignment and accuracy. Moreover, with a more powerful LVLM Qwen2-VL-72B, ThinkDiff achieves a new SoTA on most tasks.

Q2: Data scale.

We double the sample number to 3.4M to include more diverse datasets. We train a new model for the same steps. The table below shows with more data, our model can generally improve the results in most tasks.

Q3: Contrastive loss.

In Q1 and Q2 of Reviewer 37KH, we use the contrastive loss similar to the ImageBind. It has some disadvantages compared to our method and is inferior in accuracy. However, it still shows potential to effectively combine the explicit contrastive alignment and our implicit alignment in future work.

Q4: Real-world application.

Our method not only can handle the tasks defined by the CoBSAT benchmark but also can handle different general tasks. To demonstrate this point, we further evaluate our model on general generation capabilities on other benchmarks, i.e., COCO, GenEval, and DPG-bench. Our method shows clearly better results compared to other competitors. Please refer to the COCO table in Reviewer BZ6o Q1, the GenEval, and the DPG-bench tables in Reviewer j31i Q3.

Q5: Other multimodal fusion techniques.

The Flux-pro actually uses a fusion method similar to IP-Adapter which injects image features by attention. As shown in Paper Figures 6, 11, and 13 in the main paper, our method clearly shows advantages in coherently composing different multimodal instructions.

Q6. Formal analysis of feature space alignment.

The alignment quality of ThinkDiff is implicitly evaluated by the accuracy and consistency of the reasoning benchmark and composing benchmark. Since our alignment is implicit, it is important but not very straightforward to directly analyze the feature space. To investigate the theoretical derivations of the alignment, we may rely on the development of vision-language training in VLMs. We humbly leave this for future work.

Q7: Other multimodal reasoning tasks.

In this paper we mainly target multimodal generation tasks. The LVLM in our model supports captioning or VQA tasks but the whole method itself does not target them. However, our alignment is a general alignment method and can be possibly applied to other multimodal tasks. This is a very interesting direction for future research.

Q8: Other diffusion models.

Once aligned, our ThinkDiff can be applied to other models that use T5 as encoder in our experiments. For example, beyond FLUX, we also applied ThinkDiff to CogvideoX in Appendix Figure 14, where a coherent video is generated by seamlessly integrating images and text. This demonstrates ThinkDiff's flexibility. Extending ThinkDiff to even more diffusion models is straightforward and is left as future work due to the tight schedule.

Q9: Failure cases.

This anonymous link (https://anonymous.4open.science/r/anonymous-4DF1/failure_case.png) shows two failure cases on CoBSAT. We think the main sources are from two aspects. One is the imperfect of LVLM which gives wrong reasoning, such as the "cow in the desert" case. Using a more powerful VLM can address this problem. The other is the imperfect alignment that does not accurately condition diffusion models, such as the second "white apple" case. We think a possible way to effectively address this problem is to include high-quality datasets for end-to-end training.

Reviewer j31i

We thank Reviewer j31i for the insightful comments and suggestions. We provide our responses as follows and will add additional literature in the reference.

Q1: Why alignment?

There are several advantages in terms of the alignment compared to text-based methods on the multimodal generation task. We elaborate on them below:

1. An important aspect of multimodal generation is to capture multimodal information in the generator. The aligner in our method is differentiable. It can convey detailed multimodal information that cannot be expressed by texts effectively. However, the text-based methods discard the detailed multimodal information in the middle.
2. The model is an end-to-end model. The submodels in it can be optimized together to benefit each other.
3. The features passed through the aligner are more representative and can convey more dense spatial context information than sparse texts.

Q2: Text-based evaluation.

The CoBSAT benchmark mainly focuses on the correctness of logical reasoning. To better validate the benefits of the alignment that can convey more multimodal information, we evaluate the image-condition generation of ThinkDiff-LVLM on COCO, as shown in the table below. The input of ThinkDiff-LVLM is an image. The input of the text-based method is the texts generated by the LVLM. As shown in this table, the alignment performs better on the CLIP image metric (CLIP-I) and the FID score. This experiment further validates that with the proposed differentiable aligner and dense representations, more multimodal information can be passed to the diffusion decoder, which is important for multimodal generation tasks.

Q3: Evaluation benchmarks.

We evaluate on more datasets such as the COCO, GenEval and DPG-Bench.

The GenEval:

The DPG-Bench:

The COCO results are in Q2 and Reviewer BZ6o Q1.

Compared to the previous arts, ThinkDiff-LVLM achieves notable gains in all benchmarks.

Reviewer BZ6o

We thank Reviewer BZ6o for the insightful comments and suggestions. We provide our responses below.

Q1: Generation quality.

We evaluate the general image-conditioned generation of ThinkDiff-LVLM on 1k images in COCO. The models are conditioned by an image in the experiment. We show the FID, the CLIP image metric (CLIP-I), and the CLIPScore (CLIP-T) in the table below. Our method can achieve much better performance than existing competitors such as SEED-LLaMA, Emu, and GILL.

Q2: Inference time.

ThinkDiff-LVLM replaces the T5 encoder in Flux with an LVLM. The LVLM model is highly optimized in the community framework such as vLLM and adds minimal latency to the inference. Qwen2-VL-7B model takes less than 0.2 seconds for one prompt that consists of both images and texts. Although the T5 encoder (typically costs 0.05 seconds for one prompt) is faster, they both are a marginal overhead compared to the diffusion process of Flux (typically over 2.5 seconds).

Q3: ThinkDiff-LVLM vs ThinkDiff-CLIP and their evaluation.

ThinkDiff-LVLM can handle different tasks including general image-conditioned generation, similar to ThinkDiff-CLIP. As shown in Q1, we evaluate ThinkDiff-LVLM on the image-conditioned generation task. Our method obtains better CLIP-I, CLIP-T, and FID scores compared to SEED-LLaMA, Emu, and GILL. This shows the general applicability of ThinkDiff-LVLM beyond the reasoning.

On the contrary, ThinkDiff-CLIP is limited by the CLIP image encoder and the T5 encoder. Therefore, in our experiments in the main paper, we already observed that ThinkDiff-CLIP cannot handle very complex logical questions as ThinkDiff-LVLM does, but it has strong capabilities for multimodal composing.

The above additional evaluation further shows their respective strengths and weaknesses.

Reviewer 37KH

We thank Reviewer 37KH for the insightful comments and suggestions. We provide our responses below.

Q1. Experiment with Imagebind.

We conduct an experiment with ImageBind-style alignment to align the LVLM decoder and the T5 encoder. The input of LVLM is an image and a text prompt. It generates token features and text. The input of the T5 decoder is the LVLM's generated text. Instead of using only one token similar to the original Imagebind, we extract 32 semantic tokens from both LVLM and T5 to compute the alignment loss. This gives stronger capacity to the alignment. As shown in the table below, our ThinkDiff achieves significant improvements on the CoBSAT benchmark over the ImageBind-style alignment.

Q2. Advantages over ImageBind.

While the contrastive alignment in Imagebind shows promising results in multimodal alignment, ThinkDiff introduces more advantages:

1. ThinkDiff exploits the capabilities and knowledge of the LVLM decoder and the LLM (T5) decoder for the alignment, which enables data-efficient and training-efficient alignment, and has been validated in vision-language research areas. On the contrary, Imagebind needs web-scale datasets to train from scratch.

2. ThinkDiff provides a fine-grained and flexible alignment method. Since it does not compute element-wise distances, the tokens of different modalities can have different lengths. This also supports training strategies such as masked training. The alignment is not element-wise minimization but is the high-level semantic alignment.

3. ImageBind typically uses one token to compute the token distances, which is a coarse alignment. ThinkDiff excels at fine-grained alignment for longer token sequences. Even if using more tokens in ImageBind, the lengths of two types of tokens must be the same.

Therefore, as shown in the experiment in Q1, our method can achieve much better accuracy compared to the ImageBind-style alignment.