Semantic Memory Guided Diffusion Networks for Image-to-Long Text Generation

Dear reviwer 1tdH,

Thank you for your valuable comments and suggestions, which will help make our paper clearer. We answer your main concerns as follows.

In section 2.2, the initialized matrix contains a series of semantic vectors to cover all possible concepts, but how to get the semantic vectors of these concepts if not mentioned.

In our preliminary experiments, we randomly initialize the semantic matrix, where semantic vectors in the semantic matrix are optimized during training to help the concept prediction process. We will explain this more clearer in the final version of paper.

The statement of the semantic conditional memory is not clear. In section 2.3, the description of “the memory stores the information in aligning image and texts” is ambitious.

The memory networks store a series of memory vectors that contain the image-text matching information between the input image and its long text description. Specifically, we use the stored information through retrieving all memory vectors with the semantic representation (i.e., $\mathbf{h}^s$ computed by Eq. 5), and select the top-$\mathcal{K}$ most related vectors to compute the weighted sum of them to enhance the semantic representation, so that the memory-enhanced representation is able to provide guidance for the diffusion networks to generate the image description with alignment to the input image. To obtain such information, we randomly initialize the memory networks at the beginning of the training process, and the memory networks learn to match the input image and output description until the training is convergence. We will explain it more clearer in the final version of paper.

Although the purpose of the proposed approach is to solve the long text generation problem, I think it is necessary to test on some short caption benchmarks, such as MS-COCO. There have been some other Transformer-based methods [1,2] for image captioning task are proposed. Among them, [1] is also a diffusion-based method... The metric CIDEr is missed, which is a very important metric in the image captioning task.

As you mentioned, it would further enhance our reseach if we evaluate our approach on MS-COCO and compare our approach with SCD-Net and HAAV. Therefore, we run experiments to compare our performance with them on MS-COCO, and illustrate the results (including CIDEr) in the following table (the results of HAAV come from their paper). We will add the resuts in our the final version of the paper.

Our approach obtains better performance than HAAV and SCD-Net. Specifically, for SCD-Net, it uses semantic guidance to enhance the diffusion networks through retrieving related sentence for the generation process. Our approach outperforms them with semantic concepts for description generation, thereby providing more fine-grained guidance to generate elaborated description. For HAAV, it adopts multiple encoders to extract multi-view features from images, so as to integrate these features and provide more informative visual information for description generation. By comparing the integrated features with our semantic concepts, semantic concepts are able to offer more fine-grained guidance for description generation than the integrated visual features from different views.

I think the comparison in Table 2, 3 is not fair. Only SEMDIFF is transformer-based while others are all ResNet-101-based.

For your concerns of vision backbones, our approach uses ResNet-101, which is the same as other approaches (e.g., KiUT, Organ, R2GenRL) on MIMIC-CXR. Therefore, the comparison is fair. For the compared approachs on CC-SBU, LN, COCO-LT, the image captioning approaches use Fast R-CNN with initialization of ResNet's model weights, while LLM-based approaches adopt vision Transformer (ViT), where the compared approaches adopt more powerful vision backbone models than ours. We will expain it clearer in our the final version of paper.

In Table 4, the results of existing state-of-the-art solutions reported are zero-shot or fine-tuned, and SeMDiff is the zero-shot or fine-tuned? If the results of SOTA are zero-shot, how about the fine-tuned performance on these datasets?

In our experiments, the reported results of existing state-of-the-art solutions are evaluated in a zero-shot manner while SeMDiff is trained from scratch. We will expain it clearer in our the final version of paper.

For the fine-tuned performance of existing state-of-the-art approaches, SCD-Net (fine-tuned) is able to obtain 0.0008 BLEU-4 score, which is limited compared to our performance (0.0934).

Dear reviwer FnS9,

Thank you for your valuable comments and suggestions, which will help make our article clearer. We summarize your main concerns as follows.

...can the methods developed for MIMIC-CXR also be efficient on them? The authors did not explore this in the experiments. Also, some LLM are evaluated on them, while it is not sure if they are finetuned on the training sets.

In our preliminary experiments, we have actually tried adapting the methods developed for MIMIC-CXR to the image-to-long text generation task, which only obtains around 0.7 BLEU-4 score on COCO-LT dataset. The reason behind is the domain gap between natural images and medical X-ray images, where X-ray images differ greatly from natural images from both structures and contents.

Also, some LLM are evaluated on them, while it is not sure if they are finetuned on the training sets. As a simple 6-layer transformer encoder-decoder can achieve 0.054 of BL4, it is confusing why the LLM like LLAVA achieves only 0.06 after finetuning. I think this is an interesting point the author needs to explore further in the paper.

In our experiments, we evaluate LLAVA without fine-tuning. For the explanation of similar performance of ``Trans'' to LLM-based approach, we find that LLM-based approaches have possibilities in generating irrelevent texts in the produced image description, thereby obtaining inferior performance on image-to-long text generation datasets.

It is hard to tell if the proposed method is really better than the existing ones. One option here to prove their effectiveness is to adapt the proposed modules to the SoTA method.

We find that existing studies have already obtained high performance on MIMIC-CXR, so that the performance gain of our approach over them is limited. Thus, we further evaluate our approach on LN, CC-SBU, and COCO-LT , where our approach obtains more significant improvement compared to previous approaches. Specifically on CC-SBU, LN, and COCO-LT, our approach obtains 0.109, 0.092, and 0.093 BLEU-4 scores, respectively.

Please note that the improvement on MIMIC-CXR is assessed (t-test) wth significance (at p<0.05 level), and meanwhile, such performance gain is similar in all previous RRG studies and ours seems even bigger than existing studies, which adequately proves its effectiveness, not mention that it also shows consistent and considerable improvements on other datasets.

... Is the semantic concept set the larger the better? Are the predicted concepts the more the better for the long captioning generation?

We run experiments of performance on COCO-LT with respect to the semantic matrix size, which is presented as follows.

Results indicate that the semantic matrix has an optimal value. When the semantic matrix is relatively small, the model gradually obtains better performance as its size enlarges, where the matrix is able to cover more related concepts with more essential semantic information stored with its vectors. Once the optimal value is reached, the model performance starts to degrade when the size keeps enlarging, since larger matrix size does not help in storing extra helpful semantic information.

I am also confused about Fig. 15 and 16. It seems $\mathbf{h}_1$ is a probability of $\mathbf{h}_1$, $\mathbf{h}_2$, ... with other variables. While h is usually used as the "hidden state". I am not sure why $\mathbf{h}_0$ equals the products of a set of probabilities. it seems $p(\mathbf{h}_0|\mathbf{h}^s, \mathbf{h})$ would be more reasonable here. Please correct me if I have some misunderstanding.

We agree with your opinion that h could be improved in writing.$\mathbf{h}_0$ represents the clean bit representation that is totally de-noised from Gaussian noise. We use $\mathbf{h}_0$ as the variables since we regard the intermediate representation of diffusion networks as a type of hidden state. We will fix this in the final version of our paper.

Dear reviwer Hjjf,

Thank you for your valuable comments and suggestions, which will help make our article clearer. We summarize your main concerns as follows.

The definition of in Eq (12) is absent from the document. The cross-entropy loss, denoted as, is not formally introduced.

We follow the standard noising process of denoising diffusion probabilistic model (DDPM) in Eq (12). We will make this clearer and introduce the cross-entropy loss formally in the final version of this paper.

While the authors might perceive some of these notations and concepts as commonly understood within the field, it would enhance the clarity and comprehensiveness of the paper to formally define them. Furthermore, this would enhance the presentation clarity and explain these losses relate outputs from various modules, offering readers a more cohesive understanding of the methodology.

Thank you for pointing this out. We will make all notations and concepts clarified in the final version of our paper.

While the authors have illustrated the effects of diffusion decoding across different timesteps through experiments, the results primarily lean towards a qualitative nature.

For your concerns of performance under different timesteps varying from T=100 to T=1000, we report the quantitative results on COCO-LT as follows.

It is observed that there is a slight improvement as T increases from T=100 to T=1000, since diffusion networks are able to perform a more refined generation process through more iterations. Notably, the greater T also requires more inference time to generate an entire long text description, where we find that generating an image description with around 100 words takes 5 seconds given T=1000. We will add this experiments in the final version of our paper.

Some properties of diffusion models are not studied in the paper. For example, the guidance of diffusion models are known for enhancing the correlation between the generation and the semantic condition for better controllability.

In this work, we follow the standard setup of Bit Diffusion, which does not use any diffusion guidance approach. Nonetheless, we try improving the image-to-long text generation task with diffusion guidance after the paper submission, and have already achieved promising results, which will be released in later studies.

The diffusion models are also known for its computation complexity in the generation, as it requires thousands of NFE in the generation. How does it increase the model computation compared with baselines is not fully studied in the paper.

We agree with you that the computation complexity problem is existed in the image generation task, where diffusion models usually require thousands of NFE to generate images. However, in the text modality that exists in the form of tokens, we draw the opposite conclusion. The computational complexity of autoregressive (AR) models grows quadratically as sentences become longer, which is particularly significant in the image-to-long text generation task. Due to the characteristic of parallel generation of all tokens, the computational complexity of diffusion models is less than that of AR models. We will make this clearer in the final version of our paper.

The configuration regarding the settings of diffusion models is not clearly presented in the paper. How do you choose the diffusion schedule? Do you take DDIM/ODE or DDPM/SDE steps in the reverse process?

We follow the standard setup of diffusion schedule and diffusion sampler in Bit Diffusion [1], which use cosine noise schedule and DDIM sampler by default. We will add this details in the final version of our paper.

Reference: [1] Chen, Ting, Ruixiang Zhang, and Geoffrey Hinton. "Analog Bits: Generating Discrete Data using Diffusion Models with Self-conditioning." arXiv preprint arXiv:2208.04202 (2022).

Dear reviwer 52FQ,

Thank you for your valuable comments and suggestions, which will help make our article clearer. We summarize your main concerns as follows.

The proposed method requires semantic concepts to be obtained from existing datasets. There is no indication that the validation/test set was not used It would be cheating if the method indeed used validation/test set for extracting semantic concepts. There would be strong hints provided to the model.

We would like to clarify the details of extracting the semantic concepts. For the training of SCP, we obtain the semantic concepts based on the following procedures. First, we employ the NLTK POS tagger to annotate POS tags for each word in all image descriptions. Then, we set a threshold to filter out infrequent words. Based on the aforementioned processes, we are able to obtain three semantic concept lists corresponding to the training, validation, and test set. For the concept list of training set, we manually delete the semantic word that appear in the lists of the valiadtion/test set, where we use the resulted list to train SCP. Therefore, we make sure that there is no unfair comparison in our experiments. We will make this clear in the final version of our paper.

Not sure which visual backbone is used. Is it comparable to backbones used by other methods?

For your concerns of vision backbones, our approach uses ResNet-101 as visual feature extractor, which is the same as other approaches (e.g., KiUT, Organ, R2GenRL) on MIMIC-CXR. For the compared approachs on CC-SBU, LN, COCO-LT, the image captioning approaches use Fast R-CNN with initialization of ResNet's model weights, while LLM-based approaches adopt vision Transformer (ViT), where the compared approaches adopt stronger vision backbone models than ours. We will expain it clearer in our the final version of paper.

In 2.3, the word "conditional" is not mentioned at all. What is the point of calling the corresponding component as Semantic "Conditional" Memory?

We call the memory module as "semantic conditional memory", since it enhance the semantic representations from SCP with its stored information, and provides conditional guidance for the diffusion networks. We will make this clear in the final version of our paper.