Advancing Multimodal Unified Discrete Representations

1. Novelty: After extracting features with existing models, the proposed method combines the existing CLUB and CPC frameworks and VQVAE to construct its architecture. The idea of selecting informative dimensions seems to follow APE (Zhu et al., 2023). The proposed TOC follows the similar idea of using inter-class distances and intra-class variances to reduce dimensions as in LDA (linear discriminant analysis). Therefore, the novelty of the proposed method seems to be limited.
2. Experimental evaluations: The proposed method is evaluated on audio and video retrieval tasks as well as conditional image generation tasks on specific datasets. However, there are only a few comparison methods. Especially for generation tasks, the proposed method is mainly compared with VQ-VAE. At present, most state-of-the-art methods are based on diffusion models. Therefore, the experimental evaluation is insufficient, and experimental comparisons with more state-of-the-art methods are needed to demonstrate the effectiveness of the proposed method.
3. In terms of ablation studies, the authors verified the effects of different modules. However, most of these modules already exist.
4. The paper tests different quantities of embeddings in the codebook, but since the focus is on leveraging certain effective features from the embeddings, it would be better to include corresponding evaluations on different embedding dimensions.

1. I believe that the content of the related work section should ideally be included in the main text rather than the appendix, even though I understand there are page limitations.
2. In Equation 9, $sg$, $\phi_k^m$ are not explained.
3. The calculation of $U_k$ in Equation (9) and how it is updated require further clarification.
4. Some obvious formatting errors need to be corrected, such as the missing letter 'n' in "Sectio3.1" on line 125 and the missing closing parenthesis in the expression $p(x,y$ in Equation 2. Besides, in line 428, the ‘Count’ should be `Count’.
5. How to obtain $o_m^t$ needs to be clarified in line 174.
6. The article's innovation does not seem very strong. Compared to the method by Xia in the 2023 NIPS, it only adds a text modality and a coarse-grained contrastive learning approach. It seems that only the proposed TOC is novel.

1.In terms of writing, the symbols in Figure 1 cannot be displayed on some PDF readers

2.Lack of innovation, the similarity between the two methods and previous work like DCID and APE is too great

3.Lack of an algorithm description to assist in understanding the entire training process

• Clarity and Self-Containment: Many sections require additional elaboration to make the paper self-contained and accessible to readers:

  • Equation 2: The distinction between the two representations beyond using different encoders is unclear. More detail on the objectives and the roles of each representation is needed. For instance, why one representation is model-general and the other model-specific, despite both using the same input data. Further clarification is also needed on whether the specific encoders are pre-trained or jointly trained with the rest of the model.
  • Equation 3: The transformation from $q_{\theta}(y|x)$ to $q_\theta(\bar{f}_i^m |{f}_i^m)$, along with the second part of the transformation, is hard to parse. The same question persists for Equation 6.
  • Equation 4: It is unclear how Equations 3 and 4 integrate into the final learning objective and why Equation 4 relies solely on model-general features rather than including model-specific ones as well. Equation 7 raises similar questions.
  • Combination of Losses in Equation 10: Additional detail on how the losses are combined, the purpose of each, and whether hyper-parameters are used or if they are equally weighted would enhance understanding.
  • (Minor) There is a missing parenthesis in Equation 2 in $p(x,y)$.

• Experimental Design:

  • Uni-Modal Baselines: For non-retrieval methods, including uni-modal baselines would help readers assess the standalone performance of methods that focus exclusively on one modality.
  • Confidence Intervals: Confidence intervals for the results would help quantify the benefits of TOC, which are currently difficult to justify without them, especially in Tables 1, 2, and 6.
  • Codebook Size: The authors mention that increasing the codebook size beyond 400 degrades performance, but more discussion is needed to clarify why this occurs.

1. The contribution of this paper is very limited. A very serious problem is that this article is very similar to article [1] published in NeurIPS 2024, and the similarities include language expressions, symbols, formulas, and so on.

[1] Achieving Cross Modal Generalization with Multimodal Unified Representation, NeurIPS 2024.

2. Considering the first of the Weaknesses, the only contribution of this paper is the TOC. the authors describe in Section Introduction that one of the innovations of this paper is that the TOC does not require additional training, however, in Section 3.2, the authors do give the need for the TOC to solve the optimization problem Equation (13).

The writing should be improved. For example, “Specifically, FCCID, TOC, and their combination outperformed before SOTA by 2.16%, 1.06%, and 2.96% respectively, on four downstream tasks.” Whether these percentages represent average improvements across the four tasks? Besides, some writing errors need to be corrected, like line 39: has-> have, in line 125, Sectio -> Sectio, etc.

In line 83, could the authors point out what the relationship between “the inherent constraints of the codebook and the quantization method based on Euclidean distance.” and “treats all feature dimensions equally”, it’s rather unclear about the relation here.

Is lamda a hyper-parameter? How is the lamda value chosen? It’s better for the authors to include a hyper-parameter investigation in their manuscripts.

“TOC reduces training parameters and time”: It could be better for the authors to include a table or figure comparing the number of parameters and training time with and without TOC across different models or tasks.

It’s better for the authors to re-organize the layout of some figures and tables and introduce their related works in the main text for better review. Like using some techniques such as wrap figure in latex.