Towards Multimodal Open Set Recognition

1. There lacks clear and sufficient analysis about the fusion degradation challenge. In-depth insights about this challenge should be discussed and confirmed.
2. The technical novelty is limited. From the technical idea of the multimodal fusion, I could not learn more about new knowledge. Cross-attention and mixure of experts are very general strategies proved to be effective. However, the proposed method is simple application of such techniques. Therefore, it is unclear why and how these techniques overcome the challenge of fusion degradation.
3. Figure 1 may have no useful information about the proposed method, and even this figure is removed.
4. As figure 3 shows, only two modalities are considered in method design, which may not be general.
5. MoE will cause more computation, which should be taken into account in method selection.

1. The model is overly simplistic. The innovative point of the MRN's adaptive fusion module merely employs an expert mixture model.
2. When dealing with reject samples (belonging to unknown classes), the method uses softmax values less than the parameter τ, but does not provide a basis for selecting this parameter. If it is a hyperparameter, how was it determined? If it is a learnable parameter, how is it learned?
3. In the experimental setup section, the experimental environment and other parameters are not specified.
4. In the experimental results, for the open-set multimodal tasks proposed in this paper, the advantages of CSSR-MRN are not significant for most datasets.

While in general the results indicate reasonable improvements, there are some points that need to be resolved before the paper is of sufficient quality to be published:

1. The structure of the document and the description of the proposed method would enjoy improvements:

   a) The organization of the paper can be improved. It is not clear why results are presented in section 3, before discussing details of the method and the evaluation?

   b) For the preliminary experiments in section 3.2, what are the models used to extract features for text and image modalities? What does it exactly mean to "add[ing] representations from both modalities"? If this is this simple element-wise addition, why would this be a reasonable fusion strategy? Why not using score fusion techniques which are typically better suited for MM fusion? What are the two variations Fusion and Fusion-OSR reported in table 1?

   c) Section 4.3 proposes a specific evaluation procedure, but such a procedure is not used in their evaluation. Why do the authors talk about such standard evaluation?

   d) There are many results in table 3 that are neither described nor discussed in the text. Particularly, the final block appears to show variations of the proposed method, which are explained nowhere.

   e) The role of the visualizations in figures 7 and 12 is unclear. What do the authors expect to see? Are these representations for known or unknown samples? To me, all visualizations highlight reasonable locations, I cannot follow the conclusion drawn by the authors that "representations of MRN are more accurate and comprehensive than the baseline". The authors should consider sacrificing these plots (or move all of them to the supplemental) and use the space to discuss their results better.

   f) There is quite a lot of additional information in the appendix, which is not mentioned anywhere in the main paper. The authors should add indications into the main text, and discuss why we see such supplemental material.

   g) In algorithm 1 (in the supplemental material), it is not clear which parameters are included into the set \theta. Specifically, are the parameters of the encoder networks also included in these parameters, or are pre-trained networks used?

2. The evaluation needs to be improved:

   a) The conclusions drawn for the initial experiments are solely valid for the applied OSR and MM methods and might not transfer to other methods. The authors need to acknowledge this and tone down claims such as "open space risk [...] cannot be resolved by simply integrating OSR methods".

   b) Evaluation is based on ACC, AUROC and AUOSCR, all of which are mainly influenced by improved closed-set classification. This is perfectly shown in figures 4 and 5 where AUROC and OSCR values mostly correlate with ACC. A better evaluation would plot OSCR curves [Dhamija2018] which incorporate closed- and open-set evaluation into a single curve. Also, OpenAUC (Wang et al., 2022) discusses why ACC and AUROC are not good metrics to evaluate OSR.

   c) The employed Max-probability baseline OSR technique is very simplistic and not SOTA. For a similarly simply, but much more effective, OSR technique, the authors should rather use the MaxLogits technique [Hendrycks2022].

   d) The ablation study in section 5.3.2 only varies the number of experts in between 10 and 20. It misses showing that the module itself provides improvements over not at all using a mixture of experts. Such results are only presented in the supplemental material.

   e) The visualization of high-dimensional features is done via t-SNE, which has been superseded by UMAP. The authors should consider to use UMAP as well. Representations are not "downscaled", but "dimension-reduced".

3. The mathematical details need to be improved:

   a) In (1), why is the same value z_2 input to function C_1 twice?

   b) Above (2), a value \epsilon is defined, but not used in (2).

   c) Below (2), it is mentioned that the logits f are the output of Softmax, while by definition logits are the inputs to Softmax.

   d) In (3), g_I and g_L are used without providing equations on how these values are computed.

   e) The number of experts is called E in section 4.2.2 and N in section 5.2.

[Dhamija2018] A. R. Dhamija, M. Gunther, and T. E. Boult, "Reducing network agnostophobia," in Advances in Neural Information Processing Systems (NeurIPS), 2018.

[Hendrycks2022] D. Hendrycks et al., "Scaling Out-of-Distribution Detection for Real-World Settings," in International Conference on Machine Learning (ICML), 2022.

(1). This paper illustrates the issue of fusion degradation after modality fusion using only visualization in Figure 2, lacking theoretical analysis and proof.

(2). The proposed MRN framework lacks novelty, as it merely integrates existing methods, i.e., Cross Attention and MOE, without substantial innovation.

(3). Compared to existing multimodal fusion methods (e.g. MLA), the improvements in various OSR task metrics presented in this paper are not sufficiently significant.

(1) There is another works claiming the importance of the multimodal open-set learning. Could the author talk about the difference between this work and the reference [1]? (2) We can find some improvement comparing the visualizations from this work and existing works on the dataset Food-101 (Fig. 2 & Fig. 6). However, the performance in Flower-102 seems not good. Could the author show some results of visualizations of existing methods? (3) Could the author analyze the performance on CREMA-D which is not good.

[1] Towards Multimodal Open-Set Domain Generalization and Adaptation through Self-supervision

• Experiment in Table 1: The evaluation in Table 1 appears to be somewhat unfair. Using simple summation to fuse representations from different modalities may not be effective, as the embeddings from two distinct models (modalities) are not necessarily aligned. It would be useful to see results with a simple concatenation of embedding vectors as an alternative.

• Line 240: The writing style could be improved, particularly in the phrase "modality 1 data x_1."

mutually enhanced fusion - A major writing issue is that while the notation is presented for two modalities for clarity, it is not clear how the "mutually enhanced fusion" module would generalize when there are more than two modalities. Pairwise enhancement for an arbitrary number of modalities may be inefficient and needs further clarification.

• Addition of Gaussian Noise (Line 286): It is unclear how adding Gaussian noise with zero mean and unit variance improves robustness. More explanation would help in understanding the rationale behind this choice. (no theoretical or experimental support or reference )

• Results in Figures 4 and 5: The results show that the performance is not sensitive to the number of experts. It would be insightful to see the distribution of weights assigned to each expert, to determine if the model is selectively using experts or simply averaging across them. Additionally, it is important to visualize that no single expert is consistently dominant across different scenarios.