Label-Focused Inductive Bias over Latent Object Features in Visual Classification

Thank you for all your efforts and detailed comments.

Weakness1. The experimental improvements over the baselines are relatively modest, suggesting that the significance of Output-Domain focused Biasing (ODB) may be limited.

Our results show consistent and statistically significant improvement. We hope that the contribution on raising a new problem and its generality are more considered rather than the significance of performance gain determined by an unclear threshold.

Question1. Is it possible for the authors to validate the efficacy of ODB in the context of domain adaptation?

We can validate the efficacy in domain adaptation, but it requires more study, because domain adaptation allows not only tasks using the shared world knowledge, but also task-specific knowledge. We hope that the knowledge of ODB is sufficiently general to work in even target domains for adaptation, but the conflict of task-specific knowledge may dominate the validation results. Unfortunately, recognizing which knowledge causes the conflict in neural networks is still challenging as far as we know.

Question2. Is there a meaningful application of ODB in addressing input-domain bias when dealing with diverse visual domains?

We think that potential applications of ODB are most supervised learning in visual domains where labeling may rely on some implicit knowledge of humans. Particularly, if a domain uses an input containing many latent objects to determine its label, it can be a more suitable application of ODB.

Thank you for all your efforts and constructive comments. We have highlighted the revisions in our paper with purple text color based on the recommendation.

Weakness1. The term of output-domain knowledge is very vague/difficult to interpret. At one point in the introduction you instead write 'undescribed world knowledge', which I think refers to the same thing, and is more clear to me. Would it make sense to use another name than ODB? Specifically to change the OD to something else. This would make the paper clearer. Output domain is the domain where you test, which I think does not really do the job here.

We agree that the term ‘output-domain knowledge’ does not intuitively align with our paper's interest. We revised terminology to avoid using 'OD' as follows

Label-focused inductive bias is the inductive bias determined by categorization of latent objects based on only labels, while undescribed world knowledge indicates all unexpressed world knowledge on the objects in human labeling. We use the term ‘UWK’ for ‘Undescribed World Knowledge over latent objects in human labeling’

We revised related sentences (including titles) to use the terminologies, too.

1. ‘Output-domain focused inductive biasing’ (the method) -> Label-focused Latent-object Biasing (LLB)’
2. ‘output-domain focused inductive bias’ -> label-focused inductive bias
3. implicit relations over latent objects used for human labeling(output-domain) -> undescribed world knowledge

Weakness2. In the related work, a knowledge graph is mentioned briefly while describing the method of one work. However, I miss a small section about the general idea of including hierarchical information or knowledge-graph representations into visual representations, which these authors are not the first to explore. For that reason, it becomes a bit difficult to buy when the authors claim they are the first to raise the issue of the implicit 'output-domain' knowledge missing during training, and a more complete related work section here would make the paper stronger (see for example the related work of Pan et al.)

We appreciate your valuable suggestion, as it notably helps to strengthen our proposal. We added a paragraph and revised other paragraphs in the related work. In the second paragraph (Conflict Reduction via Knowledge Graph) in the related work section, we discuss previous works of using structured information such as knowledge graph in order to reduce input-domain focused bias. It discusses the following references as follows.

• Kenneth Marino, Ruslan Salakhutdinov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2673–2681, 2017.
• Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6857–6866, 2018.
• Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing. Rethinking knowledge graph propagation for zero-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11487–11496, 2019.
• Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems, 35:9125–9138, 2022
• Chenchen Zhu, Fangyi Chen, Uzair Ahmed, Zhiqiang Shen, and Marios Savvides. Semantic relation reasoning for shot-stable few-shot object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8782–8791, 2021.

To the best of our knowledge, previous works focus on the effective use of external knowledge, while undescribed world knowledge over latent objects in human labeling of target training data has not been utilized and even recognized. Please check the related work section for more detailed discussions.

Weakness3. The conclusion and future work section does not really contain recommendations for future work.

In the conclusion of our paper's revised edition, we implemented the following revisions: Beyond using the plug-in model, effectively harmonizing the undescribed world knowledge with input-domain focused bias on a simple network still remains an open question. Also, we hope that our approach will evoke further research on diverse visual domains.

Thank you for all your efforts and thoughtful comments. We have highlighted the revisions in our paper with brown text color based on the recommendation.

Weakness1. Humans benefit from world information being able to avoid input domain bias. This actually benefits from humans having more prior information and being able to build better semantic relationships between classes. While Output-Domain focused Biasing (ODB) is more like a feature enhancement method, which improves performance through decoupling and enhancement of features.

We totally agree with the benefit of human knowledge. However, we hold a differing view that the absence of human knowledge is a weakness. because this paper aimed to improve neural networks and actually intended not to use external resources. This is one of the common goals of AI literature to reduce human efforts in building better AI for the last decades.

Weakness2. Figure 2 shows that the class centroids between some semantically unrelated classes are close to each other. Using triplet loss or contrast loss can also achieve the effect of widening the class centroids distance. It is recommended to add comparative experiments with this type of method.

Thanks for your suggestions.

We would like to emphasize that decoupling is one of the phenomena induced by ODB(revised to LLB). The primary distinction between ODB and traditional contrastive learning resides in the suppression of cluster positioning based on similarity within the input domain. This effect does not occur in methods simply oriented to decoupling representations. For example, final representations generated from visually similar objects should be located in adjacent clusters even if the clusters are decoupled as shown in Figure 4. (In (c), class 0-1 are very close, even overlapped, compared to 0-2. However, in (d), class 2 completely separates class 0 and class 1.) For this reason, ODB more aptly aligns with a method aimed at inducing a bias relatively insensitive to input-domain features, as opposed to a method solely focused on decoupling.

As the reviewer’s recommendation, we added the qualitative analysis of models using contrastive learning on the same data in Section 3.1. Figure 11d shows that the visually similar cluster centroids are not detached as ODB does. The results are briefly shown as follows. Using contrastive learning, the centroids are slightly decoupled compared to supervised learning. However, contrastive learning still fails to widen the gap between ’462: Broom’ and ’746: Puck’, where two class labels are visually similar in stick parts, but semantically distinguished between brush and puck. This observation shows that contrastive learning on CNN can reduce the input-domain focused bias, but still fails in some cases. Please check Section C.2 in Appendix for figures and more details.

Plus, the reason why we do not consider contrastive learning is that SOTA models on ViT do not include explicit loss based contrastive learning.

Weakness3. The paper is not easy to follow, especially the descriptions of Visual Dependency Disconnection and Non-visual Feature Structuring need more details.

To improve the readability, we revised Section 3.2 including ‘Visually Dependency Disconnection’ and ‘Non-visual Feature Structuring’ paragraphs.

The revisions are briefly shown as follows.

1. Notation definition of visual and non-visual features We revised the notation for ‘visual features’ in Section 3.1 and ‘non-visual features’ in Section 3.2.

2. Section 3.2 Visually Dependency Disconnection For clear readability, we defined a notation for the output of the disconnect network as patch-wise non-visual features in the third paragraph of Section 3.2. Also, we added explanations of each notation in Equation (3).

3. Emphasized points To improve the clarity, we generally clarified the emphasized points in Section 3.2. We updated details at main components in our implementation.

Please check Section 3.1 and Section 3.2 for more details.

Weakness4. The proposed method has limited improvement in performance.

Our results show consistent and statistically significant improvement. We hope that the contribution on raising a new problem and its generality are more considered rather than the significance of performance gain determined by an unclear threshold.

Question1. Is the ODB method equally effective in convolutional networks?

The effect of the ODB (revised to LLB) method is basically not relying on specific architecture if its representations are directly affected by loss, so we think it is also effective in convolutional networks. The only reason why we selected ViT is because of its wide use and state-of-the-art performance. To provide more insights, as in Section 3.1 preliminaries, we visualized the distribution of centroids of all output features in each class of ImageNet, extracted from ResNet50 as shown in Section C.1 in Appendix. We found that adjacent classes from Figure 2b, which are visually similar but semantically unrelated, were also closely positioned in CNN (Broom, Puck, Crutch, Mop) (Figure 11c) . We think that the dominance of the input-domain focused bias also exists in CNNs, and therefore ODB can be effective in CNNs. Check Section C.1 in Appendix for more details.