KEFI: Kernel-based Feature Identification for Generalizable Classification

We sincerely appreciate your thorough and insightful feedback. Below, we outline our efforts to address the valuable points you have raised.

 

The discussion on related works is indeed insufficient. Basically, this method focuses on learning interpretable representations with explicitly decomposed components. However, there are works that also consider the decomposition or encoding of attributes from different perspectives, e.g., information bottleneck [r1] and rate reduction [r2].

Reply:

In response to [r1, r2], both methods are centered on learning invariant representations, which stands in contrast to our approach that focuses on capturing a broad spectrum of features from source domains. Specifically, [r1] promotes the learning of invariant representations by integrating the invariant risk constraint of IRM and a sparsity constraint into the objective function, through the lens of mutual information, a concept they term the Invariant Information Bottleneck (IIB) principle. On the other hand, [r2] aims to find an optimal (invariant) linear discriminative representation of data by optimizing the rate reduction objective.

 

Discussion on related works:

In term of feature decomposition, It is difficult to find research with analogous characteristics to our method:

• (i) capture a wide range of attributes from source domains
• (ii) provide a machanism for Attribute-Identifiability, essential for discerning if an attribute in the target domain has been previously encountered in the source domain.

While existing methods e.g, [1,2,3] in the literature aim to learn a variety of features, they generally utilize all learned attributes uniformly across various target domains, without distinction.

 

In term of kernel-based method for DG, we have expanded the related work section to include additional kernel-based methods, highlighted in blue, and provided a discussion contrasting our approach with these previous methods as follows:

Diverging from these existing methods, our proposed approach also falls under kernel-based methods but takes a distinct route. Unlike traditional techniques that utilize the kernel as a tool for measuring divergence or similarity and rely on the kernel trick to avoid explicit computation of the feature map, our method takes a direct approach. We compute the feature map through a random finite-dimensional feature map. This enables us to leverage a clustering-based perspective of kernel methods, which is instrumental in characterizing and refining representations for effective domain generalization.

 

[1] Huang, Zeyi, et al. "Self-challenging improves cross-domain generalization." ECCV 2020

[2] Bui, Manh-Ha, et al. "Exploiting domain-specific features to enhance domain generalization." NIPS 2021

[3] Chattopadhyay, Prithvijit, , et al. "Learning to balance specificity and invariance for in and out of domain generalization." ECCV 2020

 

In the methodology part, only the kernel-based feature selection procedure is presented, while the justification and in-depth analysis are also necessary. Specifically, compared with other methods that also aim at learning attributes, what are the limitations in the existing methodology? Correspondingly, what are the advantages and weaknesses of the proposed method in learning components?

Reply:

Our approach includes a mechanism for Attribute-Identifiability, which is crucial for determining whether an attribute in the target domain was previously present in the source domain. However, it's important to note that this is not solely a kernel-based feature selection process.

To elaborate further, we would like to offer additional insights into the fundamental aspects of our methodology:

• Attribute-based representations are often expected to be easily interpretable and meaningful to the users of the model. Typically, learning and evaluating such models necessitates specialized domain knowledge. In our research, we reduce the need for such in-depth expertise by aligning our work with a specific downstream task: classification, as outlined in Definition 1.

• We have developed a thorough framework for attribute identification, framing it as a kernel-based optimization problem (Section 3.4). In particular, we utilize the clustering perspective of kernel methods for attribute identification. To the best of our knowledge, no prior studies have applied the cluster-induced viewpoint of kernel methods at the attribute level for addressing domain generalization. Our work is pioneering in exploring this novel approach.

• Beyond just identifying bases of attributes, our sophisticated formulation also promotes the distinctness of attribute clusters (Remark 1). This approach effectively enhances the discriminative power of the attributes we learn, contributing to the overall robustness and efficacy of our model.

We sincerely appreciate your thorough and insightful feedback. Below, we outline our efforts to address the valuable points you have raised.

 

Confused about the basis of attributes?

Reply:

In our study, we initially segment the latent space into $K$ groups, where all attributes (sub-latent vectors) within a single group form a basis. Typically, attribute-based representations are generally presumed to be interpretable and meaningful for the end-users of the model. Typically. Learning and evaluating such models require domain-specific knowledge. In our work, we mitigate this requirement for domain expertise in attribute-representation by focusing on a downstream task, specifically classification, as follows:

"A basis of attributes in relation to a given dataset is defined in terms that the attributes within a basis should be capable of predicting the corresponding class-labels for all samples in the dataset."

Formal definition:

(Basis of attributes): For a dataset $\mathbb{D}=\\{(x^i,y^i)\\}_{i=1}^N$ and feature extractor $g$, a set \mathcal{A}\_{k}=\\{ a^1_k,a^2_k,...,a^{N}_k \\\} where each $a^i_k=g(x^i)_k$ is considered as basis of attributes with respect to $(\mathbb{D},g)$ if there exists a function $h_k$ such that $h_k(a^i_k)=y_i, \forall i=1...N$.

This constraints can be obtained by optimize objective function in Equation.(3): training attribute-based classifiers in source domains.

 

How could you find the basis of attributes by formulation (3) without any attribute labels?

Reply:

We formulate the attribute detection in the target domain as an outlier detection problem.

We start with cluster induced representation of kernel method. The hyperplane on kernel space when projecting back into the original attribute space, resulting in a series of contours that envelop the target attribute. These contours serve as cluster boundaries, with points within each distinct contour being assigned to the same cluster. Additionally, it is found that these contours effectively outline the support of the underlying probability distribution, essentially highlighting high-density regions in the distribution of target attribute. This characteristic is particularly valuable for outlier detection.

During the training process, as we divide the latent space into $K$ groups, all attributes within the same group constitute a basis. To learn the cluster boundaries (hyperplane) $\Gamma_k$ for a basis, we treat attributes from other bases as negatives.

Referring Section 3.4:

Specifically, the max-margin hyperplane $\Gamma_{k}$ is built in such way that: (i) it separates push-forward kernel-features of positive attributes $C\_{+k}=\\{\phi\_{\sigma}(a) \\mid a\in\mathcal{A}\_{k}\\}$ and the negative attributes $C\_{-k}=\\{\phi\_{\sigma}(a)\mid a\in\mathcal{A}\_{i\neq k}\\}$ and (ii) the margin w.r.t the hyperplane $\Gamma_{k}$ defined as the closest distance from negative in $C_{-k}$ to this hyperplane is maximized.

It is important to note that our hyperplane in the kernel feature space is integrated with the encoder $g$. Hence, optimizing hyperlanes on kernel space also updates representation of attributes. This integration allows for a more compact refinement of attribute clusters from the basis $A_k$, while also efficiently distinguishing them from attributes derived from other bases.

Upon completion of training, we obtain a set of hyperplanes $\\{\Gamma_k\\}_1^K$, which function as tools for outlier detection. These hyperplanes help in determining whether an attribute in the target domain was learned in the source domain.

 

How to find K (K group), by tuning? If yes, the ablation study on K is missing

Reply:

In current work, K is Hyper-parameters. We have provided ablation study with different value of $K$ in Section 4.3.2.

 

Why are the dimensions of attributes equal? An explanation or proof is missing

Reply:

In our study, we adopted a simple and effective strategy for dividing the latent vector, aimed at facilitating a fair comparison. This method entails segmenting the latent vector into several sub-features, while consciously avoiding the introduction of extra modules or transformations that could modify its fundamental structure. For ease of implementation, we opted to split the latent vector into attributes of equal dimensions.

This discussion has been added into Section 8.1: Implementation Details in the revised version (highlighted in blue).

The paper does not report the experimental results of the proposed method in identifying whether an attribute appears in the target domain or not. Therefore, the motivation and claimed effectiveness of this method is difficult to prove.

Reply:

As kernel-models (hyperplanes) $\\{\Gamma_k\\}_1^K$ characterize high-density regions in the distribution of target attribute, our model identify whether an attribute in the target domain was learned in the source domain based on set of kernel-model $\\{\Gamma_k\\}_1^K$ in the sense that new attribute in target domain is lied on high-density regions in the distribution of learned attributes from source domain or not. However, we lack ground-truth labels for qualitative assessment.

To validate this, we use t-SNE for visualizing the distribution of learned attributes, as shown in Figure 4. Our analysis focuses on:

• Photo domain which contains rich information to predict the label in comparison with three other domains as the target domain. Hence,It is assumed that the target domain in this case may draw upon information from all source domains.

  From (Figure.4. Right), we observe that the photo domain effectively utilizes attributes from all three source domains, as most of its attributes are selected.

• Sketch domain which consists solely of colorless images. This choice is intentional, as the Sketch domain is markedly different from the others, leading to a more pronounced divergence between source and target.

  In comparison to the Photo domain, the Sketch domain (as shown in Figure 4, Right) exhibits a smaller number of selected attributes, reflecting its unique characteristics and distinctiveness from other domains. urthermore, it is noticeable that the selected attributes (marked in Yellow) fall within the region of the source domains, while the deselected attributes (indicated in Black) are positioned outside the source domains' region.

We have clarified it in Section 4.3.1 (highlighted in blue)

 

How does the method in this paper perform in a multi-source domain setting? The experimental results in a simple single-source domain setting are not sufficient to prove the superiority of the approach.

Reply:

We showcase the performance of our method in multi-source domain settings (using the well-known DomainBed benchmark) in Table 1, and in a straightforward single-source domain scenario in Table 2.

 

Why are there no up-to-date state-of-the-art methods for comparison, the methods reported in the paper were published in 2021 and before. Additional results with the latest methods are needed to demonstrate the validity of the proposed methods.

Reply:

In this work, our primary interest lies in exploring the properties of kernel-based attributes and their advantages, rather than pursuing state-of-the-art performance.

Recent approaches, such as [1, 2, 3, 4], frequently adopt an ensemble strategy, particularly one based on SWAD [1]. This approach complicates the evaluation of the effectiveness of their latent representations. As a result, our research primarily focuses on methods that rely on a singular model for evaluation. Nonetheless, to provide a comprehensive view, we have broadened our related works section to include these methodologies. Additionally, we have conducted extra experiments where our KEAI model is trained using an ensemble strategy similar to SWAD, as shown in the following table.

These experiments reveal that KEAI not only performs competitively when compared to state-of-the-art (SOTA) methods but also shows notable effectiveness in domains with limited information, such as the Cartoon and Sketch domains.

 

Table 1: Classification Accuracy on PACS with ResNet18

[1] Pcl: Proxy-based contrastive learning for domain generalization, CVPR2022.

[a] Junbum, Cha et al. (2021). “SWAD: domain generalization by seeking flat minima.” In: NeurIPS.

[b] Xu, Chu et al. (2022). “DNA: domain generalization with diversified neural averaging.” In: ICML.

[c] Kim, Jaeill, et al. "VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution." C. PRV 2023.

We sincerely appreciate your thorough and insightful feedback. Below, we outline our efforts to address the valuable points you have raised.

 

Contributions and novelties of this paper compared to previous studies are not explicitly presented in the paper. The novelty of this paper is limited since the kernel-based methods have been widely studied.

Reply:

We have expanded the related work section to include additional kernel-based methods, highlighted in blue, and provided a discussion contrasting our approach with these previous methods as follows:

Different from these existing methods, our proposed approach also falls under kernel-based methods but takes a distinct route. Unlike traditional techniques that utilize the kernel as a tool for measuring divergence or similarity and rely on the kernel trick to avoid explicit computation of the feature map, our method takes a direct approach. We compute the feature map through a random finite-dimensional feature map. This enables us to leverage a clustering-based perspective of kernel methods, which is instrumental in characterizing and refining representations for effective domain generalization.

We would like to further clarify the key aspects of our contributions as detailed below:

• Attribute-based representations are often expected to be easily interpretable and meaningful to the users of the model. Typically, learning and evaluating such models necessitates specialized domain knowledge. In our research, we reduce the need for such in-depth expertise by aligning our work with a specific downstream task: classification, as outlined in Definition 1.

• We have developed a thorough framework for attribute identification, framing it as a kernel-based optimization problem. In particular, we utilize the clustering perspective of kernel methods for attribute identification. To the best of our knowledge, no prior studies have applied the cluster-induced viewpoint of kernel methods at the attribute level for addressing domain generalization. Our work is pioneering in exploring this novel approach.

• Beyond just identifying bases of attributes, our sophisticated formulation also promotes the distinctness of attribute clusters (Remark 1). This approach effectively enhances the discriminative power of the attributes we learn, contributing to the overall robustness and efficacy of our model.

 

The approach in this paper first performs attribute representation learning and then utilizes a kernel-based approach to categorize the attributes. So, how does the method determine whether an attribute is present in the target domain or not since the training process does not include a structure to process the attribute's origins?

Reply:

We formulate the attribute detection in the target domain as an outlier detection problem. We start with cluster induced representation of kernel method. The hyperplane on kernel space when projecting back into the original attribute space, resulting in a series of contours that envelop the target attribute. These contours serve as cluster boundaries, with points within each distinct contour being assigned to the same cluster. Additionally, it is found that these contours effectively outline the support of the underlying probability distribution, essentially highlighting high-density regions in the distribution of target attribute. This characteristic is particularly valuable for outlier detection.

During the training process, as we divide the latent space into $K$ groups, all attributes within the same group constitute a basis. To learn the cluster boundaries (hyperplane) $\Gamma_k$ for a basis, we treat attributes from other bases as negatives.

Referring Section 3.4:

Specifically, the max-margin hyperplane $\Gamma_{k}$ is built in such way that (i) it separates push-forward kernel-features of positive attributes $C\_{+k}=\\{\phi\_{\sigma}(a)\mid a\in\mathcal{A}\_{k}\\}$ and the negative attributes $C\_{-k}=\\{\phi\_{\sigma}(a)\mid a\in\mathcal{A}\_{i\neq k}\\}$ and (ii) the margin w.r.t the hyperplane $\Gamma_{k}$ defined as the closest distance from negative in $C_{-k}$ to this hyperplane is maximized.

It is important to note that our hyperplane in the kernel feature space is integrated with the encoder $g$. Hence, optimizing hyperlanes on kernel space also updates representation of attributes. This integration allows for a more compact refinement of attribute clusters from the basis $A_k$, while also efficiently distinguishing them from attributes derived from other bases.

Upon completion of training, we obtain a set of hyperplanes $\\{\Gamma_k\\}_1^K$, which function as tools for outlier detection. These hyperplanes help in determining whether an attribute in the target domain was learned in the source domain.

We sincerely appreciate your thorough and insightful feedback. Below, we outline our efforts to address the valuable points you have raised.

 

The interpretation of attribute-based representation.

In our study, we initially segment the latent space into $K$ groups, where all attributes (sub-latent vectors) within a single group form a basis.

We concur with the reviewer's viewpoint that attribute-based representations are generally presumed to be interpretable and meaningful for the end-users of the model. Typically, learning and evaluating such models require domain-specific knowledge. In our work, we mitigate this requirement for domain expertise in attribute-representation by focusing on a downstream task, specifically classification, as follows:

"A basis of attributes in relation to a given dataset is defined in terms that the attributes within a basis should be capable of predicting the corresponding class-labels for all samples in the dataset."

Formal definition:

(Basis of attributes): For a dataset $\mathbb{D}=\\{(x^i,y^i)\\}_{i=1}^N$ and feature extractor $g$, a set \mathcal{A}\_{k}=\\{ a^1_k,a^2_k,...,a^{N}_k \\\} where each $a^i_k=g(x^i)_k$ is considered as basis of attributes with respect to $(\mathbb{D},g)$ if there exists a function $h_k$ such that $h_k(a^i_k)=y_i, \forall i=1...N$.

This constraints can be obtained by optimize objective function in Equation.(3). However, we further employ cluster-induced viewpoint of kernel method to characterize attributes in bases. To learn the cluster boundaries (hyperplane) $\Gamma_k$ for a basis, we treat attributes from other bases as negatives.

It is important to note that our hyperplane in the kernel feature space is integrated with the encoder $g$. Hence, optimizing hyperlanes on kernel space also updates representation of attributes. This integration allows for a more compact refinement of attribute clusters from the basis $A_k$, while also efficiently distinguishing them from attributes derived from other bases.

Upon completion of training, we obtain a set of hyperplanes $\\{\Gamma_k\\}_1^K$, which function as tools for outlier detection. These hyperplanes help in determining whether an attribute in the target domain was learned in the source domain.

In another interpretation, the basic of attributes finally are characterized by the kernel-models $\\{\Gamma_k\\}_1^K$, which we refer to as kernel-based attributes.

 

The performance improvement.

In this work, our primary interest lies in exploring the properties of kernel-based attributes and their advantages, rather than pursuing state-of-the-art performance.

Recent approaches, such as [1, 2, 3, 4], frequently adopt an ensemble strategy, particularly one based on SWAD [1]. This approach complicates the evaluation of the effectiveness of their latent representations. As a result, our research primarily focuses on methods that rely on a singular model for evaluation. Nonetheless, to provide a comprehensive view, we have broadened our related works section to include these methodologies. Additionally, we have conducted extra experiments where our KEAI model is trained using an ensemble strategy similar to SWAD, as shown in the following table.

These experiments reveal that KEAI not only performs competitively when compared to state-of-the-art (SOTA) methods but also shows notable effectiveness in domains with limited information, such as the Cartoon and Sketch domains.

 

Table 1: Classification Accuracy on PACS with ResNet18

 

[1] Junbum, Cha et al. (2021). “SWAD: domain generalization by seeking flat minima.” In: NeurIPS.

[2] Xu, Chu et al. (2022). “DNA: domain generalization with diversified neural averaging.” In: ICML.

[3] Pcl: Proxy-based contrastive learning for domain generalization, CVPR2022.

[4] Kim, Jaeill, et al. "VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.