LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization

W1: List more computational cost as a limitation.

A1: We thank the reviewer for the suggestion, we will include this limitation in our revised paper.

W2: Experiments with ViTs.

A2: We conducted further experiments by evaluating our method, the baseline ERM, and some leading methods in Tab. 1 with a ViT-Base model using the same setting detailed in the manuscript. Results in Tab. 7 in the general response show that our method can still obtain favorable performance against existing arts despite using different networks.

Q1: Why hard sample mining can improve performance in TerraInc dataset but not the PACS dataset.

A1: We infer this is mainly because compared to the PACS dataset, TerraInc is more difficult for the corresponding expert, which can provide more hard samples to boost generalization (briefly mentioned in Line 623 in our paper). As shown in Tab. 5 in the manuscript, the experts can achieve an average accuracy of 96.7% when tested with the source data in the PACS dataset, indicating that there are very few samples in PACS that can be regarded as hard for the experts. On the other hand, the in-domain results for the experts in TerraInc are 91.1% (Tab. 1 in the general response), suggesting that the dataset can provide more hard samples to guide the target model than PACS, thus improving the performance. We will include this analysis in our revised paper.

The author's rebuttal allayed some of my concerns, and I chose to raise my rating, considering that the paper still had some sparkle to it.

W1: The performance does not show much improvements in Tab. 5

A1: We want to clarify that the in-domain comparisons in Tab. 5 are only used to verify our claim that the target model has evolved to be an expert in all source domains. In the DG setting, out-of-domain performance is often regarded as the metric for evaluating different methods, and our method is shown to outperform many sophisticated designs. When compared to ERM and the different expert models, our method also shows large improvements (Tab. 4). We note that the out-of-domain performance of our method is regarded as impressive by other reviewers.

W2: The compared baseline models are not clearly introduced.

A2: The primary baseline models (in both Tab. 1 and 2) are ERM that directly trains the model with data from all source domains. According to the existing benchmark [20], in the DG task, ERM is a strong baseline that can outperform many SOTAs when evaluated with strict evaluation protocols. As for other compared models, we use most of the implementations in the commonly-used DomainBed [20] for evaluation. It would be better to refer to the benchmark for details.

W3: Comparisons with foundation model that uses data from all source domains is not discussed.

A3: We want to clarify that the referred foundation model is ERM, and we include comparisons with it in all our experiments (i.e. Baseline in Tab. 2 and ERM in all other tables), where our design can improve the performance in almost all evaluated datasets.

Q1 (part 1): Difference between the proposed approach and foundation models, why not directly learn a model using all the source domains?

A1 (part1): Compared to the foundation model, the target model is trained with an additional logit regularization term $\mathcal{L}_{guid}$ (without it, the target model degrades to the foundation model), which is designed to help the target model to be expert in all source domains. As explained in Sec. 4, we reveal that the logit regularization term can help the target model to use more information and mine hard samples from the experts, which are both beneficial for improving generalization.

Q1 (part 2): Why not compare with the Segment Anything Model (SAM).

A1 (part2): Our experiment settings in the generalizable semantic segmentation task are the same as previous mainstreams [32, 48, 12], which train the target model on synthetic data and test it in unseen real-world scenes. We do not compare with SAM mainly because (1) SAM and mainstreams use different network structures: SAM uses ViT and CLIP text encoder in their model, which has a total of 636M parameters. Differently, the mainstreams use DeepLabv3+ ResNet50 as the backbone, consisting of 39M parameters, which is only 6 percent the size of SAM; (2) SAM is trained over 1 billion data, while the synthetic data used for training in our setting is only 34,366. We thus do not compare with SAM.

Our work focus the generalization task, which can be applied to many downstream vision tasks, semantic segmentation included. Unlike SAM that aims to solve segmentation with large models and more data, this work can be applied to situations when there is only limited training data from certain domains, and the task is to generalize the model to new domains, for example in particular medical situations. We leave such exploration to potential future works.

Q2 (part 1): What’s the difference between the proposed approach and the baseline ERM.

A2 (part 1): ERM is the referred foundation model that is trained with all data (by using only $\mathcal{L}_{cla}$ in Eq. (3)). Please refer to our response for Q1 (part 1) for their differences.

Q2 (part 2): Improper to say “only one additional parameter upon ERM” since a target model is involved.

A2 (part2): We appreciate the suggestion. This sentence is to underline that except for the only one weight parameter, our model does not involve more heuristic hyper-parameter settings compared to ERM. We will clarify it in our revised version to avoid confusion.

Q3: How does $q_{\ast}^E$ relates to hard samples and what are those hard samples?

A3: As the classification loss is computed as $-\sum y \log q$, smaller $q$ indicates larger loss, which corresponds to a more difficult sample. This is is consistent with the previous work [27]. In Sec. D.4 in the appendix, we find that hard samples from the domain experts contain more domain ambigous data than that from the model fed with all data, explaning why domain experts must be involved. Please also refer to our general response for details.

L1: No discussions on limitations and potential negative societal impact of the work are provided.

A1: Please note that we discuss the limitation of this work in Sec. A in the appendix. For the potential negative societal impact, this work designs a new idea for the task of domain generalization that holds significant potential for a wide array of real-world applications. It may bear potential societal consequences similar to other machine learning systems. We will include this in our revised paper.

L2: The performance of the proposed approach can be upper bounded by the performance of the expert models, why not directly learn from all source domains?

A2: Our method outperforms domain experts in both out-of-domain (i.e., Tab. 4) and in-domain (i.e., Tab. 5) settings, indicating that the performance is not bounded by domain experts. This is because the target model uses outputs from all experts and gt as training labels (see Eq. (3)), thus the experts can not bound the performance of the target model. When compared to the referred foundation model that directly trains with all data, our method also shows large improvement (i.e., Tab. 1 and 2). Please refer to our response for Q1 (part 1) for the explanations.

W1. Novelty

A1: Differences between Meta-DMoE and LFME are as follow.

1. Domain experts in these two works serve different purposes

Meta-DMoE aims to adapt the trained target model to a new domain in test. To facilitate adaptation, their target model should be capable of identifying domain-specific information (DSI). Therefore, the target model is enforced to extract DSI similar to those from domain experts. Notably, their trained experts are expected to thrive in all domains, which are used to extract DSI not in their trained domains but rather in an unseen one. In a word, domain experts in Meta-DMoE serve as supervision to ensure that the target model can adapt to a new domain. Differently, LFME expects its target model to be expert in all source domains. In our framework, domain experts provide professional guidance for the target model only in their corresponding domains.

2. Different implementations

Meta-DMoE involves meta-learning and test-time training (TTT), where domain experts are used for data from their unseen domains in both training and test. The overall process can be time-consuming due to TTT and the second-order optimization in meta-learning, which may pose efficiency problems compared to the straightforward training in LFME. Meanwhile, their idea is developed on the traditional KD idea that enforces feature similarity between teachers and student, which is shown suboptimal in our analysis.

3. Different effectiveness and efficiencies

When evaluated with the same strict setting in DomainBed, LFME leads Meta-DMoE in all datasets with an average improvement of 1.8pp. The required running time for LFME is also less than Meta-DMoE: 38' vs. 73' for one trial in PACS (even without the experts training time, Meta-DMoE still requires 45' for one trail).

We list comparisons with Meta-DMoE in Sec. 2, and are glad that Reviewer xxxE find it comprehensive. Regarding the idea in [1], it shares similar design with DAELDG that uses different classifiers as experts and averages the final outputs of all experts as the result. Similar intuition has been discussed in our analysis. We will discuss this work in the revised version.

W2: More justifications for argument in Sec. 4.1 and comparisons against LS

A2: We provide more justifications (visual examples included) for further comprehending the analysis in Sec. D.1. More empirical evidence supporting this argument is provided in D.2. Please refer to them for a better understanding.

LS uses a heuristic $\epsilon$ to adjust output probability and prevent overconfidence in the gt label: $(1-\epsilon)H(q, y) + \epsilon H(q, U)$ (given U a uniform distribution). if $\epsilon$ is improperly set (e.g. close to 1) and the dominant label approximates $U$, LS may fail. Differently, LFME uses labels from both gt and experts' outputs to excel in all source domains: $H(q, y) + \frac{\alpha}{2}||z-q^E||^2$, where we do not need to deliberately calibrate the probability. Those two methods differ inherently. Resulsts in Sec. 6.2 also show LFME performs better than latest LS methods. Notably, experiments in Tab. 4 in the general response show that the model is insensitive w.r.t $\alpha$ as it is on par with ERM even with $\frac{\alpha}{2}=1000$, because $q_{\ast}^E$ is mostly aligned with $y$ (see Fig. 2 (a)), which is a much preciser label than $U$. We thus say that LS bears a con of potential improper heuristic designs against LFME. Meanwhile, LS treats all samples equally, while LFME can specifically focus more on hard samples. That is another pro for LFME.

W3: What are those hard samples and why use domain experts

A3: The hard samples are domain ambigous data, aligned with your intuition. We find that domain experts can locate more of these data than random models, thus can better help DG. Please see our general response for details.

W4: More in-domain evaluations

A4: We conduct experiments on other two datasets (i.e. TerraInc and VLCS) and list the results in Tab. 1 in the geneal response. Results are consistent with that in Tab. 5, where the target model can lead the baseline and experts in all cases, validating that it becomes expert on all source domains.

W5: SWAD and hyper-parameter (HP) settings

A5: We apply SWAD only for the target model. We use the default settings in original implementations of Domainbed and SWAD. For your reference, we list all HP in Tab. 2 in the general response.

Q1: Why use parallel training

A1: In the parallel scheme, each training sample will go through 2 forward passes (one for expert and another for the target), and we can use a single optimization to update all parameters and a total of 5000 updating steps for a trial in PACS (see Alg. 1). Instead, if using the sequential scheme, each sample will go through 3 forward passes (one for training expert, and two for training the target), and we will require 2x5000 updating steps for a trial. Training times for these two schemes are 38' (parallel) vs. 55' (sequential), we thus use parallel scheme for simiplicity.

Q2: Is it necessary to ensure the experts' quality

A2: Different from Meta-DMoE, experts in LFME are expected to saturate only on their corresponding domains. Although less data is used, they can still outperform the baseline within their trained domains (see our in-domain experiments). Given that our interest lies primarily in the outputs of these experts within their designated training data, their general proficiency beyond these specific data is of secondary concern. Thus, it may be unnecessary to ensure the quality of the experts across broader domains.

Q3: How does the number of domains affect the proposed approach, will the method benefit from more domains or more data

A3: We answer this question by merging the source domains to create varying domain numbers. As shown in Tab. 3 in the general response, the model performance is generally positively correlated w.r.t domain numbers.

We appreciate the reviewer's feedback and thoughtful response. We answer the additional questions below.

Q: This is because originally we are doing gradient descent using ERM. When we multiply a negative value in front of it, it becomes gradient ascent instead.

A: We want to clarify that our only objetive is to minimize the loss $\mathcal{L}_{all}$ in Eq. (3) throughout the training process. It's important to emphasize that there is no gradient reversal operation in our framework; specifically, we are not performing gradient descent using ERM followed by multiplying a negative value for gradient ascent.

Q: How is $\mathcal{F}$ calculated.

A: we want to clarify that we do not need to compute $\mathcal{F}$ during updating, $\mathcal{F}$ is used as an analytical tool to show how much the gradient magtitudes is rescaled w.r.t to the baseline model that without using the logit regularization term, by comparing ratio of gradients from the two models [64]. In our case, gradients of the two models (LFME and ERM) are $q_c - y_c + \alpha (z_c - q_c^E)$ (e.g. $\frac{\partial \mathcal{L}\_{all}}{\partial z_c}$) and $q_c - y_c$ (e.g. $\frac{\partial \mathcal{L}\_{cla}}{ \partial z_c}$), there gradient ratio is thus $\frac{\partial \mathcal{L}\_{all}}{\partial z_c} / \frac{\partial \mathcal{L}\_{cla}}{ \partial z_c}$, which leads to the computing of $\mathcal{F}$ and $\mathcal{F}'$ in Eq. (6) and (8).

Q: In the given example, how the mining the "hard" sample from an expert achieved and its relation to the reduction of the logit value.

A: For the given example, where $z_1 = z_2$, it indicates that the model is with equal confidence w.r.t the two samples. Without experts involvements, these two samples will contribute equally to the target model's updating. However, when the expert is involved and $q_{\ast 1}^{E} > q_{\ast 2}^{E}$，Eq. (6) and (8) indicate $|\mathcal{F}_1| < |\mathcal{F}_2|$ and $|\mathcal{F}'_1| < |\mathcal{F}'_2|$: the gradient magtitude from $x_2$ is changing more significantly than that from $x_1$ both on the basis ERM, allowing $x_2$ to have a greater impact on the target model's update. The derivations imply that the expert can expose its hard samples to the target model, which is not applicable in ERM.

From your point of view, the objetive in Eq. (3) implicitly encourages the target model to be less confident w.r.t $x_2$. We want to clarify that this confidence reduction does not conflict with mining $x_2$, because lower confidence results in a higher training loss, prompting the target model to explore the corresponding sample more thoroughly, where both the dominant features for classification and other information shared with other classes will be explored, with the dominant features being the primary focus (because both $\mathcal{L}\_{cla}$ and $\mathcal{L}\_{guid}$ in $\mathcal{L}\_{all}$ encourage the gt logit to be the largest, which also indicates that lower confidence in LFME does not equate to lower accuracy—a fact validated by our in-domain results). On the contrary, high confidence indicates that the model may overlook the sample after obtaining its dominant features, making it less important on the updating process. Moreover, penalizing high-confidence outputs has been shown to be effective in improving a variety of learning tasks [a], our objective can thus be viewed as specifically mining hard samples from the experts for penalization. From this perspective, in our case, penalizing confident can be integrated within our explanation of hard sample mining.

We sincerely thank you for your invaluable time and the fruitful discussions. We are more than willing to provide any additional justifications or clarifications you may need.

[a] Regularizing Neural Networks by Penalizing Confident Output Distributions, in ICLR'17

We thank the reviewer for the prompt reply.

Q: Is updating LFME mathematically equivalent to doing gradient descent using ERM followed by multiplying a negative value for gradient ascent?

A: We want to clarify that the two gradient parts of $\mathcal{L}\_{all}$ w.r.t $z$ (e.g. $\partial \mathcal{L}\_{cla}$ and $\partial \mathcal{L}\_{guid}$) may better be interpreted as a whole in a updating step, as one updating step requires the combination of the two gradients for a joint classification task.

According to your suggestion, if we specifically interpret the updating process by separately considering $\partial \mathcal{L}\_{cla}$ and $\partial \mathcal{L}\_{guid}$, we can see that in both terms, $z\_{\ast}$ correspond to largest labels. That is, $z\_{\ast}$ is encouraged to be close to $y\_{\ast}$ by minimizing $\mathcal{L}\_{cla}$ and close to $q\_{\ast}^E$ by minimizing $\mathcal{L}\_{guid}$. This means the updating process will maintain its discriminative ability, and the model will not forget the sample. The different gradient directions can be interpreted as $\mathcal{L}\_{cla}$ encourages $z\_{\ast}$ to become infinitely large, focusing on dominant features, while $\mathcal{L}\_{guid}$ optimizes $z\_{\ast}$ to the opposite direction to be close to $q\_{\ast}^E$ so that other information can also be involved. Therefore, $\partial \mathcal{L}\_{guid}$ should not be sees as tries to "unlearn" what has been learned by$\partial \mathcal{L}\_{cla}$. Rather, on the basis of not compromizing the target model's discriminative ability, they serve different purposes to encourage the model to focus on different types of features.

Q: Is mining hard samples from experts counterintuitive?

A: The primary effect of knowledge distillation (KD)-based methods can be attributed to sample reweighting [b], where the weight of a training sample in the student model is related to the teacher's confidence. Although a different KD scheme is used, we apply the same rescaling analytical tool to demonstrate that one of the working mechanisms of the proposed KD form can still be seen as weighting different samples based on the teacher's confidence. Since different KD forms are utilized, the weighting strategies differ between the proposed LFME and the traditional KD approach where easier samples from teachers correspond to larger weights (comparisons between LFME and traditional KD ideas are presented in Sec. 6.3, where LFME performs better for the task of DG). Meanwhile, since hard samples from domain experts are more likely to be domain ambiguous data, enforcing the target model to learn these data more throughly can help it to explore more domain-invariant features, which are crucial for generalization [17]. In this regard, mining hard samples from domain experts is an intuitive motivation for improving DG.

Q: Arguing from the the confidence perspective.

A: We thank the reviewer for the thoughtful suggestion. We want to clarify that the hard sample mining is aligned with the proposed confidence reduction perspective: in the given example, the target model will reduce confidence of $x_2$ more than that of $x_1$ even if they are with the same output logits, consistent with our explanation that hard samples from the experts will affect more on the target model's updating. Therefore, there may be no intrinsic difference between the hard sample mining and the suggested confidence reduction idea. Meanwhile, we provide empirical evidence in Sec. D.3 to show that the classification results of hard samples from corresponding experts are indeed improved in LFME, we thus argue the working mechanism from the hard sample mining perspective.

On the other hand, merely analyzing the confidence is similar to our initial analysis in Section 4.1, where the idea of smoothing the output probability of the target model is equivalent to making it less overconfident.

We sincerely thank you for your invaluable time and all thoughtful suggestions, which definetly help improve our paper. We wish these clarifications are helpful for easing your concerns.

[b] Born-Again Neural Networks, in ICML'18.

W1: Performance in the Meta-DMoE paper is higher than that in Tab. 1, and highlight the resource usage differences.

A1: Note that we mainly use the ResNet18 model for evaluating Meta-DMoE in Tab. 1, and the experiments are conducted for a total of 3x20 trials using the default hyper-parameter settings that are within large ranges. These settings are different from those in the original paper (where ResNet50 results are reported), which can cause the differences in the reported numbers of Meta-DMoE. We will include this clarification in our revised paper.

We thank the reviewer for the suggestion. We list the efficiency comparisons in our response regarding w1 for Reviewer oULp, and we will include the differences in the related work section.

W2: Conducting ablation study to analyse the selected value range for the weight parameter.

A2: We conduct this ablation study in Tab. 4 in our general response, which shows that our method can obtain relatively better performance when the weight parameter is selected wihtin [0.1, 10].

W3: Experiments in datasets from the Wilds benchmark.

A3: We conduct experiments using three real-world datasets (i.e., iWildCam, RxRx1, and FMoW) using the default settings in Wilds. Results listed in Tab. 5 in the general response show that our method can improve the baseline in all three datasets and obtains favorable performance against existing arts.

W4: Including RIDG [1] and MIRO+SWAD in Tab. 1.

A4: We include comparisons with the referred methods in Tab. 6 in our general response. The comparisons will be included in Tab. 1 in our revised paper.

W5: Minor issues.

A5: We thank the reviewer for the notifications, these typos will be corrected in our revised paper.

Q1: Solution for datasets with a large number of domains.

A1: When encountering datasets containing numerous domains, we can utilize the existing strategy [a] that clusters the training domains into fewer super-domains, thus reducing the expert number. We leave this exploration to potential future works.

[a] Orbit: A real-world few-shot dataset for teachable object recognition, in ICCV'21

Q2: How to apply the method to the training process of large foundation models?

A2: When applied to the training process of large foundation models, a possible solution for easing the memory constraint problem is to use a sequential training strategy that first learns different experts, and then trains the target model without optimizing these experts. In this way, professional guidance from different experts can still be obtained, and we can avoid the memory constraint for training $M+1$ models at the same time (with $M$ the expert number).

Q3: Including experiments of the free lunch idea in Imagenet setting.

A3: We conduct experiments with the free lunch idea in Imagenet as suggested. Compared to the baseline, the free lunch can obtain better performance by 0.4pp for the top1 acc (76.9 vs. 76.5) and 0.5pp for the top5 acc (93.7 vs. 93.2). These results show that the free lunch idea is beneficial for large datasets.