Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws

We thank the reviewer for appreciating the technical merits of this paper. We will follow the reviewer's suggestions to improve the paper, which should be an easy task.

Q1: Would this method work for other tasks? Is there a reason to focus only on CLIP?

A: Yes! We have discussed that the existing works of employing the RHO loss for data selection or sampling for different tasks can be considered as heuristic implementation of the proposed framework (Section 4). These different tasks include classification (Mindermann et al., 2022) and training LLMs (Lin et al. 2024). We have discussed the reason for our focus on CLIP at the beginning of Section 5.

Q2: How $C_2$ on the right hand side at line 733 is derived in the proof of Corollary 4.3.

A: The constant $C_2$ in Corollary 4.3 is the same as that in Theorem 4.1. In Theorem 4.1, we derived $R(\tilde{\theta}_*)\leq \underbrace{\inf _{\theta\in \Theta} \left(R(\theta) + \sqrt{\frac{2\rho}{n} \mathrm{Var}(\ell(\theta, \cdot)- \ell(\theta _{\mathrm{ref}}, \cdot))}\right)} _{\mathrm{term1}}+ \frac{C _{2}}{n},$ where $C _2=(50\rho/3+4)M$. In the proof of Corollary 4.3, we showed that term1 in the brace is upper bounded by $R(\theta _\mathrm{ref})$. Hence, $C_2$ is the same as in Theorem 4.1.

Q3: Does the proposed method optimize the same objective as (Evans et al., 2024a;b)? Comparing with their approach seems important.

A: We would like to point out that we indeed compared with Evans et al., 2024a (JEST). We omitted the comparison with Evans et al., 2024b (ActiveCLIP) since JEST is a later work and improved over it by the same groups of authors. Our method does not optimize the same objective as their methods:

• their methods use mini-batch contrastive loss to define a RHO loss. While we define the loss using all negative data in the training dataset (instead of the mini-batch) and use a rigorous optimization approach (SogCLR) to optimize the objective.
• their loss is used for data selection of anchor data, i.e., selecting subset for training. In contrast, our DRRho contrastive loss, defined by leveraging the relationship between global contrastive loss and DRO, has an effect of data re-weighting of the negative data for each anchor data.

Moreover, their methods consume more resources for data selection due to sampling from a larger batch. We compared the performance of JEST and DRRho-CLIP on DFN-12M and DFN-192M with fixed amount of compute. The results (c.f. Tables 1 and 3 in the paper) showed that DRRho-CLIP significantly outperforms JEST.

Q4: This paper doesn't seem to be able to explain all LEAR methods. It should be more specific about which LEAR method(s) this paper is for.

A: LEAR refers to, as stated at the beginning of the abstract, leveraging a pretrained model ... through strategic data selection or weighting. Thus we focus on methods for data selection and weighting. We categorize existing works of leveraging a pretrained model into different families in Section 2, where in Lines 120-140 we provide background about the specific LEAR methods that motivates this work, such as RHO (Mindermann et al., 2022), RHO-1 (Lin et al., 2024), ActiveCLIP (Evans et al., 2024b) and JEST (Evans et al., 2024a) to make Section 4 clearer.

Q5: I would suggest that the authors provide some background about the specific LEAR methods this paper can provide insights into. This would make the purpose of Section 4 clearer.

A: Thank you for the suggestion. We will give more background about the related LEAR methods mentioned in the above question to make Section 4 clearer.

Q6: While it's the convention to call scaling law scaling law, the word "law" is actually misleading, as it is not a real law. If possible, I would suggest that you just say you scale better.

A: While we agree with the reviewer that the term "law" is misleading or sometimes overclaims, however, we hesitate to invent a new term as it has been widely accepted in the literature. Cherti et al. (2023) also used the term "scaling law" for CLIP training.

Q7: Does the x-axis of Figure 2 represent the training time? If not, why did you use different line to represent different ratio of data used?

A: The x-axis in Figure 2 represents the number of samples seen, which is proportional to training time. Different ratio of data means the training dataset size is different. For example, 100% data means the whole dataset is used during training, while 50% data means only half of the dataset is used during training. For different datasets, we set the total number of samples seen to be the same, which means on smaller datasets there will be more epochs.

Q8: Why is the trend of Figure 2 different from Figure 3?

A: In Figure 2, we use ImageNet-1K Top 1 Accuracy as the y-axis. While in Figure 3, we use ImageNet-1K Error as the y-axis, which is equal to 100% - ImageNet-1K Top 1 Accuracy. Thus the two figures have different trends.

We thank the reviewer for their valuable comments and suggestions.

Q1: More powerful reference model does not always yield superior target model performance. But Corollaries 4.2 and 4.3 show that a better reference model should lead to greater improvements in target model training.

A: There is misunderstanding of results in Corollaries 4.2 and 4.3 regarding that more powerful reference model yields superior target model performance. Corollary 4.2 shows that the generalization depends on the variance of the RHO loss for a reference model. However, a powerful reference model does not necessarily have a small variance of the RHO loss, i.e., $\mathrm{Var}(\ell(\theta_*, \cdot) - \ell(\theta_\mathrm{ref}, \cdot))$, where $\theta_*$ is the optimal solution in the considered model space. Corollary 4.3 only compares a reference model in the same space of the target model. However, in Table 1 except for ViT-B/16, other two reference models (ViT-B/32, ViT-L/14) are not in the same model space of the target model ViT-B/32. Hence, we cannot apply Corollary 4.3.

We would like to point out that this phenomenon is also empirically observed in other works that leverage a reference model, e.g. DFN (Fang et al., 2024), JEST (Evans et al. 2024a) and MobileCLIP (Vasu et al., 2024), where stronger reference models did not necessarily lead to more powerful target models.

To better verify Corollary 4.3, we have conducted the following experiments with ViT-B/16 as target model and ViT-B/16 trained on different subsets of DFN-12M as reference models. We list the ImageNet-1K Top 1 Accuracy of both the target model and the reference model in the following table, from which we can observe that the performance of the target model and the reference model is positively correlated.

Q2: Regarding interpretation of Corollary 4.3 and do Theorem 4.1 or Corollary 4.2 still indicate $n=O(m)$ sample complexity?

A: The interpretation of Corollary 4.3 about the reduced sample complexity is to guarantee that the generalization error of the learned model $R(\tilde\theta_*) - R(\theta_*)$ by our framework is on par with that of the reference model $R(\theta_{\text{ref}}) - R(\theta_*)$, where $m$ is the data size for training the reference model. If we want to use Corollary 4.2 for deriving $R(\tilde\theta_*) - R(\theta_*)$ in the same order $1/\sqrt{m}$ of $R(\theta_{\text{ref}}) - R(\theta_*)$ , it will imply that $n = \max(\sqrt{m}, m\mathrm{Var}(\ell(\theta_*, \cdot) - \ell(\theta_\mathrm{ref}, \cdot)))$. It could be still much better than $O(m)$ as $\mathrm{Var}(\ell(\theta_*, \cdot) - \ell(\theta_\mathrm{ref}, \cdot))$ could be very small. In this context, we cannot ignore $\mathrm{Var}(\ell(\theta_*, \cdot) - \ell(\theta_\mathrm{ref}, \cdot))$. Hence, it is more convenient to use Corollary 4.3 to make this comparison argument. Thus the two results do not contradict to each other.

We thank the reviewer for their constructive and positive evaluation of our work. We are happy to address any concerns the reviewer may have in later stage.

We thank the reviewer for the suggestion on experiments. We believe that the comments raised by the reviewer are not critical drawbacks of this paper. We request the reviewer to consider our contribution in terms of theoretical framework and analysis, and our experiments comparison with multiple baselines.

Q1: Why was the derivation of full scaling laws (e.g. following Cherti et al. 2023) was omitted in the work?

A: Thank you for raising this concern. We would like to note that Cherti et al. (2023) focused on reproducing CLIP, which is empirical only. In contrast, we have rigorous theoretical analysis of the generalization error, which is not restricted to any model or data scale. Our experiments include the comparison with multiple baselines in the context of learning with reference models and our scaling law experiment serve to corroborate the presented theory.

Following the reviewer's suggestion, we have added scaling law experiments for two more model scales (ViT-B/32 and ViT-L/14). The experiments have not completely finished since they take many days to run on large scales, but we can already observe that DRRho-CLIP has a better scaling trend across different model scales than OpenCLIP.

• FLOPs vs. performance, without accounting for reference model cost: anonymous link. The fitted scaling law for OpenCLIP is $E=5.77\cdot C^{-0.116}$, where $E$ is the ImageNet-1K Error and $C$ is the compute (GFLOPS). The fitted scaling law for DRRho-CLIP is $E=7.15\cdot C^{-0.127}$.
• FLOPs vs. performance, accounting for reference model cost: anonymous link. The fitted scaling law for OpenCLIP is $E=5.77\cdot C^{-0.116}$, while for DRRho-CLIP it is $E=8.78\cdot C^{-0.135}$.

Q2: It would be also good to plot FLOPs vs. performance, accounting for FLOPs used by reference model.

A: Links to the plots are provided in the answer to the above question. We want to highlight that the cost of leveraging a reference model can be amortized. In particular, since the reference model is frozen during training, we store the reference model features before training so that they can be reused multiple times across epochs and across different runs. Indeed, these features of the reference model have been already computed in the data filtering stage for creating existing datasets, e.g., LAION-2B (Schuhmann et al., 2022), DFN-2B (Fang et al., 2024) and Datacomp-1B (Gadre et al., 2023). In this case, we can directly leverage the features of the reference model without spending any resources computing them.