Questions

1. I think if possible, the authors can have more discussions on the novelty of this paper.

We thank the reviewer for their request to elaborate further on the novelty of our work. As mentioned in the related works section of the paper, our work is the first to explicitly formulate both the pre-training and fine-tuning stages within SSL as a BLO problem, which previously have not been explored in SSL research. Current SSL works that integrate BLO, do so in very different manners, utilizing it exclusively in either the pretext or fine-tuning stages, but not jointly. Additionally, we refer to our response to Weakness 1 for additional emphasis on the novelty of BiSSL through a discussion on how it differs from the implementation of BLO in meta-learning. This discussion has also been added to the appendix of the paper.

2. I am curious about the results of the proposed method in larger backbone (at least ResNet-50).

We have addressed the reviewers request, and extended the experiments accordingly. These experiments are also introduced and discussed in our response to a previous review; however, we reiterate them here for the sake of clarity. We conducted the experiments on a selection of downstream datasets (Flowers [3], DTD [4] and VOC07 [5]), where the ResNet-18 backbone has been replaced by a ResNet-50. Otherwise, the setup is identical to as in the paper. The results are as follows:

The results indicate that BiSSL still outperforms the baselines, despite using a larger backbone model. We hope this addresses the reviewer’s request and that it supports the idea that BiSSL is able to scale well to larger model architectures. We are currently expanding the evaluation to include additional downstream datasets to provide a more comprehensive assessment. We intend to include these results in the appendix of the submission after the review period, due to resource constraints that prevents us from completing these experiments within the rebuttal period.

3. The authors can have more discussions on the limitation of the proposed method.

Compared to the conventional SSL setup, BiSSL requires the additional access to the pre-training data or a subset of it, as well as knowledge of the pretext task used to pre-train the model. Additionally, calculating the upper-level gradient in BiSSL introduces some non-neglebible computational overhead. We refer to our response to reviewer kv92 for further discussion on this, as well as a quantitative overview of the GPU hours used in the experiments.

We have updated the future work section of the submission to address these limitations as potential areas for further exploration.

---

Conclusion

Once again, we thank the reviewer for their comments. We hope our response provides clarity and addresses the concerns raised. We are happy to further elaborate or answer further inquiries as needed.

---

References

[1] Chelsea Finn, P, et al. "Model-agnostic meta-learning for fast adaptation of deep networks." International Conference on Machine Learning, 2017.

[2] Aravind Rajeswaran, et al. "Meta-learning with im-plicit gradients." Neural Information Processing Systems, 2019.

[3] Maria-Elena Nilsback and Andrew Zisserman. "Automated flower classification over a large number of classes." Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.

[4] M. Cimpoi, S. Maji, et al. "Describing textures in the wild." Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.

[5] M. Everingham, L. et al. "The PASCAL Visual Object Classes Challenge 2007 (VOC2007)"

We thank the reviewer for their constructive comments. We address them as follows:

---

Strengths

1. This paper is overall well-written and easy to read, with a clear organization. The figures and algorithm illustrates the idea very clearly.

We are pleased that the reviewer found the paper well-structured and appreciated the clarity of the figures and algorithm.

2. The idea of introducing bilevel optimization into pre-training and fine-tuning pipeline is reasonable for me.
3. The experiment design is good, covering both quantitative and qualitative results, together with some analysis.

It is delighting to see that the reviewer finds the integration of bilevel optimization into this context to be reasonable and we are grateful for the recognition of our evaluation!

---

Weaknesses

1. As mentioned in strengths, the idea of this paper is reasonable, but not very new to me. The bilevel optimization is widely used in meta-learning.

While BLO has been utilized in meta-learning frameworks such as MAML [1] and iMAML [2], we would like to emphasize that our method and application differ from these approaches in several key aspects. In meta-learning, BLO is used to address few-shot learning tasks, where the focus is on adapting models to new tasks with minimal data. As present in our work, however, we leverage BLO to tackle the more complex task of SSL pre-training on a vast unlabeled dataset along with fine-tuning concurrently. Another important distinction is that in meta-learning, the upper and lower-level objectives are often closely related, with the upper level typically being a summation of the lower-level tasks. In contrast, our framework involves entirely different objectives at each level, each also using potentially fundamentally different datasets. Moreover, BiSSL is tailored to fine-tune pre-trained backbone models, in contrast to the conventional use of BLO in meta-learning, which primarily focuses on training model parameters from an uninitialized state.

We have added a section in the appendix of the paper to include a clarification on how our approach differs from meta-learning, based on the response provided above.

2. In the experiment section, the backbone (ResNet-18) seem to be relatively smaill.

While we acknowledge that ResNet-18 is a relatively small-scale model by current standards, its use demonstrates an applicable aspect for scenarios that that do not rely on relatively large architectures. However, we agree that exploring larger backbones is important to align with modern trends in SSL. To this end, we conducted additional experiments using ResNet-50 backbone. We refer to our response to question 2 for results and details.

Continued...

Dear Reviewer ZY6B,

We have responded to your constructive review, incorporating your suggestions as well as including additional experiments to address the concerns and suggestions raised. As the end of the discussion period comes closer, we kindly ask you to read our response. We are happy to provide further clarifications or discuss any further questions you may have.

Best,

Authors

Continued answer to W2...

For the second set of experiments we instead pre-trained on the Imagenet dataset for 100 epochs. Since the BiSSL stage now involves the equivalent of only 8 pretext epochs, we consider the difference in training duration between the baseline and BiSSL setups to be negligible. Hence we utilized the same pre-trained backbone for both the baseline and BiSSL-trained models in these experiments. All other experimental settings remain consistent with those outlined in the paper. The results are as follows:

In both instances, BiSSL consistently surpasses the baselines, suggesting that its advantages persist even when applied to larger datasets and models. We hope this helps to alleviate the reviewer’s concerns regarding the execution of comparative experiments on larger models and datasets. Should additional architectures or datasets be of interest, we would gladly experiment with such, provided that time allows.

We are currently working on extending all three sets of experiments presented here to include evaluation on a larger selection of downstream datasets. While resource constraints prevent us from obtaining all desired results within the rebuttal period, we will update the paper to include these findings in the appendix as soon as they become available.

---

Questions

For Fig. 2, did the author analyze why the proposed method converges quickly in the early stages of training? Is there over-fitting?

Thank you for raising this question. To clarify, Fig. 2 does not show the progression of a single model’s performance during training. Instead, it illustrates the final test accuracy achieved by separately trained models with varying durations of self-supervised pre-training followed by the application of BiSSL or fine-tuning. The general trend observed in the figure thus indicates that BiSSL consistently retains its classification accuracy advantage, irrespective of the pre-training duration.

We have updated the submission to include a revised description of the experimental setup underlying the figure, along with an updated figure caption.

---

Conlusion

We again thank the reviewer for their useful feedback. We hope our responses address the raised concerns, and we remain happy to provide further clarifications or answer additional questions

---

References

[1] Jean-Bastien Grill, et al. "Bootstrap your own latent-a new approach to self-supervised learning." Advances in neural information processing systems, 33:21271–21284, 2020.

[2] Maria-Elena Nilsback and Andrew Zisserman. "Automated flower classification over a large number of classes." Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.

[3] M. Cimpoi, S. Maji, et al. "Describing textures in the wild." Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.

[4] M. Everingham, L. et al. "The PASCAL Visual Object Classes Challenge 2007 (VOC2007)"

We thank the reviewer for their constructive feedback and insightful comments. Below, we address the raised comments.

---

Strengths

1. The paper is well-written and easy to follow.

We are happy to hear that the clarity and structure of the paper is appreciated.

2. The idea of using bilevel optimization for SSL is novel and interesting.

It is gratifying to see that the novelty of our approach is recognized, and we are encouraged by the reviewers interest!

---

Weaknesses

1. This study relies solely on SimCLR as a comparison method, lacking sufficient comparative experiments with approaches better suited for small-scale datasets, such as BYOL and SwAV. Additionally, it omits necessary explanations of more recent self-supervised methods like DINOv2.

We agree that expanding our comparisons to include other SSL pre-training methods strengthens the experimental evaluation. To address this, we have conducted additional experiments using BYOL [1] as the pre-training method, and evaluated performance on a selection of the downstream datasets evaluated in the paper (Oxford 102 Flowers [2], DTD [3], and VOC07 [4]). In this experimental setup the predictor network uses an architecture identical to the projector network, and the EMA decay follows a scheduler as in [1] with a base decay of $\tau_{\text{{base}}}=0.9995$ and the base learning rate is additional adjusted to $1.0$, aligning the experimental setup more closely with [1]. Otherwise the setup remains consistent with the one in the paper. The results are presented below:

We observe that BiSSL in general demonstrate improved downstream performance of the baselines, suggesting that the benefits of BiSSL is not necessarily confined to the pre-training method being SimCLR. While BYOL is expected to be better suited than SimCLR on small-scale datasets, the observed decline in performance could be attributed to differences between our experimental setup and the original BYOL implementation [1] (e.g., size of MLP, uniform weight decay on all layers, hyper-parameter adjustments to dataset, etc.).

Regarding addressing more recent SSL methods, such as DINOv2, we acknowledge this point and have updated our introduction to ensure a more up-to-date overview of the known impactful SSL methods, including DINOv2.

2. There is a lack of comparative experiments on different network architectures, as well as the results of SSL's standard dataset ImageNet. In the absence of sufficient data, the performance of SSL is not obvious compared to SFL.

We recognize that additional exploration with larger models and pre-training datasets would further enhance the robustness of our findings. To address the reviewer's concerns, we conducted two additional sets of experiments. These experiments are also outlined in our response to a previous review, but we present them here again for the sake of clarity. First we replaced the ResNet-18 backbone used in our initial experiments with a larger ResNet-50 backbone to assess whether BiSSL provides performance benefits on larger architectures. Results are summarized in the table below:

Dear Reviewer NGXe,

Thank you again for your constructive feedback. We have addressed your concerns and included additional experiments as part of our response. With the end of the discussion period slowly approaching, we would appreciate if you could read our response so we can address any further questions that you might have.

Best,

Authors

References

[1] M. Everingham, L. et al. "The PASCAL Visual Object Classes Challenge 2007 (VOC2007)"

[2] M. Cimpoi, S. Maji, et al. "Describing textures in the wild." Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.

[3] Ting Chen, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning, pp. 1597–1607. PMLR, 2020.

[4] Jean-Bastien Grill, et al. "Bootstrap your own latent-a new approach to self-supervised learning." Advances in neural information processing systems, 33:21271–21284, 2020.

[5] Maria-Elena Nilsback and Andrew Zisserman. "Automated flower classification over a large number of classes." Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.

[6] Junjie Yang, et al. "Provably faster algorithms for bilevel optimization." Advances in Neural Information Processing Systems, 2021.

[7] Feihu Huang. "Optimal Hessian/Jacobian-free nonconvex-PL bilevel optimization." Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 19598–19621. PMLR, 21–27, Jul 2024.

We thank the reviewer for their comments. We are very pleased that the novelty and demonstrations of the BiSSL framework has been recognized!

Weaknesses + Questions

Weakness/Question 1

Limited Experimental Scope: The evaluation of BiSSL is confined to classification tasks, and the models and datasets used are relatively small-scale. This restricts the demonstration of the framework's scalability and its ability to handle larger and more complex datasets, which is a critical aspect for real-world applications.

We acknowledge the reviewer’s concern regarding the scale of the datasets used in our evaluation. The downstream datasets in our experiments, such as VOC07 [1] and DTD [2], are commonly used to benchmark self-supervised learning methods, including works like SimCLR [3] and BYOL [4]. Hence, we argue that these benchmarks provide a meaningful basis for downstream evaluation. However, while the STL10 dataset used for pre-training in our experiments provides reasonable scale and complexity, we acknowledge the importance of experimenting whether the benefits from BiSSL persists with larger-scale pre-training datasets like ImageNet. To address this limitation, we have extended the experiments using ImageNet for pre-training. In these experiments, we conducted pre-training on Imagenet for 100 epochs, using the same pre-trained backbone for both the BiSSL and baseline models. The following table outlines downstream classification accuracies on a selection of the downstream datasets utilised in the experiments of the paper:

Additionally, to address the similar concerns regarding experiments using larger-scale models, we repeated the same experiments as in the paper, but with a larger ResNet-50 instead of a ResNet-18 as the backbone model:

In both cases, BiSSL generally outperforms the baselines, implying that its benefits remain present on larger datasets and models. We hope this additional evidence addresses the concern regarding BiSSL’s applicability on larger datasets and models. Furthermore, we are in the process of expanding both sets of experiments to encompass a wider range of downstream datasets. Although computational resource limitations prevent us from obtaining all desired results within the review period, we will incorporate these findings into the appendix of the paper as soon as they become available.

Continued...

Weakness/Question 2

Compromise on Generalizability: The pre-training setup of BiSSL aims to align with specific downstream tasks, which might sacrifice the generalizability of the pre-trained models. This alignment could potentially limit the framework's effectiveness for a diverse range of tasks that were not considered during the pre-training phase.

While BiSSL may limit the backbone’s applicability to other tasks, its primary objective is not to create a general-purpose model. Rather, the goal is to enhance the performance of a pre-trained backbone on a specific downstream task, akin to the purpose of fine-tuning in general. This is also reflected in our training pipeline, which initiates BiSSL with a general-purpose backbone obtained through self-supervised pre-training. To disclose, we did in our experiments opt for shorter pre-training durations before applying BiSSL to facilitate more direct comparisons with the baselines. However, this does not suggest that BiSSL inherently requires less pre-training to be effective. Practitioners can apply BiSSL to any sufficiently pre-trained backbone, as evidenced in Fig. 2 of the paper, where BiSSL consistently outperforms the baseline once pre-training exceeds a certain duration.

We appreciate the opportunity to clarify this point and hope this explanation addresses the reviewer’s comment. We are happy to elaborate further if desired.

Weakness/Question 3

Increased Training Complexity: The introduction of bilevel optimization makes the training process more complex compared to the traditional pre-training and downstream fine-tuning paradigm. This added complexity could pose challenges in terms of computational resources and may require additional expertise to implement and optimize, potentially hindering its adoption in simpler or more streamlined workflows.

We acknowledge that bilevel optimization introduces additional computational steps compared to the conventional self-supervised training pipeline. To provide transparency regarding the computational requirements, we document the GPU hours required for pretext training and BiSSL for the various documented experimental setups:

While the computational cost of BiSSL increases with model size, it remains relatively constant with respect to the size of the pre-training dataset, as the number of gradient steps for BiSSL training is fixed by design. We aim to include this table in the appendix once the aforementioned experimental results are available. Additionally, the conjugate gradient algorithm used in our implementation is not the most recent or efficient approach for calculating upper-level gradients in bilevel optimization. Future work could explore more recent techniques [6,7] to reduce BiSSL’s computational overhead without compromising performance. We updated the submission to reflect this as an opportunity for future work.

Regarding implementation complexity, BiSSL is designed to be plug-and-play, requiring minimal modifications to existing pre-training and fine-tuning workflows. Any practitioner familiar with SSL and fine-tuning should find it straightforward to integrate BiSSL into their pipeline. To lower the barrier to adoption, we have made our implementation open-source and publicly available on GitHub, accompanied by documentation and examples.

We hope this demonstrates that while BiSSL introduces some additional complexity, it remains feasible for broader adoption.

---

Conclusion

We thank the reviewer again for their constructive feedback. We hope our responses address these concerns, and we remain happy to provide further clarifications or answer additional questions.

Dear Reviewer kv92,

We have responded to your review, covering your concerns and adding new experiments. As the end of the discussion period is approaching us, we would greatly appreciate it if you could review our response and let us know if there are any further questions or clarifications needed.

Best,

Authors

Thank you for considering our rebuttal. We would like to clarify the intent of the statement you highlighted. That particular statement was intended to explain that BiSSL is a fine-tuning method rather than a pre-training approach. When deploying a pre-trained backbone for a specific downstream application, fine-tuning is applied. After fine-tuning for a specific task, the backbone’s applicability to other tasks may naturally be limited. Fine-tuning is a critical step in self-supervised learning pipelines, and BiSSL enhances this process, as evidenced by its consistent performance gains over baselines in our experiments. To the best of our knowledge, BiSSL is the first to explicitly incorporate knowledge of the pre-training objective in conjunction with downstream fine-tuning.

In the meantime, we have conducted additional experiments to further address the reviewer’s comments regarding the use of larger models and datasets. Specifically, we used a ResNet-50 backbone pre-trained on ImageNet, which aligns closely with the experimental scope of SSL works such as SimCLR and BYOL. The results are as follows:

Once again, BiSSL demonstrates superior performance compared to conventional fine-tuning.