Model Developmental Safety: A Safety-Centric Method and Applications in Vision-Language Models

We thank the reviewer for acknowledging the value of our work and providing helpful comments. Below we would like to answer the remaining questions.

RQ1: It’s challenging to distinguish this work from conventional Continual Learning (CL) approaches.

A: Our proposed framework differs from conventional continual learning in two folds. (a) Learning setting: In typical settings for continual learning, models are always trained to learn new tasks, with limited access to previously trained data. In contrast, our work may be utilized to either learn new tasks or improve existing tasks, with sufficient number of old data for ensuring zero-forgetting on protected tasks. (b) Goal: the goal of continual learning is to have a good average performance for all the learned tasks, our work prioritze preserving some protected tasks while improving the target tasks. Therefore, our work exhibits substantial distinction with continual learning.

RQ2: Image classification in CLIP carries relatively low safety risk. It could be beneficial to explore a more safety-critical application domain.

A: Note that experiments in our paper like weather detection and scene recognition are directly related to safety-critical autonomous driving systems. In these scenarios, enhancing the detection of one type of weather at the expense of reduced performance for other types could pose significant safety risks. Moreover, our proposed retention-centric optimization framework is generic and CLIP model with classification task is a demonstration, it can be easily extended to other losses or other models, such as a supervised finetuning loss for LLMs or a standard cross-entropy loss for learning a lightweight model. We hope our work can inspire researchers in safety-critical application domain for further exploration.

RQ3: How does the definition of DevSafety differ from similar metrics, such as the forgetting measure commonly used in continual learning?

A: As pointed out by reviewer, DevSafety is defined to take worst-case of all the protected tasks to measure if all the protected tasks are strictly preserved. In contrast, forgetting measure commonly used in continual learning is defined as average performance drop of protected tasks. Note that average performance doesn't drop doesn't mean each individual protected task performance doesn't drop, e.g., some protected tasks get better and some protected tasks get worse. Due to the intrinsic nature of each protected task, such as each task is associated with detecting one kind of disease for medical diagnosis, it may lead to potential unsafe deployment even when the average performance doesn't drop. So, the primary aim of DevSafety is indeed to achieve zero forgetting for safety-critical applications.

RQ4: It would strengthen the paper’s claims to compare the proposed method with more recent baselines mentioned in the related work.

A: Thank you for your suggestion. We are working on adding a recent replay-based contrastive continual learning baseline [1]. However, given limited time we are still working on the experiments. We would like to emphasize that all the baselines we compared are continual learning baselines, with FLYP standing for direct-replay methods, GEM as a typical continual learning method, WCCL and RM as the regularized-based continual learning baselines tailored to our setting. We expect that continual learning can not ensure model developmental safety as they focus on trading off between protect tasks and target tasks.

[1] Cha, Hyuntak, Jaeho Lee, and Jinwoo Shin. "Co2l: Contrastive continual learning." Proceedings of the IEEE/CVF International conference on computer vision. 2021.

RQ5: Suggestions about the definition of s(x;w), discussion of future directions, literature review on Safe Reinforcement Learning (SafeRL)

A: We thank the reviewer for helpful suggestions for improving the paper. We have revised the writing accordingly. We'd like to mention that the reason for deferring the definition of s(x;w) to line 179 is that the definition in line 179 is the special form for CLIP models, so we present the special explicit form of s(x;w) after introducing CLIP models to avoid confusion. As suggested, we have included more discussion of further directions of this work. Please refer to our new version of the paper.

Q: Concerns regarding RQ1 and RQ4 remain insufficiently addressed.

A: We thank the reviewer for the prompt response. Regarding R4, we've finished the experiments with a recent replay-based baseline, namely Co$^2$L[1]. Following their paper, we tune their $\tau$ in {0.05, 0.1}, $\kappa$ in {0.1, 0.2}, $\kappa^*$ in {0.01, 0.1}, $\lambda$ in {0.1, 1, 10}. The results are presented below. We can see that even recent SOTA continual learning method still fails to ensure model developmental safety indicated by that Retention Ratio is zero and DevSafty measure is less than zero. This is anticipated as conventional continual learning focuses on trading off between protect tasks and target tasks, without ensuring zero-forgetting. We included the results in the revision. The experiments further highlight the distinction between conventional continual learning and our approach, demonstrating that existing continual learning methods cannot achieve the model developmental safety explored in this paper, as related to RQ1.

[1] Cha, Hyuntak, Jaeho Lee, and Jinwoo Shin. "Co2l: Contrastive continual learning." Proceedings of the IEEE/CVF International conference on computer vision. 2021.

Thank you for your constructive comments. We've revised our paper to address raised concerns.

RQ1: The use of the terms "safety" / "safety-centric" in this paper is confusing because it doesn’t engage with the broader ethical and operational safety considerations commonly associated with the term.

A: We thank the reviewer for the constructive comments. We agree that some usages of term "safety" in the paper might be confusing, such as "safety-centric method", "safety of safety". To address ambiguity, we replaced "safety-centric method" with "retention-centric method", "safety of safety" with "retention of safety", "safety ratio" with "retention ratio", since our work, as summarized by the reviewer, focuses on designing constraints to retain protected task performance. But we prefer to keep the term "model developmental safety" as our work underscores the importance of strictly preserving existing protected ability in favor of potential safe applications and development efficiency in the model development process. Other words like "stability", "preservation" are wildly adopted in the existing continual learning literature but these works just mitigate the forgetting but not achieve strictly perservation or zero-forgetting. I believe the term "model developmental safety (MDS)" will be helpful for readers to identify the difference between our work and exisiting literature for iterative model development process. Furthermore, to prevent any ambiguity around the term "model developmental safety", we define it at the beginning of the paper, and revise the paper to ensure that every mention of "safety" is preceded by "developmental", when referring to MDS, to clarify its meaning for readers.

RQ2: CLIP is only one kind of Vision Language Model (VLM), 'aka' is inappropriately used in the paper and other representative variants such as LLaVA and BLIP that can generate languages are not evaluated in the paper.

A: Thanks for pointing out the inappropriate use of "aka". To avoid confusion, we have revised the paper with "… we study how to develop a pretrained vision-language model, specifically the CLIP model, …" . Note that the focus of the paper is to propose a constrained optimization framework to strictly preserve the existing protected capabilities while improving target task performance in iterative model development process. To demonstrate the proposed framework, we apply the framework to develop a CLIP model for acquiring new capabilities or improving existing capabilities of image classification. Our framework owns the potential to be applied to other variants of VLMs such LLaVA and BLIP with corresponding adaption with objective and constraint design, but as they are not the focus of this paper, we leave them for further exploration.

RQ3: The relationship between this paper and knowledge/representation editing on VLM/LLM.

A: We thank the reviewer for the constructive suggestion. Knowledge/representation editing on VLM/LLM is related to our work but has a different focus. Knowledge/representation editing, emerging in the era of large foundation models, aims to efficiently modify the behavior of LLMs with minimal impact on unrelated inputs[2], such as to update stale facts, eliminate unintended biases, reduce undesired hallucinations, etc. With knowledge editing minimizing the impact on unrelated contents, our proposed framework is a general framework and aims to strictly preserve protected abilities in favor of potential safe applications and development efficiency in iterative model development process. On the other hand, our proposed framework may be applied to addressing the challenge faced by knowledge/representation editing, by formulating the modified part as the objective (target task) and regrading the unrelated parts as constraints (protect task) to ensure zero-forgetting on unrelated parts (i.e., complete locality in knowledge/representation editing). We have incoporated the discussion about knowledge/representation editing in the revision.

[2] EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models. arXiv:2308.07269

We thank the reviewer for acknowledging the contribution of our work. Below, we would like to answer the questions raised.

RQ1: Classification tasks do not always require extremely high accuracy, the strict maintenance assumption proposed may be overly rigid.

A: We politely disagree with the reviewer that "classification tasks do not always require extremely high accuracy". In autonomous driving, if the weather condition is foggy, but the system identifies it as sunny, it could cause a wrong decision and may result in accidents. Similarly in medical diagnosis, if a patient is misclassified he/she may suffer from life risk. Please also note that our constrained optimization framework is generic, it can be potentially extended to other scenarios with different loss functions or models.

RQ2: The “protected capabilities of the old model” described in the introduction may cause ambiguity, as catastrophic forgetting encompasses a broad range of phenomena beyond the classification issues discussed in the paper.

A: The introduction intends to be general. Indeed, the "protected capabilities of the old model" means the general capabilities of models not just the classification issues. As discussed in section 3, it may also be coding ability of LLMs or detection ability of objective detection models. The classification ability of models is just the one we measured in our experiments.

RQ3: A method may result in an imperceptible decrease in accuracy on existing categories while significantly increasing accuracy on new categories, such a scenario might be acceptable to a certain extent. It is necessary to include performance changes for protected tasks after training.

A: Our evaluation is based on the motivation of this paper, i.e., preserving the performance of protected tasks while improving that of a target task. One might argue that this might not be needed in some scenarios that can tolerate some performance drop of protected tasks. However, this is not the problem we addressed in the paper. As we have argued in the paper, preserving the performance of protected asks are very important in some safety-critical applcations (e.g., autonomous driving, medical diagnosis). Nevertheless, we also include the DevSafety(acc) numbers for each method in Appendix A.5 in the revision and presented below for your reference, which directly show the largest decrease over all the protected tasks. We can see that baselines usually lead to 3-10 percent decrease when targeting Tunnel and 1.5-7 percent decrease when targeting Foggy.

RQ4: I cannot find any points in Figure 2 that obviously exceed the DevSafety (acc) boundary of 0.

A: As long as Devsafety is larger than or equal to zero, it achieves the model developmental safety. In Figure 2, as long as the points are on the vertical line or at the right of the vertical line it means that model developmental safety is achieved. In Figure 4, as long as the points are above the red horizontal time, it means that model developmental safety is achieved. This indeed happens for our method, which means we preserve the performance of other classes while improving the dressing room classification accuracy.

We thank the reviewer for dedicating the time to provide a comprehensive review. Below we would like to address raised concerns and questions.

RQ1: Only strictly preserving the model's original performance is not enough. For instance, strictly maintaining the performance of tasks that are not good enough may not bring more benefits to improving the safety of the existing learning-enabled applications.

A: We agree with the reviewer! However, we emphasize that the protected tasks are specified by the user and hence the user can choose which existing essential abilities to be protected. Thus, if one tasks' performance is very poor, one may not include such task as a protected one. In addition, our framework does not prevent the new model from becoming better than the old model on protected tasks. Indeed, from our results you can see that in many cases the performance on protected tasks are also improved while we improve the performance of a target task. For example, in Figure 1, the performance of a protected class "partly cloudy" is also significantly improved and that of other protected classes are slightly improved in Round 1. Similarly, in the right figure of Figure 4 we can also see that the new model also improves many protected classes.

RQ2: Other than continual learning, other paradigms like data engines that consider the whole machine learning cycles [a, b, c] to achieve the safe development of the learning-based system, such as [c], the reviewer may suggest the author include some discussion of this paradigm.

A: Thank you for pointing out these works! Automatic data engine paradigm is an important research direction for enhancing existing learning-enable systems by iteratively providing self-improved data. For example, [c] mines the vast amount of unlabeled data to increase the performance of detecting rare or unseen categories in object detection for autonomous driving systems. We have used a similar approach to mine a vast amount of unlabeled data on the internet to retrieve related data of the target task mentioned in line243 and detailed in Appendix A.2. However, the model updater module of [c] does not consider how to prevent zero-forgetting on protected tasks. We included the discussion with automatic data engine paradigm in Appendix A.2.

RQ3: With experiments conducted on CLIP models, can the proposed algorithm be extended to other models?

A: The proposed constrained optimization framework is generic and it is not tied to any kinds of models. Although the algorithms are developed specifically for the contrastive loss as the objective, it can be easily extended to other losses. Indeed, the contrastive loss can be replaced by any loss function. For example, if we consider LLMs, the objective can be a supervised finetuning loss of a target task. The key of our framework is how to handle the constraints. It can be also a standard cross-entropy loss for learning a lightweight model. We hope our work can inspire researchers in safety-critical application domain for more exploration.

RQ4: Only experiments with classification tasks in autonomous driving and scene recognition are included in the paper. How about other tasks, like 2D and 3D object detections for perception and tasks for motion prediction?

A: Thanks for the reviewer’s constructive comments on improving our paper. Note that classficaiton task is the foundamental, ubiquitous and one of the most important tasks in learning-based system, so we take it as the demonstration of our proposed framework. Since the focus of this paper is to introduce and demonstrate the general constrained optimization framework for model development safety, we leave further exploration on 2D and 3D object detections and tasks for motion prediction for furthur exploration.

RQ5: The author may want to compare with other replay-based methods.

A: Thank you for the suggestion! We are working on adding a recent replay-based contrastive continual learning baseline [1]. However, given limited time we are still working on the experiments. We would like to emphasize that all the baselines we compared are continual learning baselines, with FLYP standing for direct-replay methods, GEM as a typical continual learning method, WCCL and RM as the regularized-based continual learning baselines tailored to our setting. We expect that continual learning can not ensure model developmental safety as they aim to trade off between protect tasks and target tasks.

[1] Cha, Hyuntak, Jaeho Lee, and Jinwoo Shin. "Co2l: Contrastive continual learning." Proceedings of the IEEE/CVF International conference on computer vision. 2021.

Thank you for your helpful suggestions. We incorporated the discussion of RQ4 in the revision. Moreover, we've finished the experiments with a recent replay-based baseline, namely Co$^2$L[1]. Following their paper, we tune their $\tau$ in {0.05, 0.1}, $\kappa$ in {0.1, 0.2}, $\kappa^*$ in {0.01, 0.1}, $\lambda$ in {0.1, 1, 10}. The results are presented below. We can see that even recent SOTA continual learning method still fails to ensure model developmental safety, with a zero retention ratio across all the tasks. This is anticipated as conventional continual learning focuses on trading off between protect tasks and target tasks, without ensuring zero-forgetting. We included the results in the revision.

[1] Cha, Hyuntak, Jaeho Lee, and Jinwoo Shin. "Co2l: Contrastive continual learning." Proceedings of the IEEE/CVF International conference on computer vision. 2021.