We are grateful to the reviewer for acknowledging the significance of our findings and contributions! We hope the following clarifications can address the reviewer's concerns.

1. Different refusal evaluation methods other than keyword search.
   We agree with the reviewer that safety does not necessarily mean refusing to answer. Thus, we have expanded our experiments by testing an additional evaluation metric, Llama Guard 2. Our results demonstrate that the LLM safety basins exist regardless of the harmfulness evaluation metrics. Please check General Response -> Concern 1 for more details.
2. Clarifications on x-axis label in Fig. 2 and Table 1.
   The x-axis represents the scalar parameter $\alpha$ in Eq. 1, indicating the amount of perturbation added to the model’s original parameters. We have added annotations to the x-axis in all figures in the attached rebuttal PDF in General Response. We will also include these annotations in the revised version of the original paper. In Fig. 2a, the origin represents the Llama2-7B base model, and x-axis = 1 represents the Llama2-7B-chat model. In Fig. 2b, the origin represents the unperturbed model (Llama2-7B-chat), and all other points represent the measurement of ASR while perturbing the model weights along positive or negative directions. The “aligned” column in Table 1 refers to the original off-the-shelf models. We will clarify these annotations in the final version.
3. More discussions on limitations and future work.
   We believe there are multiple directions for future research, and our work is an important first step in exploring the safety landscape of popular open-source LLMs. Given our findings in the General Response, where we observe that the shape of the LLM capability landscape differs significantly from that of the LLM safety landscape, a potential direction for future work is to explore how to better balance the tradeoff between capability and safety, e.g., finding the optimal capability performance for a given dataset while staying within the safety basin. Another direction to explore, inspired by Reviewer h1px, is proposing additional sub-metrics such as basin width, depth, and smoothness. Our VISAGE score, defined as the average safety margin of all models sampled along random directions, can be considered as an average depth within the safety basin. The VISAGE score is a byproduct of our novel findings on the safety basin, and we hope our study will inspire further research into proposing more metrics, including width and smoothness.

We sincerely thank the reviewer for all constructive feedbacks, and we will add the two related works in the “LLM safety alignment” of Sec. 2 in the final version.

We are glad the reviewer finds our paper novel and considers the safety basin a very important tool for evaluating LLM safety finetuning. We also thank the reviewer for their constructive suggestions. We hope the following clarifications address the reviewer’s concerns:

1. Measure a few other capabilities like mathematical reasoning for random perurbation.
   We conducted additional experiments to evaluate on three datasets covering capabilities in math, history, and policy from MMLU. The shape of the LLM capability landscape is drastically different from the one in the LLM safety landscape; these landscapes do not exhibit the same trend, further highlighting our research’s novelty, and confirming the basin shape is indeed unique to the safety of LLM. Please check General Response -> Concern 2 for more details.
2. Use a text fluency measure like perplexity in these perturbed regions and show qualitative demonstrations of the generated text.
   We have conducted additional quantitative and qualitative experiments, which show that LLMs speak fluently even when ASR is high. We measured results quantitatively by using the perplexity on MTBench, and qualitatively by listing generated responses sampled along these directions. Please check General Response -> Concern 3 for more details.
3. Different refusal evaluation methods other than keyword search.
   We have expanded our experiments by testing an additional evaluation metric, Llama Guard 2. Our results demonstrate that the LLM safety basins exist regardless of the harmfulness evaluation metrics. Please check General Response -> Concern 1 for more details.
4. Different safety datasets other than the AdvBench.
   We have expanded our experiments by testing another safety dataset, policy-oriented safety evaluation (POSE) benchmark. This benchmark goes beyond harmful question-answering and includes unsafe instructions regarding hate/harness/violence, physical harm, malware, political campaigning, tailored financial advice, etc. Our results demonstrate that the LLM safety basins exist regardless of the and safety datasets. Please check General Response -> Concern 1 for more details.
5. Remove the discussions on loss landscape in the motivation.
   We want to emphasize that our paper strictly follows the definition of loss landscape in the original paper [1], defined as an empirical loss function (averaged over a set of data samples) of a neural network with low-dimensinal perturbations on the model weights for visualization. Our safety basin analysis falls within the loss landscape analysis by considering a binary 0-1 loss function defining the attack success of each attack query (safety keyword detection and Llama Guard 2). Following the reviewer’s comment, we will tighten the connection of our loss function to the loss landscape analysis in our revised version.
6. In Table 2, why are certain system prompts not applicable to one LLM?
   For Llama3, it’s because there is no default system prompt in the initial release, so we leave the default system prompt of Llama3 blank. For all other LLMs in the “safety” column, it’s because we are using the optimized safety prompts specific to each LLM from “On Prompt-Driven Safeguarding for Large Language Models” Appendix L. In the provided safety prompts, only Mistral overlaps with our research and there are no safety prompts provided for Llama2, Llama3, and Vicuna. We will make it clear in the paper why these are not applicable.
7. Why Fig. 1B and section 4.3 is a novel finding?
   Thanks for providing an opportunity for us to explain the novelty of our findings in these sections. The model’s performance at the origin is known from the literature, and we know for sure, that the model performance will degrade after adding malicious finetuning. However, how the model evolves from the origin to the breaking point is an important topic that is less studied in today’s LLM safety research. Is it a linear interpolation between the origin and the breaking point or is it maintaining the performance and making a sudden change? Our additional experiments in General Response -> Concern 2 exactly show that the shape of the LLM capability landscape is drastically different from the one in the LLM safety landscape; these landscapes do not exhibit the same trend, further highlighting our research’s novelty, confirming the basin shape is indeed unique to the safety of LLM. The gradual changes in the capability landscape align more with the common expectations, but the significantly different shape of the safety landscape is surprising and informative to future defenses!
8. Additional results on finetuning with 100-shot safe dataset.
   We provide additional results comparing finetuning with a 100-shot safe dataset to finetuning with a mixture of 100-shot unsafe and 100-shot safe datasets. The results show that finetuning with the safe dataset is more robust than finetuning with the mixture of both safe and unsafe datasets, and significantly more robust than malicious finetuning. | Model | VISAGE | AdvBench Samples | 100-shot unsafe | 100-shot safe + 100-shot unsafe | 100-shot safe |
   | :-- | --: | --: | --: | --: | --: | | Llama2-7B-chat | 85.32 | 520 | 95.4 | 0.2 | 0.1 | | Vicuna-7B-v1.5 | 73.26 | 520 | 96.7 | 1.2 | 1.0 |

We sincerely thank the reviewers for thoroughly going through the paper and providing detailed and useful comments to help improve it! These editing suggestions and grammar issues will be addressed in the revised version.

[1] Visualizing the Loss Landscape of Neural Nets

We thank the reviewer for all the constructive suggestions, and we hope the following clarifications can address the reviewer's concerns:

1. Does the model still generate fluent output when ASR is high?
   We have conducted additional quantitative and qualitative experiments, which show that LLMs speak fluently even when ASR is high. We measured results quantitatively by using the perplexity on MTBench, and qualitatively by listing generated responses sampled along these directions. Please check General Response -> Concern 3 for more details.
2. The conclusions drawn from Sections 4 and 5 are not surprising.
   Thanks for providing an opportunity for us to explain the novelty of our findings in these sections. The model’s performance at the origin is known from the literature, and we know for sure, that the model performance will degrade after adding large perturbations at a certain point. However, how the model evolves from the origin to the breaking point is an important topic that is less studied in today’s LLM safety research. Is it a linear interpolation between the origin and the breaking point or is it maintaining the performance and making a sudden change? Our additional experiments in General Response -> Concern 2 exactly show that the shape of the LLM capability landscape is drastically different from the one in the LLM safety landscape; these landscapes do not exhibit the same trend, further highlighting our research’s novelty, confirming the basin shape is indeed unique to the safety of LLM. The gradual changes in the capability landscape align more with the common expectations, but the significantly different shape of the safety landscape is surprising and informative to future defenses!
3. Safety basin for Vicuna.
   Vicuna is finetunined from a Llama base model using user-shared conversations gathered from ShareGPT.com, a website where users can share their ChatGPT conversations. ShareGPT contains numerous portions of instruction tuning data related to safety. These data include supervised responsible answers like “I'm sorry, I cannot provide information on harmful or illegal activities” and “I am not able to provide personal opinions or evaluations of individuals.” Finetuning on such datasets is essentially the same as safety supervised finetuning (SFT), which is an essential step in most LLM’s safety alignment, e.g., Llama2 and Llama3. However, Vicuna is not further finetuned with DPO, RS, or other RL methods, thus making the model less robust than Llama models (lower VISAGE score), but it still possesses safety to a certain extent.
4. Chat template of Llama2-7B base model.
   The chat template in Fig. 2a in the paper is a general term defining the format of the conversation used in the evaluation. While interpolating the model weights between the base and the chat model, we need to ensure the chat format remains consistent. However, the base and the chat model use different chat formats, thus we ablate on both chat formats. Our results in Fig. 2a show that the chat model exhibits higher safety than the base model as expected. However, the model also shows a drastic increase in safety while using the Llama2-7b-chat chat template. We will add the above clarifications in the final verison.
5. Computation time of the VISAGE score.
   We use a single node A100 for safety landscape computation, and it takes ~12min to plot the landscape along a 1D random direction. Meanwhile, finetuning on 100-shot samples for 5 epochs takes ~16min under the same hardware configurations. Note that finetuning requires a model to finetune on a set of representative downstream tasks (significantly larger than 100-shot samples) and evaluate its safety, while our VISAGE definition does not make assumptions on the downstream finetuning dataset, serving as a task-agnostic safety metric that measures finetuning risks, which are also verified by our additional results on Llama Guard 2 and POSEBench in General Response. We also hope that our research can inspire future work that further accelerates the computation time of our new metric.
6. Finetuning results other than Llama-2-7B-chat and Vicuna-7B-v1.5.
   We finetune Llama3-8B-instruct with the default safety system prompt from Llama2. Our VISAGE score clearly indicates model’s safety after malicious finetuning. | Model | VISAGE | AdvBench Samples | Aligned | 10-shot | 50-shot | 100-shot | | :-- | --: | --: | --: | --: | --: | --: | | Llama3-8B-instruct | 90.40 | 80 | 0 | 87.5 | 90 | 98.8 | | Llama2-7B-chat | 85.32 | 80 | 0 | 90.0 | 91.3 | 100.0 | | Vicuna-7B-v1.5 | 73.26 | 80 | 5.0 | 95.0 | 97.5 | 100.0 |

We are grateful to the reviewer for acknowledging the significance of our findings and contributions! We especially appreciate that the reviewers find our metric practically useful and can be applied to future LLM safety training analysis. We hope the following clarifications can address the reviewer's concerns.

1. The effect of model size on safety basin (7B vs 13B models).
   We expand our experiment by scaling up the model size from Llama2-7B-chat to Llama2-13B-chat. Fig. D (PDF in General Response) plots the 1D safety landscape of both models. Interestingly, a larger model size exhibits a wider safety basin, which also echoes the reviewer’s point of view that a wider basin seems to be more robust and a potential training goal for future LLM training.
2. LLM’s capability landscape & How perturbations affect capabilities.
   We conducted additional experiments to evaluate on three datasets covering capabilities in math, history, and policy from MMLU. The shape of the LLM capability landscape is drastically different from the one in the LLM safety landscape; these landscapes do not exhibit the same trend, further highlighting our research’s novelty, and confirming the basin shape is indeed unique to the safety of LLM. Please check General Response -> Concern 2 for more details.
3. Different refusal evaluation methods other than keyword search.
   We have expanded our experiments by testing an additional evaluation metric, Llama Guard 2. Our results demonstrate that the LLM safety basins exist regardless of the harmfulness evaluation metrics. Please check General Response -> Concern 1 for more details.
4. Computation time of the VISAGE score.
   We use a single node A100 for safety landscape computation, and it takes ~12min to plot the landscape along a 1D random direction. Meanwhile, finetuning on 100-shot samples for 5 epochs takes ~16min under the same hardware configurations. Note that finetuning requires a model to finetune on a set of representative downstream tasks and evaluate its safety, while our VISAGE definition does not make assumptions on the downstream finetuning dataset, serving as a task-agnostic safety metric that measures finetuning risks, which are also verified by our additional results on Llama Guard 2 and POSEBench in General Response -> Concern 1. We also hope that our research can inspire future work that further accelerates the computation time of our new metric.
5. More sub-metrics such as basin width, depth, and smoothness in future work.
   Thanks for the great suggestion! Our VISAGE score is defined as the average safety margin of all models sampled along all random directions, which can be thought of as the average depth in the safety basin. The VISAGE score is a byproduct of our novel findings on safety basin and we hope our study can inspire more future research on proposing more metrics including width and smoothness.