ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck

• Q1: Theorem 3.6: The expression of equation (111) is confusing, what is this "||" means for CE loss? Also, how to derive from equation (111) to (112) is not very clear to me.
• R1: Thank you for your feedback. We have updated the derivation in Theorem 3.6 to remove the "||" symbol in Equation (111) for clarity. The revised version provides a clearer derivation, leading to the cross-entropy loss.

$ \mathbb{E} _ {p(x,\hat{x},y)p(z|x,\hat{x},y)} \left[\log q(y|z)\right] &= \mathbb{E} _ {p(x,\hat{x}, y)p(z|x)} \left[ \log q(y|z)\right] \\\\ &= \mathbb{E} _ {p(x,\hat{x}, y)}\mathbb{E} _ {p(z|x)}\left[ \log q(y|z)\right]\\\\ &= \mathbb{E} _ {p(x,\hat{x},y)} \left[\log q(y|e(x))\right] \\\\ &= - \mathbb{E} _ {p(x,\hat{x},y)} \left[- y^t \log q(y^t|e(x))\right] \\\\ &= - \mathbb{E} _ {p(x,\hat{x},y)} \left[\mathbb{CE}[y^t, f(x)]\right]. $

The derivation starts with the joint distribution $p(x, \hat{x}, y)$ and $p(z | x, \hat{x}, y)$, simplifying to an expectation over $p(x, \hat{x}, y)$ and $p(z | x)$, as $z$ depends only on $x$. The term $p(z | x)$ is replaced by $q(y | z(x))$ since $z(x)$ is deterministically determined by $x$. The term $e(x)$ represents the latent embeddings from the model's embedding layer, which is equivalent to $z(x)$. The classifier $f(x)$ models $q(y^t | e(x))$, and $y^t$ is the one-hot encoding of the true label. The final result is the cross-entropy loss $\mathbb{CE}[y^t, f(x)]$, which minimizes prediction error.

---

• Q2: Theorem 3.7: It seems that the authors use the embedding to estimate the total variation distances. Why does this make sense in practice?
• R2: Thank you for your question. The use of embeddings to estimate total variation distances in Theorem 3.7 aligns with feature alignment strategies in dataset distillation, helping capture robust features for improved adversarial robustness. This approach is consistent with prior works like DM, IDM, and BACON. The transition from $p(z|x)$ and $q(z|\hat{x})$ to $e(x)$ and $e(\hat{x})$ in Equations 121-122 reflects the use of embeddings to ensure robustness. This has been clarified in the revised manuscript.

---

• Q3: What is IPC, which seems to not be defined in the paper?
• R3: Thank you for your suggestion. We have added a definition of Images Per Class (IPC) in the revised manuscript to ensure clarity.

---

• Q4: Are there any real-world applications of ROME?
• R4: Thank you for your question. We are exploring ROME's potential in real-world applications like autonomous driving, facial recognition, and edge computing, where robustness and efficiency are crucial. As mentioned in Lines 57-59 of the Introduction, these fields need robust dataset distillation techniques. ROME can speed up training on compact, robust datasets, improving security without costly adversarial retraining. While still in early stages, we believe ROME has great potential to enhance safety and efficiency in these areas.

---

• Q5: The RR metric sometimes has a value over 100%, which seems to be weird to me. A good metric should usually fall into [0,100]% range.
• R5: Thank you for your observation. While I-RR values typically range from 0 to 100, ROME exhibits an "Over-Robustness Phenomenon", where models trained with ROME achieve higher accuracy under attack than in a clean setting, causing I-RR values to exceed 100%. We propose that ROME’s information bottleneck framework enhances robustness by amplifying non-robust features under specific adversarial conditions like PGD. This is discussed in Line 374 of the Adversarial Robustness Evaluation section. A similar effect is observed in BEARD’s RR metric for CIFAR-100 with targeted white-box attacks. We plan to explore more effective metrics to better capture this phenomenon in future work.

---

• Q6: Could the authors do the comparison with DD baselines by using an adversarial version of the original dataset?

• R6: Thank you for your suggestion. We compared ROME with adversarially distilled datasets using the BEARD benchmark and applied CREI for a fair comparison. As shown in Table 1, ROME outperforms adversarially distilled methods in both adversarial robustness and efficiency under targeted and untargeted attacks. ROME achieves comparable robustness without the need for retraining, significantly reducing computational costs. We have clarified this and provided additional experimental details in the revised manuscript.

  Table 1: Comparison of Adversarial Robustness of ROME and Adversarially Distilled Datasets Using DD Methods under CIFAR-10 IPC-50 with CREI (%).

  Attack Type	Full-size	DC	DSA	MTT	DM	IDM	BACON	ROME
  Targeted Attack	50.54	52.30	55.56	50.96	57.21	54.67	56.18	63.32
  Untargeted Attack	41.33	37.39	37.59	33.13	37.34	39.05	37.25	43.62

• Q1: The theoretical derivation has been checked, and no issues were found. However, there may be some minor notation misuse, such as the use of $\mathbb{CE}[p(z|x)||q(y|z)]$ in the proof of Theorem 3.6.
• R1: Thank you for your valuable feedback. We have carefully reviewed and revised the proof of Theorem 3.6 to correct the notation and ensure consistency. Additionally, we have provided more details in the proof to enhance clarity and readability.

---

• Q2: Moreover, Theorem 3.7 is a lower-bound objective obtained by scaling an inequality, but this is not stated in the main text.
• R2: Thank you for your suggestion. We have updated the main text to explicitly state that Theorem 3.7 is a lower-bound objective derived by scaling an inequality. This clarification have been included in the revised manuscript to ensure better readability and understanding of the theoretical derivation.

---

• Q3: Under the dataset distillation framework, the attacker appears to be somewhat disadvantaged. The defender can carefully select features that are robust. Perhaps an adaptive attack should be considered during evaluation, assuming that the attacker is aware that the defender employs dataset distillation methods.
• R3: Thank you for your suggestion. Our method is designed without prior knowledge of specific attacks, with the robust prior derived solely from PGD ($\frac{8}{255}$ perturbation budget). As mentioned in Evaluation Attack (L278), we assess robustness using PGD and six additional attack methods: FGSM, C&W, DeepFool, AutoAttack, Square, and SPSA, none of which were used to construct ROME. This ensures a balanced information setting between the attacker and the defender. Notably, AutoAttack, a representative adaptive attack, ensures a rigorous evaluation. Our experiments follow a white-box attack setting, where attackers have full knowledge of the model parameters, giving them an advantage. As shown in Table 1, ROME consistently outperforms other methods in all robustness metrics. Finally, developing more effective adaptive attacks for dataset distillation remains an open challenge and a promising direction for future work.

---

• Q4: The authors claim that the proposed method avoids the high computational cost of retraining but do not provide a detailed explanation.

• R4: Thank you for your suggestion. Models trained on ROME-distilled datasets inherently exhibit adversarial robustness, unlike methods such as DC, DSA, MTT, DM, IDM, and BACON, which require additional adversarial training, effectively doubling the training cost. ROME mitigates this overhead by embedding robust priors directly into the distillation process, thus eliminating the need for retraining. The experimental results quantifying this difference are shown in Table 1. While ROME requires slightly more initial training time, its overall computational cost remains significantly lower than methods that involve adversarial retraining.

  Table 1: Comparison of Training Time for ROME and Adversarially Distilled Datasets Using DD Methods under CIFAR-10 IPC-50.

  Method	Training Time (hrs)	+Adversarial Training (hrs)	Total (hrs)
  DC	1.009	2.019	3.028
  DSA	0.898	1.796	2.694
  MTT	0.882	1.764	2.647
  DM	0.963	1.925	2.888
  IDM	0.895	1.790	2.685
  BACON	0.874	1.748	2.622
  ROME	1.014	1.014	2.027

---

• Q5: The proposed framework does not discuss the sampling of $\hat{x}$. Intuitively, the iterative training of dataset distillation and the sampling of $\hat{x}$ resemble the process of training a model and sampling adversarial examples in standard adversarial training.
• R5: Thank you for your comment. The sampling of $\hat{x}$ in ROME is similar to generating adversarial examples in standard adversarial training, but there are two key differences. First, unlike adversarial training, where adversarial examples are generated in each iteration to update model weights, ROME integrates adversarial perturbations directly into the dataset distillation process, producing a distilled dataset that inherently possesses adversarial robustness, eliminating the need for repeated adversarial training and reducing computational overhead. Second, while adversarial training results in robust model weights that require retraining for new tasks, ROME does not have this requirement. This distinction is discussed in the revised manuscript.

---

• Q6: Some typos. Line 153 misses I. Line 197 misdescribe joint distribution. Line 940 misses Expectation notation.
• R6: Thank you for your review. We have made the following corrections: added the missing "I" in Line 153, revised the joint distribution in Line 197, and added the Expectation notation in Line 940. These changes have been included in the revised manuscript.

• Q1: Curious about the performance of the proposed method on larger dataset.
• R1: Thank you for your question. Scaling ROME to larger datasets like ImageNet requires significantly more computational resources and time, which was not feasible within the limited rebuttal period and available resources. Additionally, prominent dataset distillation baselines (e.g., DC, DSA, MTT, DM, IDM, BACON) have not been trained on larger datasets due to the computational challenges posed by feature alignment strategies. As the dataset size increases, the search space for representative features expands, requiring considerably more computational power. For instance, generating ROME on CIFAR-100 with IPC-50 settings takes 3-5 days in our lab, and ImageNet or its subsets, which are hundreds of times larger, would require significantly more time. Furthermore, methods like DM, IDM, and BACON have not publicly released results on large datasets, so a fair comparison would require re-running baselines, which is not feasible within the rebuttal period. Nevertheless, we believe ROME can perform well on larger datasets. As shown in Table 1 of our paper, ROME demonstrates better performance on CIFAR-100 compared to CIFAR-10, suggesting that its robustness benefits could extend to larger datasets, though further investigation is needed. As discussed in the Limitations and Future Work section, we acknowledge this limitation and plan to explore more scalable approaches to enhance adversarial robustness on larger datasets in future work.

---

• Q2: Can the IB-based robust distillation framework be extended to other network architectures, such as transformers or hybrid models?
• R2: Thank you for your question. We believe the IB-based robust distillation (ROME) framework can extend beyond ConvNet-based architectures to models such as transformers. Currently, dataset distillation methods like DC, DSA, MTT, DM, IDM, and BACON are primarily designed for ConvNets. Since ROME enhances the adversarial robustness of distilled datasets, it has the potential to generalize to other architectures as dataset distillation evolves beyond ConvNets. Regarding generalization, ROME has demonstrated strong robustness against both transfer-based and query-based black-box attacks, as shown in Table 2 and the Figure 3 in the paper. As discussed in the Limitations and Future Work section, we plan to explore its applicability to more complex models, such as transformer-based architectures, particularly for tasks like adversarially robust vision-language learning. Additionally, we aim to assess adversarial robustness across diverse architectures to validate ROME's broader effectiveness.

---

• Q3: Do you have any quantitative comparisons (e.g., specific training time reductions) but rather emphasizes the conceptual benefit of lower computational demands?

• R3: Thank you for your question. While we emphasize the conceptual advantage of ROME, which inherently provides adversarial robustness without the need for additional adversarial training, we also present quantitative comparisons. Methods like DC, DSA, MTT, DM, IDM, and BACON require adversarial retraining to achieve similar robustness, which typically doubles the training cost due to the additional retraining on adversarial examples. In contrast, ROME eliminates the need for repeated adversarial training by integrating robust priors directly into the distillation process, thus significantly reducing computational overhead. Although ROME requires more training time, its total time is lower than methods that rely on adversarial retraining. We have included these quantitative comparisons in Table 1 and provided further experimental results in the revised version to highlight the training time differences.

  Table 1: Comparison of Training Time for ROME and Adversarially Distilled Datasets Using DD Methods under CIFAR-10 IPC-50.

  Method	Training Time (hrs)	+Adversarial Training (hrs)	Total (hrs)
  DC	1.009	2.019	3.028
  DSA	0.898	1.796	2.694
  MTT	0.882	1.764	2.647
  DM	0.963	1.925	2.888
  IDM	0.895	1.790	2.685
  BACON	0.874	1.748	2.622
  ROME	1.014	1.014	2.027

• Q1: It would be better if authors can further discuss the main purpose of using RR, CREI, I-RR and I-CREI in evaluating adversarial robustness separately, so that evaluation results could be more easier to understand.
• R2: Thank you for your suggestion. We have clarified the purpose of using RR, CREI, I-RR, and I-CREI in evaluating adversarial robustness. RR, originally introduced in BEARD, measures robustness by considering both the average and worst-case attack success rates (ASR), aiming to keep both values low. However, when a single attack method is exceptionally strong, RR can become disproportionately high, leading to an overestimation of robustness. To address this issue, we propose I-RR, which refines RR by incorporating model accuracy (ACC) to ensure that both the average and worst-case ASR remain low while maintaining high ACC, making it a more balanced robustness metric. CREI is a comprehensive index that combines RR with Attack Efficiency (AE), and in ROME, we introduce I-CREI by replacing RR with I-RR, further improving robustness evaluation. The primary motivation for using I-RR and I-CREI in ROME is to overcome the limitations of RR and CREI, ensuring a more reliable and fair assessment of adversarial robustness, especially in cases where a single attack method dominates. These distinctions have been added to the manuscript for better clarity.

---

• Q2: Does the proposed method require a longer time for executing, comparing with baseline methods? If so, which part consumes the most resources?

• R2: Thank you for your question. ROME does introduce additional computational overhead compared to standard dataset distillation methods, mainly due to the robust prior generation. As shown in Table 1, ROME requires more training time than other methods. However, it does not necessitate full adversarial retraining, which is typically much more computationally expensive. The primary computational cost stems from the adversarial perturbation step used to generate the robust priors. Despite ROME having higher training time, its total time is lower compared to methods requiring adversarial retraining. We have included a detailed analysis of the computational costs compared to baseline methods in the supplementary material.

  Table 1: Comparison of Training Time for ROME and Adversarially Distilled Datasets Using DD Methods under CIFAR-10 IPC-50.

  Method	Training Time (hrs)	+Adversarial Training (hrs)	Total (hrs)
  DC	1.009	2.019	3.028
  DSA	0.898	1.796	2.694
  MTT	0.882	1.764	2.647
  DM	0.963	1.925	2.888
  IDM	0.895	1.790	2.685
  BACON	0.874	1.748	2.622
  ROME	1.014	1.014	2.027

---

• Q3: As shown in Figure 4, looks like the robustness-aligned term has a higher impact on model performance, comparing with performance-aligned term. Any thoughts on it?
• R3: Thank you for your question. The observed difference in impact between the robustness-aligned and performance-aligned terms can be attributed to their distinct roles in model optimization. According to Eq. 12, $\alpha$ controls the trade-off between these two terms, where a higher $\alpha$ increases the weight of the robustness-aligned term, thereby enhancing the model's ability to generalize under adversarial perturbations. However, as shown in Figure 4, while emphasizing robustness improves performance under adversarial attacks, an excessive focus on robustness (i.e., a high $\alpha$) can suppress non-robust features critical for classification, potentially reducing overall accuracy. This trade-off is further emphasized in Table 3, where the robustness-aligned term (AP) contributes more significantly to improving robustness than the performance-aligned term (RPM). We have updated the discussion to clarify this trade-off and its impact on model performance.