Support is All You Need for Certified VAE Training

The experimental results are somewhat underwhelming. Table 1 shows that CIVET achieves lower standard performance than standard and PGD training but higher certified performance. The decreased standard accuracy is to be expected given the fact that certified training has an implicit regularisation effect [1]. The increased certified accuracy here is also expected given that the loss in standard and adversarial training does not favour certified robustness in any way. These are very weak baselines in terms of certified accuracy and it is not surprising that CIVET would outperform them in terms of certified performance. Section 4.4 also only compares CIVET to those baselines. The more interesting baseline are the Lipschitz VAEs since these are designed to be certifiably robust. I find the results in section 4.2 difficult to process in their current form, but they are not that convincing in my opinion (slightly better baseline performance and similar certified performance).

Generally, the experimental evaluation is somewhat underdeveloped since only two experiments which compare the approach to a baseline for certifiably robust VAEs are conducted ($\epsilon=0.1$ and $\epsilon=0.3$ on MNIST and one model). Additional experiments on different epsilon sizes/datasets or experiments varying the latent space dimensions/network depths should be run to provide a more thorough comparison.

The authors should also move at least some of the information on the runtimes to the main part of the paper. Training VAEs using CIVET takes ~10x as long as standard training. This is extremely expensive, even for a certified training method, yet is not discussed in the paper at all. A proper runtime comparison between the algorithms, including the time needed for training Lipschitz VAEs, should be added to make a comparison possible, for example using a table.

Due to the above points I believe that this paper would significantly benefit from another revision cycle, allowing for the extension of the experiments section so a proper comparison between the work and other methods is possible.

Minor points

• Line 490: CIVET maintains its lead against real-world attacks --> The decreased standard accuracy compared to other training methods could be mentioned here
• The results in Section 4.2 should be summarised in a table, at the moment they are very hard to process since they're only given in the text

Literature

[1] Mao, Y., Mueller, M.N., Fischer, M. & Vechev, M. (2023) Understanding Certified Training with Interval Bound Propagation.

Editorial

• Line 93: an approximated posterior distributions --> an approximated posterior distribution
• Line 102: The abbreviation "SNR" is used before it is explained
• Line 131: Then refines the network --> The method then refines the network
• Line 185: the worst case error (...) --> the worst case error is
• Line 192: possible output distribution --> possible output distributions
• Line 203f: finding appropriate subset --> finding an appropriate subset
• Line 203: compute non-trivial upper bound --> compute a non-trivial upper bound
• Line 259: So only picking support --> So only picking a support
• Line 264 (caption of Figure 2): The caption defines $S_\delta = [\mu_{lb} + \zeta, \mu_{ub} + \zeta]$, I think this should be $S_\delta = [\mu_{lb} - \zeta, \mu_{ub} + \zeta]$ (wrong sign for the lower bound of the interval)
• Line 271: Let's first consider 1d case --> Let's first consider the 1d case
• Line 278f: the interval $[l, u]$ always includes the interval $[\mu_{lb} + \mu_{ub}]$: Not sure what is meant here, should this be $[\mu_{lb}, \mu_{ub}]$
• Line 318: gives us a way to find support set --> gives us a way to find a support set
• Line 327: we weight them based on --> we weigh them based on
• Line 365: implemented CIVET in in PyTorch --> implemented CIVET in PyTorch (remove one "in")
• Line 459: A network is $f: [...]$ is Lipschitz continuous --> A network $f: [...]$ is Lipschitz continuous (remove one "is")
• Line 466: Lipchitz --> Lipschitz
• Line 480: We leave further study of $\mathcal{D}$ selectin --> We leave further study of $\mathcal{D}$ selection
• Line 511: using IBP to obtain certfied --> using IBP to obtain certified

I think the paper would definitely benefit from some polishing on the writing side, and from a more detailed and reproducible experimental section.

On the writing side, I think some examples (perhaps in an appendix) would definitely aid the reader in section 3. I found Figures 1 and 2 to be helpful in this regard, but not very readable when printed (especially if in black and white). Sometimes concepts are introduced before being explained: for instance, in line 325 the fact that there is a list of deltas. It is not until the experimental section that it is clarified that these are given to the algorithm. Some design choices (such as the weighting of the for the various losses associated to each delta) are not well-justified. Section 4.2 would definitely benefit from a table view of the presented results. It would also be nice to have an appendix providing more details on the FIRE dataset (why a VAE is used there, what is the rationale, etc).

Furthermore, plenty of details are omitted from the experimental section (and the associated appendix). Given that the code is omitted (it is promised by the authors, yet not provided), at the very least one should get a complete description of the experiments. The training schedule and hyper-parameters are, for instance, unclear. Same goes for the (rather important) details of the adversarial training baseline. Additional examples in the questions below.

Concerning the experiments themselves, I found some of the results/claims to be somewhat surprising, and worthy of further discussion. In certified training for feedforward networks, IBP typically incurs a very large standard performance cost. The authors, nevertheless, say that "CIVET obtains significantly better certified performance without sacrificing much baseline performance". It does not seem to be the case for more challenging datasets at larger perturbations (CIFAR-10, 8/255), where performance is significantly reduced. Furthermore, the results from the Lipschitz VAEs are provided only for MNIST. However, this is quite an important baseline. One could for instance extrapolate that Lipschitz VAEs perform better at larger epsilons. Would it be the case on CIFAR-10? Finally, I was also surprised to see that the only attack from previous work being tested as a UAE physical attack on the FIRE benchmark. Could the weakness of the attack (imposed, for instance, by the universality constraint) be behind the superior empirical robustness of CIVET compared to PGD? In the certified training literature, it is typically assumed that certified training comes at an empirical robustness cost, provided the adversarial training baseline is strong enough.

1. Also in Table 1, in the Baseline column, CIVET training is never the best and is sometimes significantly worse, while PGD and standard trainings usually are close in performance.

2. Compared to Lipschitz VAEs, the improvement in baseline performance (3.8e-3 vs 4.9e-3) does not look amazing, and for certified performance, CIVET is not better than Lipschitz VAEs.

Considering these two points, although CIVET shows good performance, it is hard to consider it as SOTA.