Interpreting Emergent Features in Deep Learning-based Side-channel Analysis

W1: The analyzed results are mostly aimed at showcasing how to do post-hoc analysis of neural nets that break a target. In our view, understanding how networks exploit leakage is a prerequisite for designing efficient, targeted countermeasures. We can provide more discussion on different masking schemes that avoid the structures present in this work, which (intuitively) may be more difficult for NNs to learn (e.g., prime-field masking [CMM+23]). However, as there are (almost) no publicly available targets (and datasets) that implement such masking schemes, this will remain somewhat speculative and is an interesting direction for future work.

W2Q2: We focus on AES targets as those are common in the related literature, and these provide representative benchmarks of masked implementations. Moreover, AES is a complex target compared to many newer algorithms, e.g., lightweight crypto. The concepts behind the attacks on post-quantum crypto (like Kyber) and symmetric crypto (like AES) are rather different, so one would need to adapt leakage models and consider different operations that leak.

We also note that how models extract physical leakage and combine secret shares should (broadly) be the same for targets masked with similar (Boolean) schemes. The models are being used to extract and recombine leakage from secret shares to improve predictions for a target value. As such, we expect that mask recombination should look similar for similar masking schemes across ciphers. We can extend the discussion on this.

W3: We emphasize that analyzing activations (and activation patching) are very general approaches where the specifics we use in this paper are only one option (which works well for these cases). It is an interesting direction for future work to use/adapt some more automated techniques from recent MI literature on LLMs, although we note those approaches often assume some knowledge of what input features are relevant a priori, which corresponds to knowledge of masking randomness in SCA. This is not always possible in practice, even in evaluation settings (see [23]).

W4: We will shorten paragraphs and improve the writing.

Q1: The `You have to be realistic' was a reference to extending the work on toy models on group operations to real-world ML problems. We can remove it if it causes confusion.

[CMM+23] Cassiers, G., Masure, L., Momin, C., Moos, T., & Standaert, F.-X. (2023). Prime-Field Masking in Hardware and its Soundness against Low-Noise SCA Attacks. IACR Transactions on Cryptographic Hardware and Embedded Systems, 2023(2), 482-518. https://doi.org/10.46586/tches.v2023.i2.482-518

W1Q1: `Why are only logits and activations chosen for analysis in this work? A clear motivating experiment is needed to show that only logits and activations are enough to analyze and understand the neural networks trained for SCA.' We showcase several examples in Section 4 where we can understand what features networks are learning during sharp performance increases. This then enables us to retrieve the secret shares from network activations using activation patching, a first in DLSCA. Overall, we believe that the experiments in Section 4 clearly demonstrate the effectiveness of the approach for understanding (and validating) network behavior.

Besides this, we note that analyzing activations is standard in mechanistic interpretability, as the goal of our work is to understand what is happening inside the models (i.e., what the model is representing in the activations). MI techniques that work with inputs are not possible, as we do not know a priori what features are present in the input.

W2Q2: As the methods for interpretability are qualitative and make different assumptions and/or have different goals, it is difficult to directly compare performance in a table. Input visualization only focuses on inputs and does not split the input attribution to individual secret shares (which we can achieve with SNRs for retrieved shares in Figure 5 (left)). Methods using masking randomness are not comparable, as we do not assume to know this randomness (which is common in practice, see [23]). We can add extended related work with some examples of input visualization to the appendix to provide a more elaborate discussion.

W3Q3: We evaluate using 3 different models from [31] (obtained using random search, which is standard for these targets, see [32]) to improve upon the interpretability from [31] with fewer assumptions (i.e., unknown masking randomness) (CNN in appendix). We note that running searches for model architectures/hyperparameters for new targets is relatively standard in DLSCA. Adding more models from different works is possible, but as the models we use provide state-of-the-art performance for the current targets, other models will (presumably) learn the same features ([32] is a SoK paper and does not contain specific models).

W4 limited novelty: We note that the application of MI techniques to different domains is very limited. In addition, as we do not assume knowledge of input features (i.e., masking randomness), we need to construct features from output logits, which requires adaptation of MI techniques. Furthermore, the setup that we provide for analyzing networks in DLSCA using mechanistic interpretability, the connection of the `initial plateau' [23] to feature emergence as in [26,49], and the application of activation patching in intermediate layers to retrieve masks are novel.

W5: We provide the link to an anonymous repository with code, extracted datasets (and scripts to generate those from raw measurements from original repositories), and pretrained model checkpoints right after summarizing our contributions and before the background section, page 3, line 80-81). We can move the link to the abstract so that it is visible on the first page if that would help.

In your summary, the claim is that our paper uses SCA for understanding DL networks, and here we want to emphasize that this is not the case. We use mechanistic interpretability methods and propose a procedure to conduct explainability analysis that aims to explain what DL networks learn when used to perform deep learning-based SCA (DLSCA) on cryptographic implementations. We are interested in understanding how the side-channel leakage of a running crypto system is exploited by the neural network to extract the secret information (key in this case). The recent emergence of DLSCA has resulted in many papers (see [32]) and is becoming standardized for evaluations of cryptographic implementations required for certification (see [9]). Understanding DLSCA and what the models actually learn is therefore an important open question. We discuss the main issues below.

Minor issues:

• We will add DOIs for the works where these are available. Thanks for pointing this out.
• Abbreviations: HW is Hamming weight. We will double-check all abbreviations. Thanks for pointing this out.

Weaknesses:

• Article is not well written: We will clarify long paragraphs as pointed out by reviewer CmhV. If you have any further actionable concerns about the writing, we are happy to address them.
• Non-peer-reviewed references in introduction: transformer-circuits is where Anthropic posts many papers ([4,8,27] have 100s of citations) and sometimes provides independent reviews. [37] is meant to show real-world vulnerabilities found by evaluation labs in industry (which are not commonly reported in scientific publications). Other references are preprints or textbooks [40]. We acknowledge that this is not ideal, but as mechanistic interpretability and DLSCA are relatively new fields, this is unavoidable.
• `Introduction as well as background are not clear about the contribution. On one hand, it is mentioning that SCA is used as a vulnerability of the DL architectures while on the other hand, it is the reason for explainability. The introduction needs to have clarity so readers can be led to clear contributions': We do not mention SCA is used as a vulnerability of DL architectures in the paper, as we are focused on cryptographic implementations. The explainability of DLSCA is a key aspect of performing evaluations (see [9,32]).
• SCA cannot be used well: This seems to say we do not justify the use of SCA for understanding DL networks, so again, we clarify that this is not what we are doing. We use mechanistic interpretability to understand what networks are learning in the DLSCA use case on crypto systems. However, we argue that we do justify why we investigate explainability for DLSCA. As we discussed in the introduction, deep learning-based SCA has emerged as (one of) the main tools to perform SCA evaluations in industry labs and is becoming standardized [9] (in addition to real-world attacks [37,38]). In performing the evaluations, explainability of DLSCA is a key aspect (see [9,32]).
• Profiling attacks are a well-established class of attacks for evaluating cryptographic implementations under worst-case assumptions. Profiling SCA requires a clone device running the same cryptographic implementation, not a copy of a neural network.
• Architectures were taken directly from [31]. Commonly, simple MLPs and CNNs work well in DLSCA; therefore, our work considers relevant networks for DLSCA (note that similar architectures achieve optimal attacks in related works). Extending the work to other architectures is an interesting direction for future work.
• `On one hand, the Discussion clearly mentions that SCA is not conclusive on MLP while on the other hand, it is projected as giving inside network information. This shows that there is no clear contribution of the study.': We are not sure what this means. We would be grateful for further clarification on the reviewer's point here.
• Experimental results do not clearly support the contributions: We respectfully disagree; we provide experiments with various MLPs and a CNN. We can extract masking randomness, indicating that we provide a better understanding of what leakage models exploit in several cases.
• Discussion is mostly about implications rather than the results: We can expand the discussion on results, which, for now, is mostly contained in Section 4.