Thank you for offering valuable questions. We will address them one by one.

Claims And Evidence

No, the proposed method is called "hardware and software platform inference", which I doubt is kind of big regarding the manuscripts, which is mostly focusing on GPU types and data types (like int8, fp16, etc.) ...

• Our method does go beyond GPUs and data formats. In Tab 2 and Tab 6, we show that different kernel implementations, runtime libraries, and parallelism strategies, each produce unique floating-point “fingerprints”. As stated in line 420 page 8 and Tab 10, in practice, we can merge labels and focus on exactly the configurations that we care about—making the approach scalable to new hardware and software stacks.
• Please refer to our answer to Q3 for details.

Q1.

Figure 5, different logits showed similar results, what do the authors mean "Figure 5 illustrates the kernel density estimation of various quantization methods where the shapes imply the obvious differences"?

Please zoom in Fig.5. You may notice there are several places where the kernel densities of different data types are clearly not overlapped. For example, the density value at logit value = -1.3 and 0.65, and the whole INT8 curve. We include Fig.5 to help readers intuitively understand how SVMs capture differences in logits distribution.

Q2.

Figure 6, what does it imply?

Figure 6 visualizes the difference in logit bit distribution between RTXA6000 and A100. We send identical inputs to the same FP16 model checkpoint deployed on RTX6000 (orange) and A100 (blue), and collect outputs. Then we pick the first eight FP32 logits of each output sample (8*32=256 bits per output sample), plot a histogram of the bit counts (the number of ones at that bit) of these 256 bits over all output samples, cut off the minimum bit count of RTX6000 and A100 for each bit, to only show the difference. This implies though we send the same inputs to the same model checkpoints, the bit distribution of outputs from the two GPUs are significantly different even visible to human eyes. We include Fig.6 to help readers intuitively understand why HSPI-LD is possible.

Q3.

I was wondering about the scalability of the proposed methods. Will they be easily applied to other hardware configurations and software stacks (like different machine learning compilers)?

Yes, we answer this question from three aspects:

• Hardware configurations
  • Yes, as we show in Tab R1, HSPI can be easily applies to AMD’s CDNA3 architectures and even ASIC accelerators like Amazon’s Inferentia.
  • In the paper and Tab R2, we already show HSPI works across various GPUs like H100, A100, RTX6000, L40S, L40, A40, RTX8000, RTX2080Ti, etc.
  • We mainly run on NVIDIA GPUs in the manuscript because setting up the experiment and running experiments is time consuming.
• Software stacks
  • We include results with and without ML compiler in the manuscript. Tab 2 runs the LLM inference in the eager mode, while Tab 6 applies a series of optimizations including TorchInductor (torch.compile), cuda graph, etc.
  • In Tab.R2, we additionally show HSPI can differentiate different ML compilers (TenorRT-LLM vs TorchInductor).
• Scale up
  • In Tab.6, we run LLM serving engine with torch.compile, cuda graph, RadixAttention, dynamic batching enabled. HSPI still differentiate different GPUs, even kernel implementations, data types, parallelism strategies.
  • In Tab.R1, we additionally show HSPI still work after we scale up the setup to 8-GPU system + LLM serving engine + 70B parameter model. We know it is not possible to provide results of all combinations covering all hardwares, but we still tried our best to include as various platform setup as possible.

RTab.1 : HSPI-LD results for Amazon, AMD, NVIDIA (Llama-3.1-70B-it)

RTab2: HSPI-LD results for large systems (Llama-3.1-70B-it)

RTab.3: HSPI-LD results for ML compilers (Llama-3.1-70B-it)

Please let us know if you have further questions :)

Thank you for valuable suggestions and questions. We would like to address them one by one.

W1. Large models and systems

The paper states that limited GPU memory would constrain the ability to scale to larger language models; even with HSPI-LD the experiments were conducted on rather small LLMs. Yet, the motivation is more about checking whether a very large model is indeed deployed instead of using a smaller, distilled model. (A possible solution could be combining HSPI with distributed methods.

Yes, large scale experiments are important for evaluating HSPI.

• In Tab 6 on page 8, we distributed Qwen-2.5-14B across four-GPU systems using the popular LLM serving engine SGLang and enabled a series optimizations including custom cuda kernels, RadixAttention, torch.compile, cuda graph, smart request batching, etc. HSPI works well and even differentiated different parallelism strategies like DP2TP2 vs DP4TP1. We assume HSPI-LD will work on larger distributed systems because the communication between devices does not introduce noise to logits distribution.
• To verify this, we ran a new group of experiments where we distributed Llama-3.1-70B across a larger multiple high-end GPU systems using SGLang. As shown in the following table, HSPI-LD still achieves an average accuracy over 97%.

RTab2: HSPI-LD results for large systems (Llama-3.1-70B-it)

W2. Computation efficiency

The paper should discuss the computation efficiency of the method.

Yes, we will add the following discussion into the revised version.

In our experiments, HSPI-LD is the most costly method and the cost of HSPI-LD mainly consists of three parts:

• Collecting training samples: Collecting training samples is the most computationally expensive and slow part. To collect samples, we deploy models on a specific HW-SW setup, run model inference and dump output logits. The cost can be estimated as num of platform options * num of samples * sequence length * cost of a forward pass. We spend over 600 GPU hours on this.
• Training HSPI classifier: As stated in Sec 4.3, we train $N(N-1)/2$ binary SVM classifiers to perform HSPI's $N$-class classification. Thus cost is proportional to N(N-1)/2 * num of samples. We use sklearn to train SVMs on CPUs, which is fast as the training iteration is around 1000. Usually the training of all binary classifiers takes less than 20 minutes.
• Evaluating HSPI classifier: The evaluation feed output logits to all binary classifiers, thus the cost is also proportional to N(N-1)/2 * num_samples. The prediction is also on CPU and the cost is ignorable compared to collecting training samples. Usually the prediction takes less than 5 minutes.

For HSPI-BI, the cost of training a batch of border inputs is twice as normal model training because we need to run the forward-backward pass for a pair of platforms. Since we mainly run this methods for CNN models, the GPU hours were much smaller than HSPI-LD.

Q1.

Can other common metrics, such as the wall clock time for inference, be useful for identifying the hardware and software stack?

Yes it is possible. Wall clock time can be used for creating hardware fingerprints, but we assume wallclock time is less reliable in this context. Deep learning serving engines like TensorRT, vLLM and SGLang may use some dynamic batching strategies which means the wall clock time changes with the request arrival rates. For example, the daytime wall clock time fingerprint is different from the one at night because the server is busier during the day.

Q2.

Is it possible that different combinations of hardware and software lead to the same prediction?

Yes, it is possible. This is one limitation we discussed in Sec 6.3 and A.4. During experiments we found when the software stack is identical, HSPI cannot differentiate RTX8000 and RTX2080Ti. This is because these two GPUs have both NVIDIA compute capability = 7.5 (Turing architecture), and a very similar number of Tensor Cores around 550 (they fall into the same EQC when the software stack is the same). For more details please refer to Sec 6.3 and A.4.

Please let us know if you have further questions :)

Thank you offering valuable suggestions and questions. We will address them one by one. Due to word limits, we uploaded three new result tables here: https://imgur.com/a/VbONCoU

E1. Essential References Not Discussed

Our work is different from these two papers in terms of goals, threat models (methods), and generalizability.

Goals

• Ref 1 extracts information about CNN models, e.g, whether the model is AlexNet or SqueezeNet, from an FPGA-based CNN accelerator, by analyzing memory access patterns.
• Ref 2 predicts information about the encrypted IPs on an FPGA-based DNN accelerator. They also recover characteristics about model architectures and extract model params.
• Our work aims to predict information about the SW-HW stack used for serving large deep learning models, including compiler, data types, GPU arch, parallelism strategy, etc. We do not aim to predict which model is being served.

Threat models

• Ref 1 and 2 assume side channel information about the HW like off-chip memory access patterns, electromagnetic traces, schematics is accessible.
• We assume the SW-HW platform serves DNNs in the cloud and we only have access to the request and responses via the service provider’s API.

Generalizability

• Ref 1 and 2 tests their methods on a specific FPGA accelerator.
• We tried our best to test the generalizability of HSPI, across model families, data types, GPU archs, parallelism strategies, etc.

We will include these two papers in our related work section and discuss the differences there.

Q1

I am very curious how a client would differentiate between "deviations" from HW and "deviations" from SW ...

One can differentiate deviations caused by HW and SW because they will be in different EQCs and we can enumerate all of them by going through all HW and SW combinations. However, in Sec 6.3 and A.4 we do get the same output from RTX8000 and RTX2080Ti (they both have NVIDIA compute capability = 7.5 and very similar number of Tensor cores ~550) when using the same SW stack, thus these two combinations are left in the same EQC. Please refer to 6.3 and A.4 for more details.

Q2

As the paper states in A.5, the model inference could have introduced some noise/perturbations ...

One would need to increase the number of queries sent by HSPI to capture statistics in different HW and SW settings (the cost increases, too). On the other hand, the introduced noise reduces the model performance, e.g, LLM generates low-quality texts.

Q3

It would be interesting to investigate what features were used to distinguish different precision, kernel, HW, ...

Our method is based on the EQC theory explained in Sec 2. Limited by rebuttal word counts, we would recommend this paper, in which they present a more detailed analysis identifying how architectural choices impact computational stability and precision deviations.

Q4

It would be more interesting to see experimentatal results for more variegated class of accelerators ...

• Yes, we are also curious about this. We believe our observation on NVIDIA GPUs still holds on other accelerators because there may be more differences captured by HSPI classifiers, such as different accumulator length in compute units. Unfortunately, limited by resources, we are not able to test HSPI on all these platforms.
• We run new experiments including Amazon Inferentia and AMD Instinct MI300X (RTab.1 in the link). HSPI still successfully differentiates them (avg.acc=96.4%).

Q5

For the black-box access only scenario in 4.1, client might not have the full picture considering that the service providers may have appended prompt ...

• During experiments, we already prepend chat prompts before our request. For example, each of our Qwen-2.5 input concatenates “You are Qwen, created by Alibaba Cloud. You are a helpful assistant” and our query “Please generate <num_random_words> random words …” . We find using these simple chat prompts actually helps to generate more random outputs.
• For complex scenarios, HSPI can be combined with prompting tricks like jailbreaking to be more robust.

Q6

Service providers may limit the inference requests ... DoS attack.

Commercial inference services are designed to support high‑throughput access. Moreover, HSPI does not need to collect all the responses in a very short period, so request rates of HSPI are lower than the limits set by service providers, lower than DoS attacks by order of magnitudes. For example, in Tab 6 experiments, we sent 6 requests/sec, and our method collects 256 inference queries in around 40 minutes, which is comparable to normal application workloads. We can also distribute queries across multiple accounts in case of strict rate limit.

Q7

How does the paper scale to larger set of HW and SW platforms...

We ran new large scale experiments (RTab2 in the link) and HSPI still works well. Please refer to our answer to Reviwer A46R.

Thank you for offering valuable suggestions and questions. We would like to address them one by one.

S1. The authors shared their code to training the classifier and run inference. We don't have the logit dataset to run but it looks legit.

• We know collecting the logits from various combinations of [model checkpoint, hardware stack, software stack] are resource consuming, so we will open-source the logits datasets once the paper is accepted.
• The supplementary materials also include the codes for producing the logit dataset. For example, in line 227 of llm_logits_svm.py, the function create_logits_dataset collects logits for training the classifier.

Questions For Authors: Q1. Given the current method for hardware and software platform inference, how to avoid the stack to be exposed to users?

There are several possible ways but may have side effects on user experience like lower generation quality and longer latency.

• In Appendix 5, we suggested random bit flips and adding random noise as potential ways of mitigation. For further explanation, please refer to Appendix 5 on page 13.
• Furthermore, adding other forms of randomizations such as reordering arithmetics randomly or switching between multiple compilation kernels would make our method more expensive, since then a statistical average would need to be calculated on a lot more samples to see a difference in distributions between different hardware and software platforms.
• Lastly, limiting model logit access, also would make it harder for HSPI. However, the logits is an important part of API enabling users to explore smart sampling strategies like test-time scaling.

We will add this discussion to the revised manuscript.

Please let us know if you have further questions :)