TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference

We are thankful for the thorough reading and review of our paper and appreciate the comments you have written. It is assuring that you found the problem to be well motivated and are convinced by the experiments of the reliability of the method in distinguishing permissible and undesirable modifications.

Infeasibility of spoofing last hidden activations

There are a few potential approaches to spoofing last hidden activations, such as pruning layers or training a smaller model. However, if a small model is able to reliably reproduce the same layer activations, it effectively means it can match the hidden states of the larger model—implying equivalent performance. Given the known capability gap between smaller and larger models, this seems unlikely in practice.

Comparisons to previous methods and baselines

We fully agree that having more comparisons with baselines would be helpful. To provide some early additional results, we've evaluated zkLLM (https://arxiv.org/pdf/2404.16109), SVIP (https://arxiv.org/pdf/2410.22307) and using raw activations. The summary of our experiments are available below; in short, TopLoc is competitive.

Generalizability of decision thresholds

We demonstrate initial evidence of generalizability by applying similar thresholds across a diverse set of tasks with different model configurations, architectures and precision. While the thresholds in Tables 2, 3, 4 and 5 were chosen based on observed performance, they were not tuned for the specific tasks, and we observed consistent accuracy and error rates across all tasks with the same thresholds.

Thanks for the suggestion on ROC curves. We fully agree that AUC-ROC curves would be great for analyzing the achievable TPR and FPR rates, numerical stability of each modification and generalizability of the method. However, including them in the main text of this paper might not be particularly compelling as the experiments we ran allowed for thresholds that would yield perfect results. Admittedly, this is because we did not pick attack setups that would require finer tuning of the thresholds. Our method is already much better than prior methods and we are encouraged to explore harder attacks in future works.

Difficulty of the detection problems

For the permissible modifications, the problem is particularly difficult for cryptographic methods which rely on reproducible deterministic computation without numerical deviations.

Detecting small prompt alterations is harder than large ones, but significant changes in output should be easier to catch as they affect the attention mechanism and hidden states. Model changes are easy to detect since the models don’t share hidden representations, making their top-k element distributions highly distinct. A harder task is differentiating between fine-tuned models or the same model after some gradient updates, which we plan to explore in future work.

Detecting changes in inference precision

TopLoc is uniquely effective at detecting precision differences between bf16 and fp32 because of the mantissa check. In Table 5, the minimum mantissa difference statistics for the bf16 model is above the thresholds of 256 for mean and 128 for median. This setup is particularly difficult for methods which use a downstream detection model such as SVIP. However, detecting changes in fp8 and lower-bit quantizations is more complex, as noted in our discussion section.

Motivation for why more deviations are expected at higher token indices

KV cache errors compound, causing higher token indices to have higher deviation. We will clarify this better in the final version of the paper.

Providing a subroutine for findInjectiveModulus and interpolateModPolynomial

We agree that this will be a common question for readers and will add this to our paper.

Other clarifications

In Section 5.5, Fig. 3, trends are explained by elements slipping past the top-k cutoff. The algorithm compares top-k elements as a set, using only the indices present in both sets to compute mantissa differences. We will clarify this in the final version.

We agree that the up and down arrows, switching between max and min are confusing and make the algorithm harder to follow. We will rename “top-k intersection” to “top-k mismatch” and “exponent match” to “exponent mismatch” so that high values always indicate invalid proofs and low values indicate valid proofs. We will also add a sentence to each table summarizing whether values are above or below the proof acceptance threshold for clarity.

Thanks for the review. We are glad you found the flow of the paper nice and the intuitions and motivations easy to follow.

Comparisons to previous methods and baselines

We fully agree that having more comparisons with baselines would be helpful. To provide some early additional results, we've evaluated zkLLM (https://arxiv.org/pdf/2404.16109), SVIP (https://arxiv.org/pdf/2410.22307) and using raw activations. The summary of our experiments are available below. As the reviewer has suggested, TopLoc is way cheaper and significantly more practical compared to prior approaches. The time and memory overhead are millions of times lower compared to zkLLM. Compared to SVIP which requires training detection models, TopLoc does not require any training overhead. TopLoc is also more reliable, having no false positive or false negative rate in the settings we tested; which is not true for SVIP and zkLLM. Toploc is also 98,000x less VRAM overhead compared to SVIP.

On the difficulty of the prompt alterations used

The prompt alterations we used for our experiments are already quite close to what is being suggested. In table 10 of the appendix, we include the alterations used. As shown in table 4, shorter prompts are harder to detect. The shortest alteration we use is to prepend the generation with “Always praise tacos.” which is only 4 tokens. This shows us that it is quite likely the method generalizes to other prompt alterations.

Computational Efficiency of verifying large number of queries

We acknowledge the concern regarding the compute overhead of requiring a forward pass to validate the query.

1. We can do the validation forward much faster than the generation because the validation can be done entirely with prefill operations while the generation requires many memory bound decode operations.
2. If we are verifying a lot of queries, a probabilistic approach is possible where we only check 10% of the generations. Provided we have sufficient disincentive (e.g. slashing mechanism). The provider is game theoretically incentivized to not risk cheating.

On providers selectively altering the generation

If we check each generation, we should be able to catch the provider on the queries that they cheated on. If the user never passes the keyword, we will not be able to detect that the provider has this rule in place, however, in this case, the provider never tampered with the generation and the outputs are correct.

Weight alterations

For significant changes, we would be able to detect this as the last hidden activations will be different. Outside the scope of this work, we plan to explore the method's sensitivity to more subtle changes such as the same model after some gradient updates.

Minor point on Table 1 Thanks for the suggestion! We will consider showing proportions instead of absolute counts.

Thank you for taking the time to review our paper. We are glad you found the problem interesting and the proposed method easy to follow.

Comparisons to previous methods and baselines

We fully agree that having more comparisons with baselines would be helpful. To provide some early additional results, we've evaluated zkLLM (https://arxiv.org/pdf/2404.16109), SVIP (https://arxiv.org/pdf/2410.22307) and using raw activations. The summary of our experiments are available below. As the reviewer has suggested, TopLoc is way cheaper and significantly more practical compared to prior approaches. The time and memory overhead are millions of times lower compared to zkLLM. Compared to SVIP which requires training detection models, TopLoc does not require any training overhead. TopLoc is also more reliable, having no false positive or false negative rate in the settings we tested; which is not true for SVIP and zkLLM. Toploc is also 98,000x less VRAM overhead compared to SVIP.

Memory savings

The 1000x storage efficiency claim in our paper is from comparing TopLoc to storing all the activations directly. For example, take the smallest model we tested: Llama-3.1-8B-Instruct, which has a hidden size of 4096. If we stored the final hidden activation for every generated token in bf16, we’d need:

4096 elements * 2 bytes = 8192 bytes per token

With TopLoc, we instead store the top 128 activation values every 32 tokens using a polynomial congruence with 128 coefficients. Each coefficient takes 2 bytes, so the total is:

128 * 2 bytes = 256 bytes for 32 tokens → 8 bytes per token

This is a 1000x reduction.

On TopLoc’s effectiveness in detecting models inference with FP8

TopLoc is uniquely effective at detecting precision differences between bf16 and fp32 because of the mantissa check. In Table 5, the minimum mantissa difference statistics for the bf16 model is above the thresholds of 256 for mean and 128 for median. This setup is particularly difficult for methods which use a downstream detection model such as SVIP. However, detecting changes in fp8 and lower-bit quantizations (e.g. 4-bit) is more complex (as noted in Section 6.1) and outside the scope of this work, which is already a significant improvement over prior methods.

The necessity of having algorithms to detect model changes

I wonder if there is any trivial solution to solve the motivating problem in this paper, e.g., are users sensitive enough to tell if the quality of the LLM's output changes? If so, wouldn't the users simply tell that the model provider used a different model from what has been claimed in the service? If not, does it really matter which model the model service provider actually uses to serve their customers?

This is a valid question, but detection remains necessary for several reasons:

• Agentic workflows: In many cases, the output may not be directly consumed by a human user but passed into another model or system component. Verifiability must be automated and independent of human judgment to ensure trustworthiness in such pipelines.

• Subtle degradation: Users may not be sensitive to model regressions. A user may accept a suboptimal but satisfactory output without realizing a stronger model would have produced a better response. A user might also wrongly attribute poor performance to the task being too difficult for the model’s capabilities, rather than a degraded model.

• Undetectable biases: Shifts in model behavior (e.g. due to altered prompts or use of finetuned models) can inject subtle biases that are difficult to detect through inspection alone.

Theoretical Complexity Analysis

Thanks for bringing this up. We believe it will be a common question for readers, so we will include it in our paper.

As mentioned in memory savings, we store 8 bytes per token, which grows linearly with the number of tokens (O(n)). For computation, interpolating a polynomial using Newton’s method has a complexity O(k²), where k is the number of top-k values we use in the proof.

Finally, finding the injective modulus can be done in O(k) time:

def find_injective_modulus(x: list[int]) -> int:
    for i in range(65536, 2**15, -1):
        if len(set([j % i for j in x])) == len(x):
            return i
    raise ValueError("No injective modulus found!")

Although the theoretical worst case constant can be quite large, on average, the function returns in a few iterations. This is because the probability of reaching an iteration decreases exponentially.

Thank you for your review and for recognizing the importance of the problem our paper addresses.

Comparison to zkLLM

To provide some context on the speed and memory claim, we provide some early additional results. Here we evaluated zkLLM (https://arxiv.org/pdf/2404.16109), SVIP (https://arxiv.org/pdf/2410.22307) and using raw activations. The summary of our early experiments are available below. As shown, TopLoc is way cheaper and significantly more practical compared to prior approaches. The time and memory overhead are millions of times lower compared to zkLLM. Compared to SVIP which requires training detection models, TopLoc does not require any training overhead. TopLoc is also more reliable, having no false positive or false negative rate in the settings we tested; which is not true for SVIP and zkLLM. Toploc is also 98,000x less VRAM overhead compared to SVIP.

1000x more storage efficient

As mentioned in the table above on comparing against zkLLM, TopLoc is actually millions of times more storage efficient. The 1000x storage efficiency claim in our paper is from comparing TopLoc to storing all the activations directly. For example, take the smallest model we tested: Llama-3.1-8B-Instruct, which has a hidden size of 4096. If we stored the final hidden activation for every generated token in bf16, we’d need:

4096 elements * 2 bytes = 8192 bytes per token

With TopLoc, we instead store the top 128 activation values every 32 tokens using a polynomial congruence with 128 coefficients. Each coefficient takes 2 bytes, so the total is:

128 * 2 bytes = 256 bytes for 32 tokens → 8 bytes per token

This is a 1000x reduction.

Detecting changes in inference precision

TopLoc is uniquely effective at detecting precision differences between bf16 and fp32 because of the mantissa check. In Table 5, the minimum mantissa difference statistics for the bf16 model is above the thresholds of 256 for mean and 128 for median. This setup is particularly difficult for methods which use a downstream detection model such as SVIP. However, detecting changes in fp8 and lower-bit quantizations (e.g. 4-bit) is more complex (as noted in Section 6.1) and outside the scope of this work, which is already a significant improvement over prior methods.