We appreciate your feedback and will proceed to address your concerns in the order they appear.

Response to Weakness

1. We have also noted the concerns regarding similar works.

   • QueryDet has little relation to this work and shares low similarity. It is not transformer-based, and its queries are not closely related to those in transformer methods. Furthermore, its approach is unrelated to model pruning.
   • The SRDD method focuses on tokens in the transformer encoder, whereas the DETR-like methods targeted in this work typically consist of only a transformer decoder structure. SRDD is more aligned with traditional model pruning approaches and has no connection to query pruning.
   • To date, we have not identified any methods similar to ours.

2. Your concerns are reasonable. We will explain why we focus on the 3D object detection field. 3D object detection is a highly practical research area with fewer conversion steps required for its application in real-world commercial scenarios. This is why we chose to focus on 3D object detection from the beginning, rather than 2D object detection or 2D image classification. Furthermore, research on model pruning for 3D object detection methods is currently limited and existing pruning methods for ViT models and 2D object detection models are difficult to be applied for 3D object detection methods. We acknowledge that the discussion on this matter in the paper was insufficient. Thank you for pointing that out. We have added this content in the updated version of the paper, Section 2.

However, because pre-defined queries also exist in DETR, we can still apply GPQ to DETR-based 2D object detection methods. We conducted experiments using Conditional DETR as an example, and the results are shown in the table below. As shown in the table, after pruning half of the queries using GPQ, Conditional DETR is still able to maintain its original performance. Moreover, the pruning process requires only 8 epochs, which is significantly more efficient compared to retraining a new model from scratch (50 epochs). These results demonstrate that our method can still achieve its intended effect when applied to 2D object detection methods.

3. Higher iteration steps generally lead to higher model accuracy but also result in longer training times. Theoretically, assuming no overfitting, infinitely long training could bring accuracy infinitely close to 1. To enable fair model comparisons, accuracy is typically evaluated with a fixed number of epochs. While models with fewer queries may eventually catch up to the accuracy of models with more queries as training iterations increase, this would come at the cost of longer training times. Therefore, following community best practices, we compare the accuracy of models with different numbers of queries under the same number of training steps.

Response to Questions

1. To date, no methods specifically focused on pruning queries have been identified, and there is no "very similar" work in this area.

2. To date, pruning queries in DETR-based models remains an unexplored area. To the best of our knowledge, GPQ is the first work to explore pruning queries in DETR-based methods. However, there are still many methods that tries to prune tokens in ViT models. As we supplemented in Section 2 of updated paper, they are difficult to be migrated to 3D object detection models due to following reasons:

   1. Nonexistent Pruning Targets. Some pruning targets in ViT models do not exist in 3D object detection models.
   2. Structural Inconsistencies. Some methods rely on specific structures that do not exist in 3D object detection.
   3. Massive Data Differences. 3D object detection models generate larger feature maps, making it challenging to transfer existing methods.

We sincerely appreciate your feedback and will address each of your comments in the order they were raised.

Response to Weakness

Due to time and computational resource constraints, our initial experiments were not comprehensive. We have since supplemented them with additional experiments:

1. We evaluate more models, including DETR3D. You can check the results via anonmyous links https://github.com/Anonymity2025/files/blob/main/main_results.md and https://github.com/Anonymity2025/files/blob/main/sparsebev.md. Results have show that GPQ works well on DETR3D.
2. We test the inference speed on a desktop GPU (RTX3090), showing that our method can accelerate model inference by up to 1.31x.
3. We change models' backbone, use higher resolution (1600x640). Results are listed in anonymous file https://github.com/Anonymity2025/files/blob/main/main_results.md, which shows that GPQ still works when using larger backbone and higher resolution.
4. We also apply our method to ConditionalDETR. Results are listed in anonmyous file https://github.com/Anonymity2025/files/blob/main/conditional_detr.md, which shows that GPQ also works for 2D object detection.
5. We compare GPQ with an existing pruning method Token Merge(ToMe)[1], which shows that GPQ is better than ToMe. Results are listed in table below:

In the table, column $r$ represents the number of tokens to be "merged" as defined in the ToMe method.

We have incorporated these revisions into the updated paper. Newly added and revised content is highlighted in red. We will be very grateful if you could refer to the updated PDF of the paper for full details.

[1] Token Merging: Your VIT but Faster. ICLR 2023.

Response to Questions

1. We have fixed Fig.1 in updated PDF.
2. Yes, we use both content query and spatial query. In fact, in all of transformer-based 3D detection methods, the content query and spatial query are both used. Content query just is the "pre-defined query" mentioned in our paper, and the "spatial query" is usually called "positional embedding", which will be added to the "content query". The positional embedding is calculated using pre-defined query, so we just need to prune pre-defined queries.
3. Due to resource limitations (our experimental resources are two RTX 3090 GPUs), it is challenging for us to support longer training durations in a short time. However, we still conducted experiments with more epochs (36 epochs, StreamPETR-36e in the table below), and the results are shown in the table below. Here StreamPETR-GPQ loads the checkpoint generated by StreamPETR-36e.

The experimental results shown in the table further validate that our method remains effective even when applied to models trained for more iterations. After pruning redundant queries, the model is still able to maintain its performance.

4. We fully agree with your point that the number of parameters and memory usage are also important metrics for pruning methods. Pruning queries has minimal impact on the model's parameters and memory usage, as the number of parameters associated with queries in the transformer decoder is inherently small. Details are provided in the table below. The primary benefit of pruning queries lies in improving the model's inference speed (see anonmyous file https://github.com/Anonymity2025/files/blob/main/main_results.md).

Dear nz4D,

We hope this message finds you well. We appreciate the time and effort you've dedicated to reviewing our work, and we hope that our clarifications address your concerns effectively.

If you have any further questions or if there are any additional points you would like us to elaborate on, we would be more than happy to provide additional information.

Thank you again for your valuable feedback, and we look forward to your thoughts.

We are grateful for your thoughtful comments and will provide answers to your queries in the order presented.

1. The comparison with ToMe is indeed very meaningful. We applied TokenMerge(ToMe) to StreamPETR to compare with GPQ. Results are listed in table below, which shows that ToMe cannot work on 3D object detection. ToMe not only failed to accelerate the model's inference but also caused the model to run slower.
   • ToMe aims to prune ViT models. There are significant differences between image classification and 3D object detection methods, with the most notable being the number of tokens. Image classification methods typically require about 200 tokens, whereas in 3D object detection, due to the presence of surround views, the number of tokens can easily exceed 4000, with some methods generating more than 16,000 tokens.
   • ToMe divides $N$ tokens into two groups with equal size $\frac N2$, then calculate a matrix of shape $\frac N2\times \frac N 2$ to merge $r$ most similar token pairs. For ViT methods, this matrix is less than 200x200 (about 200 tokens with patch size 16x16). However, for 3D detection methods, the matrix can easily larger than 2000x2000 (for example, 4224 tokens in StreamPETR and FocalPETR with 704x256 image size ) and can even reach 8000x8000 (16896 tokens in PETR with 1408x512 image size), which is 100 to 400 times larger than in ViT! The immense token size causes Token Merge to consume a significant amount of computational resources just to compute the similarity matrix when applied to 3D object detection. Consequently, instead of speeding up the computation, it actually makes the model run slower, as shown in the table.

2. Our approach is not intended to replace model quantization methods. In fact, after applying our method, it is entirely possible to simultaneously use int8/int4 quantization to compress the model further. However, model quantization is highly sensitive to hardware, and modern machine learning models often rely on numerous third-party libraries (e.g., Flash Attention). This poses significant challenges for model quantization. When deploying the same model across different hardware platforms, it often requires hardware-specific adaptations, such as restructuring and removing certain components, followed by retraining. This process can lead to performance degradation and instability.

   In our experiments, we attempted to quantize the model. However, we encountered difficulties running the quantized model on GPUs, which forced us to run the model on CPU. The comparison of performance after quantization is provided below.

The FPS is not reported because the CPU speed is too slow to be meaningful.

3. It is essential to validate the effectiveness of GPQ on well-known methods from the nuScenes leaderboard. We can provide the results of GPQ on SparseBEV with an image resolution of 704x256, which shows that GPQ works well for SparseBEV. However, for a resolution of 1600x640, unfortunately, we are unable to run the experiments due to the limited time and computational power of our GPU (RTX 3090). In fact, even with a low resolution (704x256), running SparseBEV remains computationally expensive for us.

• If you want to examine the impact of higher resolutions on our method, you can refer to the PETR experiments conducted with VoVNet at a resolution of 1600x640 (see anonmyous link https://github.com/Anonymity2025/files/blob/main/petr_vov_1600x640.md). It took us over a week to run the experiments, and we would greatly appreciate it if you could consider these results.

4. We provide results of PETR using VovNet as backbone on higher resolution 1600x640 (see anonmyous link above), which proves that GPQ can work on large backbone and higher image resolutions.

Dear Xnmm,

We hope this message finds you well. We are grateful for the time and effort you have spent reviewing our work, and we hope that our clarifications have addressed your concerns adequately.

If you have any further questions or require more information on any aspect of our submission, please don't hesitate to let us know. We would be happy to provide any additional details or clarification.

Once again, thank you for your valuable feedback. We look forward to hearing your thoughts.

Thank you for the rebuttal. I read response from other reviewers' as well. However, I maintain my original score as the authors do not address the following concerns:

• The paper does not compare performance with SparseBEV on the nuScenes leaderboard at 640x1600 resolution, even though SparseBEV releases their leaderboard model at 640x1600 resolution. This raises doubts about the effectiveness at higher resolutions, limiting its practical utility in cloud-deployments. It's absolutely crucial for 3D detection methods published in top-tier conferences like ICLR, NeurIPS, ICCV or CVPR to include nuScenes leaderboard results at 640x1600 resolution.

• Thank you for providing PETR results. I appreciate these results, but, PETR is a very old detector, and is no longer a SoTA. The key application of the proposed method is to cut down the inference time of bigger resolution / SoTA leaderboard models such as SparseBEV at 640x1600 resolution.

• AP and NDS for proposed method and SparseBEV even at 256x704 resolution take a significant hit of absolute 2 and 1 points respectively in newer results.

Thank you for your valuable insights. We will respond to your questions sequentially

Response to Weakness

1. We are sorry for typos, and we thoroughly reviewed the manuscript and corrected all identified typos and inconsistencies.

2. Our response to this comment is from two perspectives:

   • First, comparison with previous work is challenging because, to our knowledge, no previous research has focused on query pruning. Traditionally, pruning in the visual domain has primarily been applied to models for image classification tasks, such as ViT, with very little work targeting 2D object detection and almost none targeting 3D object detection methods. There are significant differences between image classification, 2D object detection and 3D object detection methods, the most notable being the number of tokens. Here, "tokens" refer to the features extracted from the input image by the image backbone. Image classification methods typically require only a few hundred tokens, 2D object detection methods use around 2000 tokens, while 3D object detection can easily exceed 4000 tokens due to the presence of surround views (StreamPETR, FocalPETR with image size 704x256). Some methods even generate more than 16,000 tokens (PETR, SparseBEV with image size 1600x640). We argue that the significant difference in the number of tokens makes it difficult for ViT pruning methods to perform effectively when applied to 3D object detection. For example, we conducted experiments using Token Merge (ToMe) [1], and the results can be found in an anonymous file https://github.com/Anonymity2025/files/blob/main/tome.md. In the table, "StreamPETR-GPQ" is our method with 900 initial queries and 300 final queries; column r represents the number of tokens to "merge" as defined in the ToMe method. As indicated in ToMe, in order to merge tokens, we need to divide the tokens into two groups and then compute a similarity matrix of the form $\frac{N}{2}\times\frac{N}{2}$, where $N$ is the number of tokens. The calculation of the similarity matrix introduces additional computational overhead, which not only fails to speed up inference, but also slows down the model. This makes ToMe ineffective for 3D object detection methods.

   • Second, due to differences in model architectures, we believe that many existing ViT pruning methods are not directly applicable to 3D object detection. In ViT, the tokens generated by the backbone serve as queries, keys and values, resulting in an attention map with a square matrix shape. This specific shape makes methods based on it unsuitable for the domain of 3D object detection [2]: in 3D object detection methods, the attention map is generated by predefined queries and keys whose numbers are not equal. As a result, the attention map is not a square matrix.

   [1] Token Merging: Your VIT but Faster. ICLR 2023.

   [2] Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers. CVPR 2024.

3. We agree that ablation experiments are important. To validate the effectiveness of our method, we complement the following ablation experiments:

   1. Reasonability of pruning criteria: Our pruning strategy involves removing the queries with the lowest classification scores. To demonstrate the validity of this criterion, we designed two experiments: one where we prune the queries with the highest classification scores(prune highest), and another where we guide the pruning using the cost generated by the assigner during the binary matching process between predicted and ground truth values(prune using cost) .

   2. Reasonability of the gradual pruning strategy: We designed an experiment that pruned 600 queries in a single iteration to demonstrate that our gradual pruning strategy is reasonable and effective (prune in one iteration).

The results show that pruning using the highest classification scores, pruning based on assigner cost, or pruning a large number of queries in a single step all degrade the model's performance after pruning. This demonstrates that the strategy designed in GPQ is currently the optimal query pruning strategy.

Response to Questions

1. We have updated the Table 1, adding memory costs. Details can be found in updated PDF (pp.2).
2. Thanks for pointing this mistake. It should be iteration $n+1$, and we have fixed it.
3. We have updated the Table 2, where following your suggestions, we newly added models' costs.

Dear xxXP:

We hope you're doing well. We truly appreciate the time you've dedicated to reviewing our work, and we hope the clarifications we offered have addressed your concerns satisfactorily.

If there are any further questions or aspects that you would like us to elaborate on, we would be glad to offer more information.

Thank you once again for your insightful feedback, and we look forward to receiving your thoughts.

Thank you for your constructive feedback! We will go through each of your points one by one and provide our responses accordingly.

Response to Weakness

1. That is correct. To help readers better understand the difference between our method and training from scratch with fewer queries, we added Figure 4 in the updated version of the paper. Figure 4 illustrates that queries obtained by training from scratch with fewer queries are more scattered and disordered, whereas queries pruned using GPQ not only retain the same spatial span but are also more organized.

   1. Pruned queries are tend to generate lower classification scores, which means these queries will predict background class. They can be removed because their results can be covered by other more confident queries as illustrated in Fig.1(a) in the paper. To visually demonstrate the differences in queries before and after pruning, we have plotted the reference points used to generate the queries, as shown in Figure 4 of the updated paper. This figure provides a clearer view of the impact of query pruning: redundant queries are removed, while the remaining queries visibly cluster into a plane. The plane formed by these points represents the solution space. After pruning, the remaining points still occupy the same space as before, which allows the original model's performance to be preserved.

   2. We try to prune top most confident queries(pruning highest) and also prune the queries just in one go(prune in one iteration), the results above show that the model hasn't been destroyed, but still suffers a perference decrease.

2. Thank you very much for your suggestion! Exploring pruning strategies is indeed necessary. We conducted experiments following your suggestion:

   1. The results of pruning down to the desired number of queries in one go are shown above, which evaluates that pruning queries gradually is better than pruning them all in one iteration.

   2. We conduct the experiment that uses cost generated by assigner as the pruning metric. The results are shown above. By using cost instead of classification scores, the model could maintain its performance but is still worse than pruning using classification scores. This is because, during inference, the only metric to justify which query would be selected is classification socre. No assigner cost would be generated because there is no gt during inference. So the classification score is more suitable for acting as the pruning metric.

3. 3D object detection is a highly practical research area with the ultimate goal of deployment on various edge devices. Therefore, we chose Jetson Nano to test its performance, aiming to closely approximate its behavior in real-world production environments. However, our method also demonstrates strong performance on desktop GPUs. The anonmyous link https://github.com/Anonymity2025/files/blob/main/main_results.md provides the results of GPQ on a single RTX 3090. Results show that GQP can also accelerate model inference on desktop GPUs. For example, in the case of PETR, pruning from 900 to 150 queries results in an mAP of 30.52%, which is 2.15 points higher than training from scratch with 150 queries (28.37%), and its speed increases from 7.1 fps to 9.3, which is 1.31x faster

4. We have added top-down view to Fig.4. Please refer to Fig.5 in the revised PDF (pp.15) for detailed results.

5. Following your suggestion, we have simplified the algorithm (pp.6 in the revised PDF).

Response to Questions

In the newly submitted PDF, we updated Figure 2 by improving its resolution, boldening the lines, and revising some text that might have caused misunderstandings.