AdaPKC: PeakConv with Adaptive Peak Receptive Field for Radar Semantic Segmentation

Thank you for your constructive and thoughtful comments. We appreciate the recognition of the strengths of our work: the innovation of our method, technically sound paper and overall clarity in writing. We are glad to answer all your questions.

Q1: The correspondence between interference in radar frequency maps and that in image frames presented in Figure 1.

A1: Thank you for your constructive feedback. However, the fluctuating interference in the radar frequency map is indeed difficult to correlate with the optical images, primarily due to the following reasons:

1. Dissimilar Coordinate Systems and Fields of View: The radar frequency maps and the optical images have different coordinate system definitions and fields of view, making it challenging to establish a direct correspondence between them. For instance, in Range-Doppler (RD) maps, targets which are spatially distinguishable in the optical image but share the same radial distance will be compressed into the same range bin in the RD map, thus probably becoming indistinguishable. Furthermore, Doppler domain reflects the velocity of target, it cannot be mapped into the optical image.
2. Different Imaging Mechanisms: The radar frequency maps and optical images are formed through distinct mechanisms. The amplitude in a radar frequency map actually represents the strength of the received echoes, which can be affected by invisible interference noise in the optical spectrum and system noise within the radar system itself. These factors do not directly translate into optical images, making it difficult to visually correlate the fluctuating interference seen in the radar frequency maps with the corresponding optical images.

Therefore, while we can observe fluctuating interference in the radar frequency map, it is challenging to visualize this interference using the corresponding optical images.

Q2: A clearer caption for Figure 2 and 3.

A2: We will add the following descriptions into the captions of corresponding figures.

1. Caption for Figure 2: The illustration of AdaPRF in AdaPKC$^\xi$. (a) illustrates the definition of PRF in PKC, whose area is governed by the reference bandwidth and guard bandwidth; (b) describes the estimation process of AdaPRF in AdaPKC$^\xi$, including denoting K candidate PRFs for each CUT, translating these PRFs into metric scores, and finally selecting an appropriate PRF as the AdaPRF with these metric scores.
2. Caption for Figure 3: The illustration of AdaPRF in AdaPKC$^\theta$. (a) illustrates an example of candidate PRFs in AdaPKC$^\theta$, where the guard bandwidth is in quadruple form; (b) describes the flowchart of the optimal guard bandwidth estimation network, which consists of two parallel branches that sample representative points in their corresponding directions, then automatically measures and selects the optimal guard bandwidth.

Q3: Analysis of the computational efficiency of the model.

A3: Please see our comments to all authors that clarify the [Complexity and FPS] of our work and PKC.

Q4: The effectiveness of the proposed method requires further validation.

A4: The effectiveness of our proposed method has been widely validated within multi-view and single-view radar semantic segmentation frameworks and across three large scale radar datasets. And we are also collecting new real-measured radar datasets to more extensively validate the effectiveness of our methods and other works.

Q5: More clearer explanation of the division of observed radar signals and derivation of Eq. (1).

A5: It is natural to devide the radar signals into the three subsets: (1) signals reflected from a target, $S_t$; (2) noise with signals that leaks out of the target, $S_{t-n}$; and (3) noise, $S_n$. For simplicity, we introduce an attenuation factor $\eta<1$ to indicate the leaking of signals from targets. For some CUT $x_c=\psi(s;W)$ and its candidate reference unit $x_r=\psi(s';W)$,

• if $s'\in S_t$, then $s'$ and $s$ belong to the same target, so $s’=s$ and $E(x_c x_r^T)=E(\psi(s;W)\cdot \psi(s';W)^T)=E(||\psi(s;W)||_2^2)$
• if $s'\in S_{t-n}$, then $s’=\eta\cdot s$ and $E(x_c x_r^T)=\eta E(||\psi(s;W)||_2^2)$
• if $s'\in S_{n}$, then $E(x_c x_r^T)=E(\psi(s;W))\cdot E(\psi(s';W)^T)=0$

From this equation, we can see that the inner product transformation assigns three statistical boundaries to $x_r$ from the three subsets, and this attribute significantly serves to facilitate the subsequent localization of reference units from $S_{t-n}$.

Q6: Advantages of AdaPKC compared to dilated convolution techniques.

A6: Compared to dilated convolution, AdaPKC has the following advantages: (1) The receptive field of dilated convolution can only be manually adjusted by setting different dilation rates, whereas AdaPKC can adaptively adjust the receptive field for each CUT based on the distribution of the CUT and its surrounding area; (2) While dilated convolution indirectly establishes a guard field around the CUT, it always couples the CUT with its surrounding points during interference estimation and band-pass filtering. In contrast, AdaPKC decouple the CUT from the surrounding points, explicitly estimate interference noise, and perform adaptive band-pass filtering for each CUT.

Q7: Would the method proposed in this paper still work if other baselines similar to PKC was used?

A7: Yes! Our AdaPKC is indeed a universal convolution operator. If you review our code, you can find that AdaPKC can be used conveniently ****just like any other convolution operator, and it can be seamlessly integrated into any convolutional framework designed for radar signal processing.

Thank you for your thoughtful comments. We appreciate the recognition of the strengths of our work: superior performance than SoTA, solid evidence of the method's efficacy by rigorous ablation studies, contribution to the development of radar semantic segmentation. We are glad to answer all your questions.

Q1: This method improves PKC in a novel way. The novelty is limited for NeurIPS.

A1: We apologize for not fully summarizing the novelty of our work. Next, we provide a detailed explanation of the innovative aspects of our research.

• Conceptually and in principle, to the best of our knowledge, adaptive peak receptive field (AdaPRF) in this work is the first attempt specifically tailored for radar signal processing to dynamically adjust the receptive field (RF) for convolution operators, which is a further great breakthrough compared with PKC. Its design is based on both (1) radar signal principles, including radar signal generating mechanism, the difference between target and interference distribution, and (2) deep learning principles, such as differentiable high-dimensional representation learning for data-driven end-to-end network optimization. With AdaPRF the original PKC is updated into a more data-adapted version, AdaPKC, and is verified in different convolution frameworks, which can also be extended to more radar-oriented learning algorithms and wider range of radar detection scenarios. Since more adaptive interference estimation is always a key topic in radar signal processing, AdaPKC is novel and significant.
• Additionally, although PKC has been proposed, its inherent limitations cannot be ignored. Continuous introduction and validation of new ideas and methods are essential for developing new radar sensing algorithms to better meet practical needs, just like YOLO series.
• From a design perspective, AdaPRF is innovative compared to existing methods. In this paper, we first analyze existing mature works in both RSS and deep learning fields, then deeply examining their shortcomings from radar perspective, such as PKC's fixed RF issue, the difficulty of making CFAR's dynamic RF learnable, and the limitations of DCN's dynamic RF mechanism in handling radar signals. Finally, rather than simply using or combining existing research, we propose the AdaPKC, and validate it with various real-measured radar data, providing new insights for this research area.
• Technically, it is worth noting that implementing AdaPRF is a non-trival task. We have summarized and explained main challenges and solutions in our paper and code, offering valuable experience for future research, such as:
  • How to reliably assess the reference points belonging to appropriate PRF in high-dimensional radar signal representation;
  • How to design the assessment method to meet the requirements of smooth differentiability and parallel computation of deep learning;
  • How to handle the changing number of sampling points for fixed convolution kernels due to dynamically changing receptive fields;
  • How to optimize inference to meet the requirements of practical radar signal processing application.
• Finally, we emphasize that the goal of this work is to meet the practical needs for radar application and contribute effective and efficient component for the next generation of radar signal processing paradigm. We are well aware of the many bottlenecks in exsiting radar signal processing, which has been stagnant for many years, and the challenges in applying deep learning to radar. Therefore, our goal is to provide a practical extension for new radar signal processing methods, freeing them from existing rigid workflows, which is of significant industrial value.

For the conference of NeurIPS, our work presents a practical study of adapting deep learning to radar recognition application, aligning with the Call for Papers section of NeurIPS that focus on Applications and Deep Learning, etc, which is also strongly represented in NeurIPS every year.

Q2: A more detailed analysis of the computational complexity and real-time performance.

A2: We compute the computational complexity and frame rates of AdaPKC and PKC on a Tesla V100 GPU and summarize it as follows. Compared to PKC, AdaPKC incurs minimal additional computational complexity and inference speed overhead.

Thank you for your response and clarification on the novelty of the work, particularly the introduction of AdaPRF.

Your explanation effectively highlights the innovative aspects of AdaPKC, including the integration of radar signal principles with deep learning techniques, and the unique challenges your method addresses. I appreciate your further explanation of AdaPKC and the additional analysis of computational complexity, which demonstrates a better balance between extra computational complexity and inference speed overhead compared to PKC.

Overall, the author has addressed some of the issues raised during the review process, reinforcing the significance of AdaPKC as an innovative approach in the field of radar signal processing. I look forward to seeing the proposed improvements.

Thank you for your thoughtful comments. We are glad to answer all your questions.

Q1: The adaptive version of PeakConv (PKC) is considered to be incremental work.

A1: Please see our response to all authors that clarify the [Novelty] of our work.

Q2: AdaPKC performance improvement over PKC is very limited.

A2: We are sorry that the illustration of performance comparison may be not clear enough that cause some confusion. However, the performance improvement of this work is not small for the Radar Semantic Segmentation (RSS) task.

• From the perspective of performance improvements in previous state-of-the-art methods, these performance improvements are rather hard and slow. Unlike optical images, radar frequency maps lack shape information for the targets and contain significant interference, making the RSS task particularly challenging. TMVA-Net shows improvements over RAMP-CNN only in the RA view, while exhibiting decreased performance in the RD view; TransRadar introduces numerous Transformer and convolution modules, yet offers only an average improvement of 1.1% over TMVA-Net in the RD view; and PKCIn-Net demonstrates almost no improvement over T-RODNet in the RA view. In summary, from the initial FCN model to our AdaPKC-Net, the mDice in RD view has advanced only from 66.3% to 74.0%. In contrast, our models, with modifications only to fundamental convolution kernels, achieve an average improvement of 1.4% over SoTA methods in the RD view of the CARRADA dataset and an average improvement of 1.5% in the RA view, which is a notably significant enhancement.
• From the perspective of comparison with DCNv2, we compare the performance improvements of AdaPKC and DCNv2 over their respective baseline convolution operators In Table R2, and results show that even when PKC has already reached much higher performance than DCNv1 on CARRADA dataset, AdaPKC still achieves a larger performance increase compared to DCNv2, highlighting the effectiveness of our proposed approach.
• From the perspective of comparison with PKC, we present the performance improvements of AdaPKC on the Ku band radar dataset in Table R3. It can be observed that even when PKC fails to deliver satisfactory results, AdaPKC is still able to improve the situation and achieve more significant performance enhancements over PKC.

Q3: The comparison with classical CFAR with adaptive reference cell selection is not included in the validation part.

A3: Thank you for your constructive feedback. We have added a comparison with various CFAR methods in Table R1 of the Rebuttal PDF, and the comparison reveals several limitations of CFAR methods: (1) they can only detect foreground targets but cannot distinguish specific categories of these targets; (2) they show poor target identification performance, struggling with complex target and interference scenarios; (3) they rely on manual parameter tuning and lack adaptive learning capabilities. Therefore, it is both necessary and practical to improve radar target perception paradigms using deep learning methods (it is also one of the motivations for both PKC and AdaPKC).

Q4: How many annotated radar data is sufficient to train the proposed network？

A4: We use the same amount of annotated data as in previous works such as PKC. For the CARRADA dataset, the training set includes a total of 8088 labeled frames. For the KuRALS dataset, the training set comprises 2064 labeled frames.

Q5: Complexity of the proposed network.

A5: We compute the computational complexity and frame rates of AdaPKC and PKC on a Tesla V100 GPU and summarize it as follows. Compared to PKC, AdaPKC incurs minimal additional computational complexity and inference speed overhead.

Q6: Is it running faster than CFAR with adaptive reference cell selection?

A6: In Table R1 of the Rebuttal PDF, we compare the fps of AdaPKC with CFAR methods. Currently, AdaPKC's inference speed is lower than that of CFAR methods, constrained by the original inference speed of PKC. However, the detection performance of CFAR methods is much worse than AdaPKC. Furthermore, we intend to leverage CUDA acceleration to improve AdaPKC's inference speed going forward.

Q7: Limitation of the proposal is not discussed in the submission.

A7: Sorry for the confusion! Due to the page limit, we have detailed the limitations in Section G of the Appendix.

Dear Reviewer,

We want to express our sincere gratitude for the time and effort you've dedicated to reviewing our paper. We're pleased to inform you that we've taken great care in addressing each of the questions and concerns you raised in your reviews. Please do not hesitate to reply if you have any further questions or suggestions!  We look forward to further improving the clarity and depth of our work with your valuable input!

Warm regards,

Paper 9766 Authors

Dear Reviewer qbsg,

We have thoroughly analyzed your questions and concerns you raised in your reviews, dedicating substantial time and effort to provide comprehensive explanations and corresponding revisions. We believe our responses should effectively address the issues you raised. We sincerely hope you could take the time to review our responses. If our response is adequate, we kindly ask you to give a fair score upgrade. Should you have any other concerns, we are eager to engage in further discussions with you. Thank you for your valuable input in enhancing the quality of our paper, and we also appreciate your respect for our hard work.

9766 Authors

Thank you for your thoughtful comments on our work. We appreciate the recognition of the strengths of our work: the practical motivation and good presentation, extensive experiments and analysis, and superior performance than SoTA. We are glad to answer all your questions.

Q1: The novelty of the proposed method is limited, considering the existing PKC work.

A1: Please see our response to all authors that clarify the [Novelty] of our work.

Q2: Although the method is practical and lightweight, the performance gain is not significant.

A2: We are sorry that the illustration of performance comparison may be not clear enough that cause some confusion. However, the performance improvement of this work is not small for the Radar Semantic Segmentation (RSS) task.

• From the perspective of performance improvements in previous state-of-the-art methods, these performance improvements are rather hard and slow. Unlike optical images, radar frequency maps lack shape information for the targets and contain significant interference, making the RSS task particularly challenging. TMVA-Net shows improvements over RAMP-CNN only in the RA view, while exhibiting decreased performance in the RD view; TransRadar introduces numerous Transformer and convolution modules, yet offers only an average improvement of 1.1% over TMVA-Net in the RD view; and PKCIn-Net demonstrates almost no improvement over T-RODNet in the RA view. In summary, from the initial FCN model to our AdaPKC-Net, the mDice in RD view has advanced only from 66.3% to 74.0%. In contrast, our models, with modifications only to fundamental convolution kernels, achieve an average improvement of 1.4% over SoTA methods in the RD view of the CARRADA dataset and an average improvement of 1.5% in the RA view, which is a notably significant enhancement.
• From the perspective of comparison with DCNv2, In Table R2, we compare the performance improvements of AdaPKC and DCNv2 over their respective baseline convolution operators, and results show that even when PKC has already reached much higher performance than DCNv1 on CARRADA, AdaPKC still achieves larger performance increase compared to DCNv2, highlighting the effectiveness of our proposed approach.
• From the perspective of comparison with PKC, in Table R3, we present the performance improvements of AdaPKC on the Ku band radar dataset. It can be observed that even when PKC fails to deliver satisfactory results, AdaPKC is still able to improve the situation and achieve more significant performance enhancements over PKC.

Q3: The practicality of the problem setting for radar semantic segmentation (RSS) in this work.

A3: We clarify each of the concerns you have raised:

• “It appears to be more similar to the object detection task found in other datasets”. In datasets like CRUW, radar object detection (ROD) tasks use a single point on the radar frequency map as target labels. However, in practical scenarios, multiple positions of a target reflect echoes, and the Fast Fourier Transform process in generating radar frequency maps can cause unavoidable spectral expansion. This results in more than one pixel belonging to the target range, thus mask labels in RSS task can better cover the target range. Furthermore, during training for ROD tasks, researchers (i.e., RODNet, T-RODNet) typically expand the single-point labels using methods like Gaussian smoothing to better cover the target area, whereas RSS labels inherently match the target distribution without requiring such preprocessing.
• “Other modalities such as cameras or LiDAR are usually available and can provide complementary information even in adverse weather conditions”. This work focuses on developing general perception methods for different radar systems and in various scenarios, while modalities such as cameras and LiDAR may fail to provide effective information in certain scenarios. For instance, the Ku-band radar used in our work for UAV surveillance and marine monitoring tasks has a maximum detection range of 6375 meters, far exceeding the effective detection range of cameras and LiDAR.
• “Scanning Radar and 4D Radar sensors (such as ORR, Radiate, and K-Radar datasets), which offer much higher resolution, are also becoming increasingly popular”. High-resolution radars are more expensive and generate much larger data volumes within the same detection range, making them unaffordable for the aforementioned long-range detection scenarios. Additionally, datasets including ORR, Radiate, and K-Radar only provide bounding box labels, which offer a lower level of granularity in depicting target areas compared to the mask labels used in RSS tasks.

Q4: Frame rates for AdaPKC and PKC.

A4: Please see our response to all authors that illustrates the [Complexity and FPS] of our work and PKC.

Q5: Relationship between the guard bandwidth and the specifics of the Radar sensor.

A5: The guard band is indeed influenced by the specific characteristics of the radar sensor. Setting the guard band ensures that the energy at the center point does not leak into the interference estimation process. Appropriate guard band settings are affected by the i) electromagnetic scattering characteristics of the target, including effective radar cross section (RCS), the appearance material , motion characteristics; ii) the detection environment of radar, such as environmental clutters and weather, which would cause different interference distributions; iii) radar's own operating mode, frequency band, waveform modulation and transmit power, thus resulting in different range and Doppler resolution. Finally, various factors will affect the gurad bandwidth setting. Hence, motivated by the impact of various factors on guard bandwidth, we try to design an adjustment mechanism that could be both automatically learned (absent in CFAR) and data-driven (missing in PKC), and that leads to AdaPKC.

Dear Reviewer,

We want to express our sincere gratitude for the time and effort you've dedicated to reviewing our paper. Your feedback has proven to be invaluable in elevating the quality of our work. We're pleased to inform you that we've taken great care in addressing each of the questions and concerns you raised in your reviews. Please do not hesitate to reply if you have any further questions or suggestions!  We look forward to improving the clarity and depth of our work with your valuable input!

Warm regards,

Paper 9766 Authors

Dear Reviewer ixjc,

We have thoroughly analyzed your questions and concerns you raised in your reviews, dedicating substantial time and effort to provide comprehensive explanations and corresponding revisions. We believe our responses should effectively address the issues you raised. We sincerely hope you could take the time to review our responses. If our response is adequate, we kindly ask you to give a fair score upgrade. Should you have any other concerns, we are eager to engage in further discussions with you. Thank you for your valuable input in enhancing the quality of our paper, and we also appreciate your respect for our hard work.

9766 Authors

Dear Reviewers,

Sorry to bother you, but this is the last reminder from the authors. We sincerely invite all reviewers to review and evaluate our responses. We thank you for your hard work in the early stage and hope to get your short attention again.

Since there are only three hours left for discussion, we attach great importance to this submission and would like to buy you a little time at the end.

Warm regards,

9766 Authors