RETR: Multi-View Radar Detection Transformer for Indoor Perception

Thank you for the detailed comments and valuable feedback! We provide our point-to-point responses below. Due to space constraints, we have shortened your comments for brevity.

Most commercial radars have 2D virtual antenna arrays, e.g., 16*8, instead of only a pair of 1D antenna arrays. Is there any specific reason for considering such kinds of radar antenna arrays?

Thanks for raising this point. The main reason for using a configuration of two cascading radars in horizontal and vertical orientations is to harness finer angular resolution in both azimuth and elevation domains to support all three perception tasks, especially for pixel-level segmentation. Both HIBER and MMVR datasets used this configuration.

Most commercial radars have a typical configuration of $4$ Rx and $3$ Tx antennas, yielding a virtual array of $8$ elements in one angular dimension and $4$ in the other. Examples include TI's IWR1443 and IWR1843 chipsets and, more recently, NXP's TEF81xx and TEF82xx series. This usually leads to an angular resolution of $15^\circ$ in one angular dimension and about $30^\circ$ in the other angular dimension.

The configuration of two cascading radars ($12$ Tx and $16$ Rx) yields a virtual array of $86$ non-overlapping half-wavelength-spaced elements in both vertical and horizontal dimensions, offering an angular resolution of $1.3^\circ$, more than $10 \times$ better. The resulting high-resolution multi-view radar heatmaps can provide fine-grained radar features and support not only BBox estimation and pose estimation but also the more challenging pixel-level segmentation.

The explanation in Section 2 does not fit the real process of multi-view radar heatmap generation. Better to introduce what types of radar the dataset used.

Due to space constraints, the current version of Section 3 - Generation of Radar Heatmaps presents an abbreviated description of the full process by skipping steps such as MIMO waveform separation for virtual array processing, integration over the Doppler domain, and projection onto the azimuth and elevation domains. In the updated paper, we plan to include a new section in the Appendix to introduce the dual (horizontal-vertical) radar configuration and provide a detailed explanation of the generation of the two radar heatmaps.

In the experiments, only radar heatmap-based methods are used for comparison. Better to incorporate the results of radar point cloud-based and camera-based approaches for a more complete comparison.

Thanks for the suggestion. We have two response points:

1. Dataset: To the best of our knowledge (see Table 4 in the PDF of the global response), we cannot find an indoor radar dataset with both radar heatmap and point cloud formats for evaluating a given method (either RETR or a baseline).

2. Baseline: We have included DETR, a camera-based method, in our baseline comparisons. We will make an effort to incorporate a radar cloud-based baseline in the updated paper.

The survey of related work is also not complete, ignoring recent works in radar-based object detection for autonomous driving [1-3], which also use radar heatmaps: [1] Liu, Yang, et al... [2] Paek, Dong-Hee, ... [3] Skog, Mikael....

Thanks for pointing out these recent radar datasets featuring radar heatmaps. We will include them in the updated paper. Specifically, we plan to discuss [3] in Section 2 - Related Work, as it is particularly relevant to our task of human perception using radar heatmaps.

Line 28: better to explain why radar heatmaps are more preferred for challenging perception tasks.

In the updated paper, we plan to highlight the difference between radar heatmaps and point clouds. We will also explain how high-resolution, multi-view radar heatmaps may be better suited for supporting more challenging perception tasks, such as pixel-level segmentation.

The relationship and difference between previous works in radar-based detection and segmentation...is not well explained.

In Fig. 2 of the main paper, we intended to highlight the major differences between RFMask and the proposed RETR. RFMask uses regional proposals and features from only the horizontal radar view with a fixed height. RETR, on the other hand, employs a detection transformer, fuses multi-view radar features, and exploits the unique multi-view radar setting via TPE and radar-to-camera coordinate transformation. We will expand the RFMask with Refined BBoxes section in the Appendix to clarify these differences further.

Better to show the tokenization step in Figure 3....

see Fig. 2 of the PDF in the global response.

implement the top-K feature selection? And how to supervise...?

For the Top-$K$ selector, we just select feature tokens with the highest norms computed over the channel dimension. We observed that direct supervision is unnecessary for this step because radar features tend to be extremely localized. Please refer to Fig. 6 of the main paper for visualizing the cross-attention between selected features and detected BBoxes.

Line 229, why the author claimed that the calibration may be only accurate for a limited interval of depth and angles.

This is mainly due to the varying cross-range radar resolution over depth and angles as the radar operates in a polar coordinate system. For a given angular resolution, the cross-range cell resolution at $3$ meters will be about $3\times$ larger than that at $1$ meter. To compensate for such resolution differences, one may need to repeat the calibration for different depth/angular intervals.

The MMVR dataset can not be found online.

We reached out to the MMVR authors who provided us with the "P2" split at the time of submission. The MMVR dataset should be available now by searching "MMVR Dataset", although we are not allowed to share any links in the rebuttal.

Minor questions: Line 31:... Line 135: .....provide captions for table 1 and 2

We will make the changes accordingly.

Thank you for the detailed comments and valuable feedback! We provide our point-to-point responses below.

Radar perception datasets primarily utilize point clouds as the data form. Methods based on heatmaps for indoor multi-view radar perception are not yet sufficient. The author mainly compares with customized DETR and RFMask, but the comparison is insufficient. Have the authors considered conducting further comparison experiments on the HUPR [1] dataset? If the authors can further demonstrate the method's generalization ability, I would consider giving a higher score.

Borrowing Table 1 from the MMVR paper, also listed as Table 4 in the PDF of the global response, you are correct that indoor radar perception datasets primarily use point clouds. However, an increasing number of datasets employ multi-view radar heatmaps to support more diverse perception tasks: RF-Pose, HuPR, HIBER, and MMVR, with the latter three collected since 2022. RF-Pose and HuPR have a resolution of $15^\circ$ in the two-view radar heatmaps, while HIBER and MMVR offer a finer resolution of $1.3^\circ$.

Thanks for pointing out the HuPR dataset. First, HuPR focuses on pose estimation (keypoints), while our work targets object detection (BBox estimation) and segmentation (pixel-level masks). To utilize the HuPR dataset for training and evaluating our RETR pipeline, we would need to extract bounding box and segmentation labels from each HuPR frame which, unfortunately, cannot be completed within the rebuttal period.

Second, even with BBox and mask labels from HuPR, our RETR pipeline requires geometric information about the radar and camera coordinate systems, including the radar-to-camera transformation (rotation matrix and translation vector) and the 3D-to-2D camera projection matrix (pinhole camera model). Both HIBER and MMVR datasets provide this calibrated information. While our learnable radar-to-camera transformation can reduce some geometry dependency, the 3D-to-2D projection matrix is still necessary. We have reached out to the HuPR authors for this additional geometric information. Once we receive it, we will report the RETR performance on at least BBox estimation.

Third, we believe that the proposed RETR pipeline and its evaluation over two separate datasets (HIBER and MMVR) have sufficiently demonstrated the generalization capability. It is noted that the radar-to-camera geometries in these datasets are completely different but known (via calibration or learning). RETR is purposely designed to handle these differences by incorporating calibrated/learnable radar-to-camera transformations, 3D-to-2D projections, and tri-plane loss functions.

The method presented is an improvement on the DETR and combines existing methods in a novel way. While innovative, it's not highly original for NeurIPS.

We believe our contributions to be novel and original as they are not a simple combination of existing methods. Indeed it is true that we build upon DETR, yet we propose several contributions that make our RETR unique. First, we highlight how we adapt DETR to the multi-view scenario thanks to the self-attention mechanism combined with the Top-K selection and by reusing the cross-attention mechanism to avoid traditional object-to-multi-view-feature association.

Second, we introduce a depth-prioritized feature similarity via a tunable positional embedding (TPE), incorporating a crucial inductive bias of shared depth across the two radar views to enhance downstream tasks.

Third, we propose a tri-plane loss from both radar and camera coordinate systems which, to the best of our knowledge, has never been considered for object detection from radar heatmaps.

Research on indoor multi-view radar perception using heatmaps is relatively scarce. The author is unable to provide source code, potentially limiting contributions to this field.

We plan to release our code after the paper acceptance.

Question 1: Could you provide a detailed description of the design of the Top-k selector?

Regarding the Top-$K$ selection, we begin with the features map extracted from the shared backbone. Each cell in the feature map can potentially be considered as a patch/token for input into the subsequent transformer encoder. To alleviate the time complexity of the attention module, we select only the tokens with the highest norms computed over the channel dimension. In this way, we propagate only the most relevant information to the following modules and increase the inference performance, while keeping the complexity low.

In Fig. 6 of the main paper, we visualize selected top features (and their locations in the radar heatmap) in the two radar views that are used to detect bounding boxes in the image plane by inspecting the cross-attention module.

In Table 1 of the PDF in the global response, we have included an additional ablation study examining the impact of $K$ on detection performance.

Question 2: Can you provide a more detailed analysis of the impact of the proposed RETR on real-time performance?

We report the inference time in Table 3 of the PDF in the global response. We used an NVIDIA A40 GPU for evaluation. RETR has a comparable inference time as RFMask ($20.89$ ms against $23.75$ ms of RETR). We compute the average inference time across all radar frames in the test data.

Thank you for the detailed comments and valuable feedback! We provide our point-to-point responses below.

For indoor perception with radar, multi-path is expected. It would be good to add a paragraph to discuss this issue and how to mitigate multi-path or its impact to the perception/segmentation performance.

Thanks for the insightful comment. We agree that multi-path reflections from the ground, ceiling, and other strong scatterers (e.g., metal) can cause (first-order or second-order) ghost targets and elevate the noise floor. One way to address this issue is to employ classical signal processing techniques into the radar heatmap generation to remove these ghost targets and the static background reflection.

We may address this issue directly in the end-to-end radar perception pipeline by labeling these ghost targets in standard radar heatmaps (although it is difficult and costly to have these labels) and directly classifying RETR object queries to one of {$\{\emptyset, person, ghost\}$}, alongside regressing queries to the bounding box parameters. As you suggested, we will add a paragraph to discuss the multi-path issue and potential ways to mitigate its impact.

The motivation of applying transformer for indoor segmentation is not well discussed. What are the benefits and unique challenges of applying transformer to radar based indoor segmentation?

We agreed that in the main paper, we primarily used the object detection example to motivate the RETR, while the segmentation part was deferred to Appendix B Segmentation. In the updated paper, we will emphasize that the segmentation is an integral part of the RETR by highlighting the following points in the main paper:

1. The segmentation head uses the estimated bounding box (BBox) as a prior or constraint to classify each pixel within the BBox (see Fig. 8 Illustration of Segmentation Head in Appendix).

2. The pretrained RETR components, such as the backbone, top-$K$ selection, and the detection transformer are reused with frozen weights to train the segmentation head (the lower branch) in Fig. 3 of the main paper.

We will point out the challenges in extracting finer-grained radar features, fusing features from two radar views, and utilizing the prior BBox from the detection head to support the pixel-level segmentation task. We will emphasize how to address these challenges by leveraging the DETR architecture to avoid cross-view radar feature association and introducing additional modifications, such as tunable positional embedding, radar-to-camera coordinate transformation, and tri-plane loss.

For the introduction of Generation of Radar Heatmaps, please consider citing the following paper: S. Sun, A. P. Petropulu and H. V. Poor, "MIMO Radar for Advanced Driver-Assistance Systems and Autonomous Driving: Advantages and Challenges," in IEEE Signal Processing Magazine, vol. 37, no. 4, pp. 98-117, July 2020.

Thank you for the suggestion. We will add the suggested paper to the reference list.

Is there any interference (cross-path signals) between the two radars deployed in vertical and horizontal directions?

Thanks for raising this important point. To prevent cross-radar interference between the horizontal and vertical radar sensors, the two radars were configured to operate at different frequency bands. In the HIBER dataset, the horizontal radar operates in the $77 - 78.23$ GHz band, while the vertical radar is in the band of $79-80.23$ GHz. In the MMVR dataset, the horizontal radar operates in the $77-78.36$ GHz band, while the vertical radar is in the band of $79-80.36$ GHz. In both cases, there is a minimum gap of $500$ MHz between the operating frequency bands of the two radars.

Usually a large amount of high quality radar data is required to train the transformer. It is highly recommended to carry out validation with different amount of training data.

Good point. In Table 2 of the PDF in the global response, we report the impact of training data size on detection performance using the MMVR dataset. We compare the original data size (x1.0) with $190,441$ radar frames against reduced data sizes of half (x0.5) and one-tenth (x0.1). The result shows a gradual improvement in detection performance with an increase in data size, particularly at higher IoU thresholds, such as AP$_{75}$.

Thank you for the detailed comments and valuable feedback! We provide our point-to-point responses below.

I am wondering how the Top-K Feature selection impacts the performance of the network. How can we choose the number of K? It seems that there is no experiment to validate the choice of K. It will be better if the authors conduct an ablation study to explore the effectiveness of choosing K.

We appreciate Reviewer DJHb's suggestion. In response, we have conducted additional experiments to examine the impact of $K$ on detection performance. The results, detailed in Table 1 of the PDF in our global response, indicate that increasing $K$ improves object detection performance. However, as noted in our detailed response below to your next comment on the computation cost and inference time, choosing a larger $K$ also significantly increases the training and inference time.

What is the computation cost of the proposed RETR model? It will be better if the authors report the inference time/computational cost of the proposed model.

Thanks for pointing out the computation cost and inference time.

First, following the computational complexity notation used in the DETR paper, every self-attention mechanism in the encoder has a complexity of $\mathcal{O}(d^2 2K + d (2K)^2)$ where $d$ is the embedding dimension and $K$ is the number of selected features from the Top-$K$ selection. The cost of computing a single query/key/value embedding is $\mathcal{O}(d' d)$ (with $d=Md'$ where $M$ denotes the number of attention heads and $d'$ the dimension in each head), while the cost of computing the attention weights for one head is $\mathcal{O}(d' (2K)^2)$. Other computations may be negligible. In the decoder, each self-attention mechanism has a complexity of $\mathcal{O}(d^2 N + d N^2)$ where $N$ is the number of queries, and the cross-attention between query and multi-view radar features has a complexity of $\mathcal{O}(d^2(N + 2K) + d 2NK)$. In conclusion, the overall complexity of our RETR model is $\mathcal{O}(4d^2 K + 4d K^2 + 2d^2 N + d N^2 + 2d NK)$.

Second, regarding the inference time, we report the average inference time in milliseconds in Table 3 of the PDF in the global response. We used an NVIDIA A40 GPU to evaluate the inference time over all frames in the test data. RETR achieved an average inference time of $23.75$ ms that is comparable to $20.89$ ms of RFMask.

For the learnable radar-to-camera coordinate transformation, while I acknowledge the performance improvement according to this proposed model, I have several questions related to this module. Does the entire dataset use the same set of learnable vector and translation vector ? Or each radar sample in the data will have a different vector and translation vector ?

The learned vectors $\omega$ and $t$ (or, equivalently, the rotation matrix $R$ and translation vector $t$) are fixed and applied consistently to all test frames.

During training, $\omega$ and $t$ were updated from one minibatch to the next, as they are part of the learnable parameters in RETR.

How can we verify that the learnable radar-to-camera coordinate transformation is accurate? Is there any way to evaluate it using the ground truths?

Good question. One way to verify the coordinate transformation learning is, for a given point in the 3D radar coordinate, by checking the distance between the two transformed points using calibrated and learned coordinate transformation. Although the calibrated coordinate transformation (including both rotation matrix and translation vector) is NOT ground truth due to radar resolution (also see our responses to Reviewer XrXa), it serves as a reasonable baseline or reference benchmark for the learned coordinate transformation.

In the PDF of the global response, Fig.1 shows the distance difference between the two transformed points over the training steps using the MMVR dataset. We randomly initialize the learned rotation matrix and translation vector at iteration 1. The results demonstrate that, as training progresses, the learned radar-to-camera coordinate transformation becomes increasingly aligned with the calibrated one, indicating that the learning is moving in the correct direction.

Finally, we'd like to add that by using the learnable radar-to-camera coordinate transformation, it is possible to incorporate the radar-to-camera geometry into the end-to-end radar perception pipeline without the need for a cumbersome calibration step, while still achieving comparable perception performance.