Probabilistic and Differentiable Wireless Simulation with Geometric Transformers

We thank reviewer yJCq for providing a detailed review of our paper. We are happy to read that they found it logically structured and that they appreciated the novel treatment of forward and inverse problems with a diffusion model. Now, we address reviewer yJCq’s concerns and also indicate if it is shared by fellow reviewers.

Limited innovation... contributions of our WiGATr vs. GATr [9] .... any additional advancements?
We believe it is unfair making a direct contribution comparison of our work with GATr [9], since both address different problems. GATr [9] tackles the problem of representing geometric data and introduces a novel architecture. In contrast, our work tackles a differentiable wireless simulation problem of prediciting wireless signal characteristics from 3D scenes. 3D scenes are inherently geometric, and to the best of our knowledge, we are the first to exploit relevant symmetries and find GATr well-suited for this task. Apart from our model contributions (e.g., tokenization, generative formulation), we tackle inverse problems in wireless (e.g., receiver localization) and propose two new large-scale datasets.

Paper mentions neural statistical models [19, 40, 42, 56] and neural ray tracers [29, 41, 58] ... however evaluation section only compares two methods ... considerations made when choosing baselines?
We choose baselines that tackles the specific problem considered in the paper: jointly modelling the 2d/3d representations (e.g., mesh, image) of the propagation environment and wireless characteristics (e.g., path loss, receive signal strength). Neural statistical models [19, 40, 42, 56] are not applicable since they typically ignore scene representations. Neural ray tracers [29, 41, 58] are more related to our work. We now introduce Sionna RT [29] as a baseline (see Fig. 3 in rebuttal pdf; faster than ours, but larger errors). The other two are not applicable: [41] does not generalize to novel scenes and [58] does not consider geometric representation. As a result, we now compare against five relevant baselines that jointly model the 2d/3d representation of the scene and wireless characteristics: PLViT, SEGNN, Transformer, Sionna-RT*, Naive Transformer* (* = based on suggestions from reviewers). In all cases, we find our previous results hold: our approach outperforms all baselines in accuracy of predicted signal strengths.

Gap between the challenges and the solutions ... lack of analysis on these issues makes the proposed solutions appear abrupt ... recommend to provide insights that lead to the proposed solution
Thanks for the suggestion. We will use the extra page in final version to connect the challenges to our proposed solutions, e.g., the inductive biases improve data-efficiency, transformer tokenization scheme facilitates processing of 3D geometry.

Qualitative comparison with ViT ... quantitative with PLViT and SEGNN ... why the difference
Good catch, the "ViT" label in Fig. 1 is a typo and should read "PLViT". We will fix this for the next version. We were able to evaluate SEGNN only on simpler datasets (only Wi3R) due to its memory requirements.

Fig. 5 lacks necessary explanations
Due to space constraints, we moved some experimental details for our probabilistic model to the Appendix D.2. Furthermore, Figure 9 compares the behavior of generated geometries with the ground truth. In Figure 5 (c), we omit a direct comparison to the ground truth, as this inverse problem admits a variety of solutions. Instead, we visualize two diffusion samples conditioned on two different values for the signal strength respectively. As the signal strength decreases, we observe that the model is more likely to occlude line of sight between the receiver and transmitter locations, which mimics the dynamics of the training dataset.

This paper introduces both differentiable prediction models and diffusion models, claiming that diffusion models overcome the limitations of differentiable prediction models. Could the author conduct experiments to compare these two models and validate this claim?
Due to their different learning objectives, diffusion and prediction models are not easily comparable directly in terms of mean absolute errors. For example, when solving the inverse problem of receiver localization, the diffusion model learns a distribution of points (see Fig. 5b), whereas the SGD-based solution with the prediction model can only recover a single mode of this distribution. This inherent measure of uncertainty is one of the main strengths of diffusion over predictive modelling. However, when learning unimodal distributions (such as signal prediction, which is assumed to be deterministic in our setup), we have confirmed that the diffusion model performs comparably to the predictive model, as can be seen when comparing Fig. 8 with Fig. 3.

Typographical errors ... L56 L57 ... transmitter localization
Thanks for pointing them out. We will fix them.

We thank the reviewer again for their detailed feedback. We hope we were able to address their concerns and look forward to discussing further.

We thank reviewer 6syy for providing a detailed review of our paper. We are glad that they appreciated the generality of the approach, and in particular that they found our paper well-written and easy to follow. Now, we address reviewer 6syy’s concerns and also indicate if it is shared by fellow reviewers.

Several fast and differentiable ray tracers ... such as Sionna RT exist ... comparison missing* (common concern - 6Syy, yJCq)
Thanks for the suggestion. For this rebuttal, we provide additional experiments about Sionna RT (see Figs. R2 and R3 of the rebuttal pdf). We will add this result to our appendix. Although Sionna RT is faster than our approach, we observe large errors (12 dB without calibration, 8 dB with calibration) on the WiPTR validation set, compared to the proposed approach (0.74 dB). This is mainly because Sionna does not implement a model for transmission/refraction. Its predictions for the received power behind walls are therefore systematically too low. It also leads to faster computation as rays split at refraction events and the complexity of the ray tracing increases. In constrast, our approach is fully data-driven, not limited to certain physics models, and is thus able to learn interaction phenomena, including transmission, from the training data.

Unclear why Sionna-RT [29] is cited as a non-differentiable raytracer
Thank you for the catch, this (line 26) is a typo. We apologize and will fix it. All other references to Sionna-RT (e.g., line 92) mention differentiability.

Main motivations of the paper ... most raytracers are too slow ... ignores Instant RM which computes coverage maps in a few milliseconds
InstantRM is concurrent work -- the code was released 20 days before submission deadline, and publicized on a blog [A] 1 month after submission deadline. We are glad to cite, discuss and potentially compare in the next version of the paper. Yet, the models used in InstantRM are more limited than Sionna RT in terms of accuracy. Our arguments about Sionna RT (lack of transmission) apply thus to InstantRM as well.

[A] https://developer.nvidia.com/blog/fast-and-differentiable-radio-maps-with-nvidia-instant-rm/

Authors claim that channel impulse responses can be generated, this is not demonstrated in the paper
We appreciate that the reviewer points this out. We explicitly mention modeling of received power (e.g., lines 12, 50, 56). We would like to reduce potential ambiguity in this regard. Could the reviewer please point us to the exact statement that seems like we are claiming that we are generating the full channel impulse response?

Description of scene geometry recovery is incomprehensible
Thanks for pointing this out. We will rewrite this section and improve clarity.

Add confidence information (e.g., standard deviation) in Fig. 3 and Fig. 4
Thanks for the suggestion. We agree and will add measures of uncertainty to our results for the final version.

Scalability to very large datasets and extremely complex scenes is unclear (common concern - LEve, 6Syy)
We already evaluate our approach on a large test dataset (e.g., 1K diverse scenes in WiPTR proposed in this paper, $>5.5$M channels in total) -- this is significantly larger than previous efforts. In addition, we ran scalability experiments and found proming results. As shown in Fig. R4 in the rebuttal pdf, WiGATr remains at sub-second latency until $\sim$10k mesh faces. Although scenes with 500k mesh faces take up to an hour with our vanilla implementation, this is faster than a conventional, GPU-optimized ray tracer based on preliminary experiments.

Unclear whether the method generalizes to electromagnetic environments that are nonreciprocal
This is a great observation. By default, the equivariance properties of our approaches consider reciprocity. However, if necessary, this can be locally disabled by providing additional orientation information that breaks the symmetry of the problem.

Unclear whether the method generalizes to scenarios in which ray-tracing is inaccurate
The accuracy of our approach depends on the accuracy of the underlying training data. As long as the training data is generated by ray tracing, data-driven models cannot compensate for this shortcoming. However, Wi-GATr could also be trained or finetuned on measurements and then yield accurate predictions in scenarios where ray tracing is inaccurate.

How well would a neural-network-based positioning pipeline do on the task used to generate Figure 4?
To the best of our knowledge, existing positioning pipelines cannot be used in our task since they assume measurements from the identical environment at both training and test time. In contrast, our approach is "zero-shot": positioning is evaluated on unseen environments at test time.

We thank the reviewer again for their constructive feedback. We hope we were able to address their concerns and look forward to discussing further.

We thank reviewer 2AZ5 for providing a detailed review of our paper. Overall, we are glad that the reviewer finds using transformers for radio propagation modeling an interesting idea and appreciates that we include relevant information in our environment input. Now, we address reviewer 2AZ5’s concerns and also indicate if it is a common concern shared by fellow reviewers.

Problem is simulation to reality gap ... relevance to real-world environment
Indeed, mitigating the simulation-to-reality gap is an important problem. However, we find it complementary to the problem considered in this paper: modelling wireless characteristics of novel 3D propagation environments by exploiting symmetries. We believe solving this problem in a large-scale controlled setting (be it simulation or real) is a precursor to the problem of simulation-to-reality gap.

Benchmarks with real-world data ... is missing (common concern - reviewers LEVe, 2AZ5)
Firstly, we remark that evaluations on simulated data have their own merits: it allows evaluation on a large-scale (e.g., diverse environments, locations of transmitter/receiver) in a controlled setting and enables better analysis of approaches and results. While we agree with reviewers that it would be ideal to extend our evaluation to real-world datasets, we are limited by their availability. To the best of our knowledge, no relevant large-scale real-world dataset exist for the problem we tackle and hence previous works [29, 41, 35, 25] have also largely relied on simulated datasets. In fact, even the simulated datasets that exist (e.g., Wi3Rooms [41], Etoile/Munich [29]) are small-scale with low diversity in terms of 3D environments. This lack of high-quality wireless datasets with diverse scenes motivated us to generate and release two new datasets with this paper. Our symmetry considerations apply in the real-world and in simulation, thus our key message - that wireless channel modelling is a geometric problem and one should use symmetry-aware architectures and algorithms to solve it - should hold both in simulation and real-world measurements.

How well proposed method simulates various wireless environment ... evaluation is missing
We are unclear on what "various" corresponds to here and would appreciate the reviewer's clairification. Nonetheless, we believe we evaluate our approach in a fairly diverse setup (e.g., >1K novel scenes, i.i.d.\ 3D locations of tx/rx) improving upon prior work (e.g., [41] on 1 scene, [29] on 1 scene, [58] evaluates on 50 scenes). This is inline with comments of other reviewers e.g., "thorough evaluation ... various tasks ... multiple baselines" (LEve)

Several high fidelity radio propagation modelling software ... important to consider datasets from such models
We use the commercial ray-tracer "Remcom Wireless InSite" (see Sec. 4 in the paper) that is popular in the literature ([41, 3]). Please let us know your concrete concerns or missing features of this software.

Modelling of radio environment is not clear ... diffration/refraction not considered
This is incorrect. All simulated paths include diffraction, transmission, and reflection effects; see Table 4 in Appendix C for details. We will clarify in the main paper.

Mobility is not considered
Mobility is an interesting and relevant wireless channel modeling research problem. However, this complements our work that studies on generalization and symmetries of novel 3D scenes.

Evaluation limited to indoor environments (common concern - reviewers LEve, 2AZ5)
Evaluation on outdoor scenarios would make a great addition. However, the physical phenomenons (e.g., diffraction, reflections) of wireless signals would largely remain the same as a function of the 3D geometry in both indoor or outdoor settings. Nonetheless, in this paper, we choose indoor scenarios since we believe it is a more challenging scenario to evaluate influence of 3D scenes towards wireless characteristics. For instance, heights of surfaces play a bigger role (since signals can reflect off floors and ceilings) and transmissions through surfaces have significant contributions towards receive powers in non-LOS regions.

NNs already shown to be useful for modelling wireless propagation ... how accurate is this yet another architecture
Indeed, ML-based approaches are shown to be successful for modelling wireless propagation. These studies are often limited to 2D representations of the scene (e.g., binarized satellite images). However, wireless propagation is inherently influenced by the 3D structure of the propagation environment and prior approaches exhibit large errors in these scenarios (details in Sec. 5.1). In contrast, our novel equivariant architecture exploits certain symmetries of the 3D environment and significantly outperforms prior relevant ML-based approaches.

Table 2 ... 80 dBm error ... explain large error of baseline
Thank you for highlighting this result. It is one of the main arguments of our paper. The baseline is strong on in-domain data (e.g., MAE of 1.32 dB on the unseen floor plans, see Table 2). However, the generalization to out-of-distribution data is a significant failure mode (e.g., MAE of 78.68 dB on rotated data). In contrast, our proposed Wi-GATr architecture improves the robustness under domain shifts by incorporating the symmetries of the problem. By construction, it is robust to translations and rotations of the scene; we also find an improved robustness to other out-of-distribution test sets.

We thank the reviewer again for the thorough review. We hope we were able to clear up some concerns and look forward to discussing more.

We thank reviewer 3vGS for providing a detailed review of our paper. Overall, we are glad that the reviewer appreciates our novel wireless tokenization scheme, our diffusion approach, and that we contribute a novel large-scale dataset to the machine learning community. Now, we address reviewer 3vGS’s concerns.

Clarify the challenges ... novel tokenization
The main challenge is the representation of diverse geometric types involved in wireless scenes: points, rays, orientations, and surfaces. Most popular representations of 3D geometry are designed for either point clouds or meshes or rays, but not their combination. In constrast, our tokenization scheme provides a unified way of representing different combinations of diverse geometric types.

Same tokenization used both in WiGATr and vanilla transformer baseline ... benefits of tokenization unclear
Great suggestion, we indeed did not clearly demonstrate the benefits of the new tokenization scheme in the original paper. We add this ablation now and find promising results. In Fig. R1 in the rebuttal result page, we compare a new "naive transformer" model that does not use our geometric wireless tokenizer to the existing "transformer" result based on the tokenizer. We find that tokenization has a drastic impact on performance: for instance, when training on 1000 rooms, using our wireless tokenizer improves the transformer performance from 2.9 dB to 1.7 dB mean average error. We will provide additional details for this experiment in the final version of the paper.

Clarify how many transmit antennas were used in receiver localization and sensing
We study SISO omni-directional antennas throughout the paper. We vary the number of SISO transmitters used for receiver localization in Fig. 4 of the paper.

Clarify why results are promising despite the limitations of using a larger number of antennas and ignoring time and angle of arrival information
We have for now focused on SISO received power prediction as an evaluation task as it has been discussed in the relevant literature before. Adaptation of the tokenization scheme is an interesting idea, as well as extendening the output other channel characteristics. Our considerations about symmetries of the underlying physics still apply.

Clarify the challenges ... diffusion-based models with the Wi-GATr model
One of the main challenges was in achieving a good performance on inverse problem solving using our diffusion model. To maximize the performance of both joint (unconditional) inference as well as marginal (conditional) predictions within the same model, one needs to tune the trade-off between unconditional and conditional log likelihood loss terms during training. Finally, the use of the DDIM scheduler proved to be crucial for achieving sufficiently fast sample generation during evaluation.

only in terms of time-averaged non-coherent received power ... missing time and direction of arrival
In our experiments, we indeed restrict ourselves to non-coherent receveived power to focus on the machine learning challenges (3D equivariance) that wireless channel modeling poses. We agree that, depending on the downstream use case of channel models, further channel information is required---and it is straightforward to embed with our tokenizer and process with our Wi-GATr backbone. Nonetheless, learning large-scale characteristics such as received power is an important open-problem and an active area of research [35, 29, 24].

Single antenna, does not cover larger bandwidths ... non-linearities can occur with larger bandwidths
Our setting is reflective of infinite bandwidth, since we sum up powers of individual physical paths (see line 245). Our model is trained directly on the received power and is agnostic to system-level details (e.g., bandwidth).

We thank the reviewer again for their detailed feedback. We hope we were able to address their concerns and look forward to discussing more.

We thank reviewer LEve for providing a detailed review of our paper. We are glad that the reviewer appreciates our equivariant approach to model diverse environments, the strong performance in our thorough evaluation, and the data efficiency and versatility of our approach on forward and inverse problems. Further, we appreciate the useful comments to improve our manuscript.

Now, we address reviewer LEve’s concerns and also indicate if it is a common concern shared by fellow reviewers.

Limited real-world testing (common concern - reviewers LEVe, 2AZ5)
Firstly, we remark that evaluations on simulated data have their own merits: it allows evaluation on a large-scale (e.g., diverse environments, locations of transmitter/receiver) in a controlled setting and enables better analysis of approaches and results. While we agree with reviewers that it would be ideal to extend our evaluation to real-world datasets, we are limited by their availability. To the best of our knowledge, no relevant large-scale real-world dataset exist for the problem we tackle and hence previous works [29, 41, 35, 25] have also largely relied on simulated datasets. In fact, even the simulated datasets that exist (e.g., Wi3Rooms [41], Etoile/Munich [29]) are small-scale with low diversity in terms of 3D environments. This lack of high-quality wireless datasets with diverse scenes motivated us to generate and release two new datasets with this paper. Our symmetry considerations apply in the real-world and in simulation, thus our key message - that wireless channel modelling is a geometric problem and one should use symmetry-aware architectures and algorithms to solve it - should hold both in simulation and real-world measurements.

Scalability and Computational Load (common concern - reviewers LEve, 6Syy)
In Fig. 7 in Appendix D.1, we show timing measurements for a single room. Predicting the received power with Wi-GATr takes 0.02 s per Tx-Rx link, which is substantially faster than the ray tracers with similar accuracy.

Thanks for the suggestion to study scalability to larger scenes, which we add in Fig. R4 in the rebuttal result page. Wi-GATr remains at sub-second latency for up to 10k tokens. The quadratic compute scaling of the attention mechanism with the number of tokens means that larger scenes with millions of mesh faces take hours to evaluate.

Generalizability Across Frequencies
Evaluating generalization across frequencies makes for an interesting experiment. However, the focus of this paper was generalization across scenes, since we believe this plays a larger role on wireless characteristics (unlike carrier frequencies, which are typically regulated and fixed). Nonetheless, our approach does not make any assumption on the carrier frequency and would make an ideal candidate for this experiment. We will consider this in future.

The paper focuses only on indoor evaluation ... considered outdoor/mixed? (common concern - reviewers LEve, 2AZ5)
Evaluation on outdoor scenarios would make a great addition. The physics (e.g., diffraction, reflections) of wireless signals would largely remain the same as a function of the 3D geometry in both indoor or outdoor settings. Nonetheless, in this paper, we choose indoor scenarios since we believe it is a more challenging scenario to evaluate influence of 3D scenes towards wireless characteristics. For instance, heights of surfaces play a bigger role (since signals can reflect off floors and ceilings) and transmissions through surfaces have significant contributions towards receive powers in non-LOS regions.

Comparison of inverse problems with traditional methods
We have provided an additional comparison to differentiable ray tracing (see Sionna in rebuttal pdf Fig. R2) on the forward problem and conclude it is not applicable to the inverse problem given its current accuracy. Other neural network methods we are aware of require data from the same scene. We evaluate on novel scenes not seen during training. We appreciate pointers to relevant methods that apply to this generalization setting.

How does Wi-GATr perform in real-time applications? ... Can it be integrated into real-time wireless network management systems ... what are the challenges?
The answer depends on the application and the required latency. Our approach predicts at sub-second latency for scenes with up to 10k mesh faces, see Fig. R2 and R4 in the rebuttal pdf. However, certain applications will require faster inference speeds. For instance, receiver localization through gradient descent might be too slow for real-time applications. Nonetheless, we expect that the inference speed of our architecture can still be optimized using standard tricks (e.g., compilation of computation graph, distillation, efficient diffusion samplers).

Have you considered additional data sources ... satellite imagery, LIDAR ... to enhance the model's accuracy and applicability
This is an interesting idea and would further highlight the versatility of learning-based simulation approaches. We have not considered it in this paper.

How does the model adapt to changes in the environment ... such as new constructions
Our approach takes the description of the environment as input, and our predictions adapt naturally to changing conditions. For instance, one of our out-of-distribution test scenarios "OOD layout" considers removing walls (see Table 2 of main paper, WiPTR dataset) and we observe Wi-GATr adapts. Similarly, our network predictions will change if the materials are changed.

Detailed case studies ... practical applications
We have shown the application of model to receiver localization in our extensive evaluation. This problem has practical use cases in robotics or, in the form of transmitter localization, in base station placement.

We thank the reviewer again for their feedback. We hope we were able to address their questions and look forward to discussing further.