MapLearn: Indoor Mapping using Audio

Q1: How to calculate envelope h?

Response: Thanks, we should have explained this better. We use Hilbert transform. We will make it clearer in the paper.

Q2: What is PatchGAN?

Response: We should have clarified this. PatchGAN is from the “Image-to-Image Translation with Conditional Adversarial Networks” paper in the reference. Instead of outputting a final real/fake value directly for the entire image, PatchGAN crops the image into patches and outputs a real/fake for each patch. A learnable function that fuses small patch results together is utilized for final real/fake decisions. This method has proven to be effective for recovering local details inside images.

Q3: Add P(d) and R(d) for figure 10.

Response: We report the P(40) and R(40) values for all the six floor plan visualizations here (we will add this table if the paper is accepted). The P and R values are all around 0.6, showing promising floor plan estimation result. For the full MapLearn pipeline, due to the fewer fake walls predicted, the P(40) value is overall better than that of HintMap only solution.

Q4: Directly use the envelope along with the hint map as the input.

Response: Thanks for the comment. The envelope should be used in conjunction with location information (i.e., the location where the measurement is being performed). We find that explicitly telling the network about measurement locations (and shifting the local map-patches to those locations) is beneficial. It is more generalizable than directly inputting the measurement location (x,y) as two other dimensions and relying on the neural network to learn.

Q5: Will the data and code be released for reproducibility?

Response: Yes, we will release the data and code if the paper is accepted.

Q6: Minor typos

Response: Thank you for identifying these issues. We will fix them.

Q1: Audio has low resolution leading to impaired mapping accuracy.

Response: True, audio signals have much longer wavelengths compared to optical cameras and LIDARs, hence much lower spatial resolution. However, this lack of resolution hides rich details about the environment, leading to better privacy. We think many in-home applications (Alexa, digital twins, localization) may favor coarse-grained maps and strong privacy as opposed to the vice versa; in this tradeoff, audio lies in the sweet spot. Finally, even extracting this coarse-grained information is not easy; traditional signal processing methods do not work well. We have demonstrated that using cGAN can be effective.

Q2: IMU location error will affect system performance.

Response: Yes, this is a valid comment. We acknowledge the accumulation of IMU error would impact mapping. However, a growing body of literature is showing improved IMU-based localization [1,2] using (re)calibration opportunities from the environment (using Wifi, BLE, magnetic, and other sensory landmarks, etc. [3,4,5]). In sum, IMU-based drifts can be reset periodically allowing the error to be bounded. We evaluated the sensitivity of map error as a function of IMU location error (Table 3). So long as the IMU error is unbiased, the mapping performance degrades gracefully with IMU error.

Q3: Application of the MapLearn pipeline.

Response: We mentioned the following to Reviewer 1 as well. Mapping with audio-reverberations is relevant when cameras or LIDARs are not acceptable (e.g., privacy concerns in homes, poor lighting conditions, and unavailability of LIDARs). Potential use-cases include:

• Digital twins of homes are useful for various applications, e.g., when away from home, a user wants to track her aging parent or her pet on a digital twin of the home floorplan. The floorplan is valuable for visualization.
• Conversational AI agents (e.g., Alexa and Google Home) intend to understand the spatial context of a conversation with a user. Knowing the user's location is useful, but contextualizing that location on an even coarse-grained floorplan offers much more context (e.g., user is in kitchen, living room, etc.). Mapping with cameras/LIDARs raise privacy concerns inside homes.
• Localization algorithms (e.g., particle filters) routinely use floorplans to update their posterior distributions. A coarse-grained floor plan with wall location uncertainties will still significantly improve trajectory estimation.
• Floorplans are useful to signal processing applications, e.g., WiFi radios may beamform better with an understanding of walls around it. On similar lines, 3D surround sound can be synthesized better with knowledge of floorplans.
• Firefighters may rapidly need to map out a floor plan or army troops may need to map out an enemy building; cameras may be inadequate due to smoke or poor lighting conditions. Audio reverberations (inaudible) should be more effective.

[1] Liu, Wenxin, et al. "Tlio: Tight learned inertial odometry." IEEE Robotics and Automation Letters 5.4 (2020): 5653-5660.

[2] Sun, Scott, Dennis Melamed, and Kris Kitani. "IDOL: Inertial deep orientation-estimation and localization." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 7. 2021.

[3] Wang, He, et al. "No need to war-drive: Unsupervised indoor localization." Proceedings of the 10th international conference on Mobile systems, applications, and services. 2012.

[4] Chintalapudi, Krishna, Anand Padmanabha Iyer, and Venkata N. Padmanabhan. "Indoor localization without the pain." Proceedings of the sixteenth annual international conference on Mobile computing and networking. 2010.

[5] Shu, Yuanchao, et al. "Magicol: Indoor localization using pervasive magnetic field and opportunistic WiFi sensing." IEEE Journal on Selected Areas in Communications 33.7 (2015): 1443-1457.

Reviewer 2

Q1: Motivation; why does acoustics not have privacy concerns?

Response: Audio sensing will record ambient voices and sound events, which is indeed a privacy leakage. However, we have used the ultrasonic audio bands of 20kHz-40kHz (mentioned at the end of page 5). Human voice and music have no footprint in these frequencies. As a result, we can preserve privacy. Additionally, note that cameras end up ``seeing'' details of the environment (e.g., bathrooms, bedrooms, closets, text and images of documents on the table and fridge, etc.). Audio preserves this notion of privacy as well.

Q2: What does hint map baseline indicate? How to interpret Fig 10 row 2 and 3?

Response: The hint-map is the floorplan prior estimated from only the first audio reflection. The later reflections capture additional environmental information and contribute to boosting the paper's final performance. We have used the hint-map as a baseline (and as an ablation study to compare the value of first vs. late reflections). In Figure 10, row 2 shows the ground truth and row 3 shows the estimation based on the hint-map alone. Row 4 shows the final results after including the late reflections. With hint maps, the estimated floor plan exhibits more fake walls because the neural network tries to insert walls at the boundary of open spaces. With the full MapLearn pipeline, fake walls are fewer.

Q3: If approximate floorplan suffices, why does P(d), R(d) matter?

Response: Thanks for recognizing this point. We have struggled to design reasonable metrics -- not too harsh, not too lenient -- that can capture the quality of learnt floorplans. Clearly, the application determines when an approximate floorplan is sufficient, but to develop some application-agnostic quantification, we decided on P(d) and R(d). This is by no means the perfect metric and we are very open to suggestions. We will continue to think about better metrics for this problem but believe that P(d), R(d) are reasonably fair measures of performance.

Q4: Application where a more accurate floor plan from audio is needed.

Response: When a user walks around a lot (covering say 80% of the home) then the hint-map generates a reasonably good floorplan (and the late audio reflections are less crucial). For such cases, the hint-map and MapLearn can both satisfy the same applications. However, when the user walks less, the hint-map degrades quicker and the value of late reflections becomes pronounced. If this paper is accepted, we will add results that visually compares floorplan accuracy between MapLearn and the hint-map, when the user has walked say 60%. This will highlight the additional accuracy gain from late reflections.

Now, what applications will need more accuracy than just a good visual approximation? Many indoor localization applications use particle filters [1] on a floorplan. Very briefly, a user's possible location is modeled by many particles, each associated with a probability. Each particle is a hypothesis of where the user may be located. As the user moves, the particles are propagated in the measured walking direction. When a particle moves through a wall, these particles must be removed since they are an infeasible hypothesis. Said differently, the floorplan helps update the posterior distribution of the user's location. The floorplan accuracy directly impacts the convergence (and accuracy) in such Bayesian methods for localization.

[1] Del Moral, Pierre. "Nonlinear filtering: Interacting particle resolution." Comptes Rendus de l'Académie des Sciences-Series I-Mathematics 325.6 (1997): 653-658.

Q1: How to get ground-truth for training?

Response: We trained our model entirely using simulated data from the ZinD 3D-scan dataset (cited and explained in Section 3). The ground-truth floorplans are part of this dataset. The real-world experiments were only performed during testing and results are promising (i.e., training on simulation data generalized to real environments). Of course, to report error for the real-world experiments, we obtained ground-truth floorplans from the facilities manager of the buildings.

Q2: What is the usage scenario?

Response: Mapping with audio-reverberations is relevant when cameras or LIDARs are not acceptable (e.g., privacy concerns in homes, poor lighting conditions, presence of fog or smoke, and unavailability of LIDARs). Potential use-cases include:

• Digital twins of homes are useful for various applications, e.g., when away from home, a user wants to track her aging parent or her pet on a digital twin of the home floorplan. The floorplan is valuable for visualization.
• Conversational AI agents (e.g., Alexa and Google Home) intend to understand the spatial context of a conversation with a user. Knowing the user's location is useful, but contextualizing that location on a floorplan offers much more context (e.g., kitchen, living room, etc.). Mapping with cameras/LIDARs raise privacy concerns inside homes.
• Localization algorithms or state estimation problems (e.g., particle filters) routinely use floorplans to update their posterior distributions.
• Floorplans are useful to signal processing applications, e.g., WiFi radios may beamform better with an understanding of walls around it. On similar lines, 3D surround sound can be synthesized better with knowledge of floorplans.
• Firefighters may rapidly need to map out a floor plan; cameras may be inadequate due to smoke and poor lighting conditions. Audio reverberations should be more effective.

Q3: Comparison with BatMapper.

Response: Thanks for bringing this up; we should have elaborated more in the paper. BatMapper uses a pure signal processing-based approach under the assumptions that: (1) floorplans are rectangular shaped rooms or corridors, and (2) users need to walk quite close to the walls. This is necessary because BatMapper only utilizes information from early reflections (similar to our hint map) to ultimately infer the parameters of the rectangle. Our model is crucially different since it uses all the audio reverberations (i.e., the room impulse response) to learn any layout of the environment (including partitions, L or U-shaped rooms, etc.). Results show that we need the user to walk less and our maps are far more complex than BatMapper (see our Figure 10 and 11 vs. BatMapper Figure 17).

Q4: Develop a prototype and test in a real building.

Response: We indeed implemented the prototype and performed the experiments in real world across 4 different buildings. Please find details regarding experiment setup in section 3 (Figure 8) and results in section 4 (Figure 11).

Q5: The need for ground-truth harms its application and user experience.

Response: We respectfully disagree with this. We only need ground-truth for training and since we train on simulation data, the ground-truth is easily available. We carefully performed the simulation (channel models developed on real signal propagation physics) so that it resembles real world as much as possible. This is why the proposed method works in real world as well. We do not need ground-truth when mapping an unknown floorplan in a real building (we validated this in 4 real buildings).