We appreciate the reviewers’ valuable feedback and constructive comments, which have helped us refine and improve our work. Thank you for your time and thoughtful consideration. Below, we provide detailed responses to the questions in the review.

Rationale for focusing on Imitation Learning. Thank you for your question. We chose to focus on imitation learning in this study because it aligns with the state-of-the-art approaches in the aerial VLN task [1]. In our study, we specifically combine imitation learning with a navigation map to investigate the effectiveness of using geographic information. By leveraging this approach, we can isolate and validate the impact of using navigation maps on improving aerial VLN performance. Using RL-based methods would introduce additional variables, potentially complicating the evaluation of the navigation map’s contributions.

[1] Liu et al., AerialVLN: Vision-and-language navigation for uavs. ICCV, pp. 15384–15394, 2023

Inference speed. Thank you for your question regarding inference speed. To address this, we measured the runtime performance of inference across a subset of episodes, and the results are as follows:

• Seq2Seq: 73.6 actions per second
• CMP: 60.1 actions per second
• MGP: 13.96 actions per second

Clarification on navigation map generation. Thank you for your insightful question. The proposed MGP method leverages both actual map-based landmark information and semantic information generated from observed images to construct the navigation map. In practical scenarios, it is reasonable to assume that UAVs are equipped with GNSS receivers, which provide access to landmark locations. However, in GNSS-free settings where landmark positions are unavailable, the success rate of navigation decreases significantly, as demonstrated in Table 5. This highlights the importance of reliable geographic data for optimal performance.

Geographical distribution of cities. Thank you for pointing this out. We apologize for missing this information in the initial submission. For this study, we used data from two urban areas: Birmingham, with a total area of 1.30 $km^2$, and Cambridge, with a total area of 3.35 $km^2$. We include this information in Section 3.

About the undo button. Thank you for your feedback regarding the absence of an “undo” button. To be transparent, the decision to omit this feature was due to the complexity of implementing it within the current system. While we did receive requests for an undo button from a few users during the data collection phase, we found that the data collection process proceeded smoothly despite its absence. Therefore, we left this feature unaddressed. We will consider implementing the undo button in future iterations of the system. Thank you for raising this point.

Annotator selection and game assignment in MTurk. For the MTurk annotation process, we specifically selected annotators from countries where English is the primary language (e.g., the USA, UK, Australia, etc.). To maintain high-quality annotations, we randomly selected annotators with significant experience on MTurk and a strong performance history, as evidenced by their high ratings on previous tasks. For each ``game,’’ corresponding to one description, exactly one person was assigned as the demonstrator.

Training Splits in Figure 4a. We did not use the Easy-Medium-Hard split during the training phase, so we did not include this categorization in the ``Train’’ row of Figure 4a. These splits are only used during evaluation to analyze the model’s performance across different difficulty levels.

Rephrasing wording.

I don't quite agree that the aerial VLN task is in general more challenging than ground VLN, necessarily as stated on Line 170

Thank you for the suggestion. We rephrased the wording of this line.

Category-level performance.

I think the paper would be strengthened by explicitly showing results at category level

Thank you for the suggestion. We added the category-level performance in Appendix D.

Real-world scenarios of aerial VLN. For clarification, we assume that there are significant demands for using VLNs in outdoor environments.

We consider that aerial VLN is essential for last-mile drone navigation in both everyday and disaster situations, especially when GPS/GNSS-based destination locations are unavailable. In disaster cases, VLN is effective for search and rescue under Visual Flight Rules (VFR) to locate individuals in remote areas who are unable to communicate their precise locations due to telecommunications failures (e.g., their phone batteries have run out, or there are blackouts at mobile cellular stations, both of which have been frequent in past disasters). In daily cases, aerial VLN is also effective for drivers delivering products when they have limited information about the destination locations. This is plausible when precise postal addresses are hidden for privacy reasons or unreliable due to outdated maps. We believe this is realistic, considering that even basic geolocation information is not always available; for example, postal addresses are not available in countries such as Dubai, and rural maps are seldom updated due to vast landscapes in continental countries. Therefore, as of the existing aerial VLN studies, we consider that the experimental settings of aerial VLN become more realistic when the precise destination location is either unavailable or unreliable, and people need to compensate for this with textual information.

Additionally, UAV operation currently demands specialized operating skills, even though many UAV applications (according to [3]) exist such as:

• Intelligent inspection of oil and gas facilities with drone solutions (https://enterprise.dji.com/oil-and-gas)
• Public safety operation by drones (https://enterprise.dji.com/public-safety)
• Land surveying, urban planning, and natural resource management by drones (https://enterprise.dji.com/surveying)

The aerial VLN research will enable UAV control for these applications through natural language instructions, without advanced manual operation. We believe that both existing and our aerial VLN datasets will serve as a benchmark for the development of such technology, contributing to the democratization of UAV usage across various fields and for diverse individuals.

Furthermore, I am confused about the experiments in disaster scenarios. The additional experiments mentioned in the appendix, which assume the failure of GPS systems during major natural disasters, further raise concerns about the practical application of this task. Allow us to clarify any misunderstanding. In the appendix, we do not claim the failure of GPS systems during major natural disasters. Rather, we conducted experiments under two distinct and challenging scenarios designed to test the robustness of our aerial VLN models: (1) environments with unreliable GNSS signals and (2) disaster situations. These two conditions are separate and address different aspects of model evaluation. Therefore, we respectfully believe it is not necessary to provide evidence and model stability for the specific scenario presented by the reviewer.

About flag for ethics review. We understand and appreciate the importance of ethical considerations in research. Regarding the use of the SensatUrban dataset in our study, we would like to clarify that, to the best of our knowledge, the dataset does not contain any information that could lead to the identification of individuals. Furthermore, the geographic data included in the dataset does not exceed the level of detail publicly available through platforms like Google Earth. We would also like to emphasize that SensatUrban is not our proposed dataset.

We hope this clarification addresses any concerns related to ethical aspects, and we are happy to provide further details if necessary.

Thank you for your suggestions. We are pleased to address the concerns raised in the review.

Analysis of the differences between human and shortest-path trajectories. We would like to provide further clarification regarding the differences between human demonstration (HD) trajectories and shortest-path (SP) trajectories, as well as the reasons behind the superior performance of HD trajectories.

In Aerial VLN, the exploration space is vast, making it crucial to narrow down the search area. To address this, our approach mimics the way humans leverage geographic information (landmarks) to reduce the exploration range. As illustrated in Figure 1 of our paper, human demonstrations rely on the landmarks mentioned in the description (e.g., Sidney Street) to navigate toward the landmark’s vicinity. Once near the landmark, humans focus their search on the area around it to find the target object. This human strategy enables efficient navigation by focusing efforts around landmarks rather than random exploration.

To validate this concept, we analyzed the trajectory data collected in the CityNav dataset, which includes geographic information. For each trajectory, we extracted the landmark name from the associated description using an LLM and computed whether the agent passed over the landmark polygon for both SP and HD trajectories. The results showed that agents passed over landmarks 36.3% of the time for HD trajectories, compared to 24.6% for SP trajectories. Then, we also analyzed whether agents passed within a certain radius of the landmark center. We observed that HD trajectories demonstrated a significantly higher proportion compared to SP, with 35.5% of HD trajectories passing within 20 meters of a landmark, compared to 24.0% for SP. Similarly, within 40 meters, 62.5% of HD trajectories passed near a landmark, compared to 51.9% for SP.

Additionally, we calculated the number of actions performed within 50m of a landmark polygon, revealing that HD trajectories averaged 95.4 actions per trajectory compared to 59.8 actions for SP trajectories. These findings highlight that HD trajectories engage in more focused and thorough exploration around landmarks, which likely contributes to their superior performance.

We also note that comparing HD and SP trajectories is also conducted in the VLN research [1]. As demonstrated in prior studies, SP trajectories lack the richness of HD trajectories, as SP typically optimizes for minimal distance and does not consider complex navigation strategies (in our case leveraging landmarks to refine search areas.)

In summary, the superior performance of HD trajectories stems from their ability to utilize landmarks to narrow down the search area and conduct more focused exploration. These qualities are demonstrated in the quantitative differences between the performance of MGP + HD and MGP + SP trajectories, as shown in Table 2. Based on the feedback, we also include this discussion in Appendix B.

[1] Ramrakhya et al., Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. CVPR, pp. 5173–5183, 2022.

Academic contribution. Firstly, from an academic perspective, the Aerial VLN task is already an established research domain, with related works published in ACL and ICCV [2][3]. Our study builds on this foundation and makes the following novel contributions:

1. A scalable trajectory data collection method through the web.
2. A dataset containing four times more language and trajectory pairs on real cities than previous studies
3. Demonstration of the effectiveness of map-based methods for Aerial VLN using human-generated trajectories

These contributions clearly advance the state of the art and provide substantial academic value.

Secondly, we emphasize the significant difference in exploration space between Indoor VLN and Aerial VLN. Indoor VLN operates within confined areas with walls, making exploration strategies such as wall-following or Frontier-based exploration [4] to approach the goal. In contrast, Aerial VLN involves vast, open exploration spaces without boundaries, where an agent may fly far off course in the worst-case scenario. This highlights the critical importance of narrowing the exploration range. In our study, we successfully achieve this by leveraging map information, which enables agents to avoid such failures and focus their search on more relevant areas.

We hope this response addresses your concerns and clarifies the academic contributions and practical significance of our work.

[2] Fan et al., Aerial vision-and-dialog navigation. ACL Findings, pp. 3043–3061, 2023.

[3] Liu et al., AerialVLN: Vision-and-language navigation for uavs. ICCV, pp. 15384–15394, 2023

[4] Yamauchi, A Frontier-Based Approach for Autonomous Exploration, CIRA, pp. 146-151, 1997

We appreciate the reviewer’s positive feedback and valuable suggestions, which have helped enhance our study. Below, we respond to the concerns raised in the review.

The majority of annotated actions are forward and down. Thank you for raising this question. The predominance of forward and down actions in the annotated data is due to the drone’s need to approach the destination. The initial altitude of the aerial agent is set between 100 and 150 meters above ground, while the average height of the 3D data is 35.96 meters. This design ensures that the agent starts at a higher vantage point to observe its surroundings and gradually lowers its altitude as it moves closer to the target, as shown in Appendix B. This natural progression toward the destination explains the higher frequency of forward and down actions.

When the drone is at a high altitude, does the annotated data make it difficult to detect small objects like vehicles? Regarding the visibility of objects like vehicles at higher altitudes, we confirm that the dataset is designed to ensure that such objects remain clearly visible. At the operational altitude of approximately 150 meters, objects such as buildings and cars which are often target objects of this dataset, are sufficiently distinguishable (see Figure 10.) In fact, even when the target objects were cars, annotators successfully identified and annotated trajectories leading to them. This demonstrates that the agent’s starting altitude does not hinder the detection of the mentioned objects.

Were collisions with environmental objects considered during data collection and model evaluation, particularly in the shortest path annotations? During the MTurk trajectory data collection process, we instructed workers to avoid collisions with objects by maintaining a relatively high altitude and approaching the target at the end. We believe that this scenario reflects real-world drone operation, where human pilots typically maintain a safe altitude to minimize collision risks (avg. height of the 3D data: 35.96 meters, avg. altitude of the aerial agent: 104.06 meters.) As a result, our experiments did not involve any interaction with stationary objects in either the training or testing paths. We hope you will find the added altitude statistics helpful (see Supplemental B).

​Performance difference between SP and HD settings. Thank you for this insightful question. The performance difference between Seq2Seq and CMA models under the SP and HD settings can be attributed to the complexity of the navigation paths of HD and the models’ inability to effectively restrict their search space without map representations.

In the HD setting, paths are more complex, which often leads the trained models to prioritize exploration. Since Seq2Seq and CMA models do not utilize map-based representations, they struggle to confine their search area effectively. As a result, even if the models encounter an object that is likely the target, they are more likely to disregard it and continue exploring unrelated areas. This behavior negatively impacts navigation efficiency (NE) and success path length (SPL), but it does not drastically affect the object success rate (OSR), as the models still encounter the correct object during exploration. In contrast, a model trained under the SP setting is more likely to move toward a target object if it seems likely to be the target, improving success rate (SR).

Small NE differences across difficulty levels. The differences in Navigation Error (NE) across difficulty levels in Table 3 can be explained by the varying characteristics of the models and their reliance on map representations.

First of all, there is a significant NE difference between the Easy, Medium, and Hard levels for the Random baseline. This is because random exploration becomes increasingly challenging as the distance to the target grows, leading to larger navigation errors for more difficult levels.

For Seq2Seq and CMA, which do not utilize map-based representations and are trained on HD settings, the models tend to explore the scene uniformly regardless of the starting point or target location. This uniform exploration results in NE values that are relatively consistent across difficulty levels.

For MGP, the model effectively leverages map-based representations to explicitly record and restrict its exploration to areas with a high likelihood of containing the target. This targeted exploration strategy allows MGP to perform consistently across difficulty levels, as it is less influenced by the distance between the starting point and the destination.

Impact of navigation map priors on model comparisons. Following the suggestion, we conducted additional experiments incorporating the navigation map into the two baseline models: Seq2Seq and CMA. The following table shows the results on the CityNav val_unseen set.

The results confirmed the effectiveness of our proposed method compared with refined baselines, all of which utilized the navigation map.

We chose Seq2Seq and CMA for this study because they are widely used and the latter is a state-of-the-art method for the existing aerial VLN work [1]. The primary focus of this paper is the proposal of a new dataset, and we believe that MGP serves as a new baseline for this dataset. Based on the suggestion, we add these results in Appendix D.

[1] Liu et al., AerialVLN: Vision-and-language navigation for uavs. ICCV, pp. 15384–15394, 2023.

Our method involves flying over the buildings at a fixed altitude rather than weaving through them. When the target object is identified, the aerial agent descends while using depth images to avoid collisions with the ground. The shortest-path trajectories are created to fit this way of moving. To prevent any confusion, we have updated the description of the shortest path. Thank you for helping us with a more accurate description.

We thank the reviewer for the positive feedback and constructive comments to improve our study. Below, we address the concerns raised in the review.

About the lack of dynamic elements and agent-object interactions. Thank you for emphasizing the importance of this topic. We agree with the importance of dynamic elements and agent-object interactions and thus we have already discussed it in Limitations and future work of Section 5. We did not consider dynamic elements or interactions with stationary objects due to the nature of the 3D city scans used in this work. But, we believe that the insights and baselines obtained in this study are expected to be valuable even when such elements are incorporated.

The generation of the dataset seems very engineering and laborious, where the instructions and trajectories are obtained by MTurk. A key contribution of our work lies in addressing the limitations of previous studies and significantly improving trajectory data collection. While trajectory data currently requires manual collection, prior research was constrained to local machines, limiting the scale of data that could be gathered. In contrast, we developed a novel interface leveraging MTurk, which allows for scalable and efficient data collection on a much larger scale. This approach enabled us to collect four times the amount of trajectory data compared to prior work. This substantial increase in data volume represents a major advancement, directly addressing the bottleneck faced by previous works.

Actually, SensatUrban just contains the images from some view of the field and sparse point clouds, while not images from each view as NeRF and Gaussian Splatting. We assume that the SensatUrban dataset contains 3D point cloud data and no image data from views. Consequently, methods like NeRF or 3D Gaussian Splatting, which rely on image-based inputs, cannot be directly applied to this dataset. Additionally, the SensatUrban dataset contains dense, high-resolution point clouds rather than sparse ones. As stated in the original SensatUrban paper [1], ``all points are subsampled to 2.5 cm, which is denser than most LiDAR datasets, such as DALES (Varney et al., 2020).’’ This high density ensures a detailed and high-resolution representation. In fact, during the trajectory collection process, annotators could exactly identify target objects such as cars and buildings in our simulation environments.

[1] Hu et al., Sensaturban: Learning semantics from urban-scale photogrammetric point clouds. International Journal of Computer Vision (IJCV), 130:316–343, 2022.

About the novelty of this work. The primary contribution of our study is the introduction of a novel aerial VLN dataset, addressing the limitations of existing datasets, such as scale and lack of geographic information, as outlined in Table 1. As a suitable baseline to demonstrate the utility of the dataset, we proposed a map-based aerial VLN method. By using this method, we clearly provided new insights by demonstrating the effectiveness of incorporating map information for aerial VLN, which has not been explored in prior research. These insights not only validate the utility of our dataset but also highlight the importance of geographic information in overcoming the challenges of large-scale, unbounded navigation environments.

We also acknowledge the reviewer’s suggestion regarding the importance of visibility for distant objects. This is indeed an interesting direction for future research, and we incorporated a discussion on this point in Section 5.