Controllable Satellite-to-Street-View Synthesis with Precise Pose Alignment and Zero-Shot Environmental Control

We thank Reviewer-KCsS for the valuable comments and suggestions. We have thoroughly revised the paper, improved the writing quality, clarified unclear descriptions, and fixed typos. Please find our detailed response to each question below.

Q1. Lack of clarity on the significance of achieving geometric consistency and more accurate street views from satellite images

To address this, we provide a demonstration of how the images generated by our method can enhance downstream tasks, such as ground-to-satellite camera localization, through data augmentation. This analysis is detailed in Appendix (Sec.A.2).

Specifically, we use the recent work by Shi et al. [ECCV2024] as the baseline for ground-to-satellite localization and retrain the model using a combination of the original training images and our generated images as augmented data. The performance comparison is presented in Tab.5 (also included below).

The results show that incorporating our generated images leads to improved localization performance, highlighting the practical value of achieving geometric consistency and generating more accurate street views from satellite images. This improvement underscores the significance of our approach in real-world applications.

Q2. Derivative process of Eq.(4) and Eq.(5) and the interrelationship between model design and formulas.

As suggested, we present the derivation process in Appendix A.4 and thoroughly revise Sec.3 and Sec.4 to make the correspondence between the formula and model design clearer. Specifically, the proposed geometric cross-attention in Sec.4.1 is incorporated in the framework in a classifier-free manner, corresponding to the formulas in Sec.3.1. Sec.3.2 describes how to further enhance pose alignment and control environmental factors in a classifier-guidance manner, which motivates our model design of the iterative pose alignment in Sec.4.2 and text-guided zero-shot environmental control in Sec.4.3.

Q4. Explanation of the sampling operation in Line-251, Line-273, and Line-277.

The "sample" operation is a process of extracting or selecting certain pixel data from the original image. In PyTorch implementation, we use F.grid_sample for this purpose. The sample operation requires two inputs: the source image and the correspondence of pixel coordinates between the source and target images. In the context of this paper, the source image is represented on the left side of the symbol $\otimes$, while the correspondence is on the right side.

After revision, the sampling operations are now in Line-247, Line-274, and Line-277, respectively. In Line-247, the sample operation involves sampling satellite image features (V) onto corresponding eight height hypotheses, as shown in Fig.12. In Line-274 and Line-277, based on the homography matrix H at that point, the images undergo a homographic transformation, where the correspondence between the source and target images is determined by H. Subsequently, the source image is sampled into the new image, as described in Fig.10. In summary, the symbol $\otimes$ in these three equations all represents the sample operation, with the distinction lying in the correspondence between the source and target images.

Q5. why choose $\epsilon$ to control the generative process with respect to the text prompt?

Our primary goal is to leverage gradient information from the CLIP model to ensure that generated samples align closely with the provided textual descriptions. By manipulating $\epsilon$, as outlined in Eq.13, we indirectly adjust $z_{t-1}$ during the diffusion process. This approach effectively integrates the semantic guidance from text prompts while maintaining compatibility with the diffusion framework.

Q6. Explanation of “potent satellite image prompts significantly undermine the influence of text prompts during training”.

Thank you for pointing out the ambiguity in our phrasing. Here is a more detailed explanation:

In Tab.4, we evaluated multi-condition generation using Classifier-Free Guidance (CFG) within LDM and ControlNet frameworks. These models used satellite images and environmental descriptions (generated by LLAVA) as dual conditioning inputs. However, we found that text-based control was largely ineffective, and the generated results did not sufficiently reflect the textual descriptions.

The underlying issue stems from the dominance of the satellite image condition during training. Each satellite image corresponds uniquely to a specific ground image, providing rich geometric and texture information. Conversely, environmental descriptions generated by LLAVA often corresponded to multiple possible ground images and lacked precise geometric details, as shown in Fig.13, Sec.A.7 of the Appendix. This discrepancy caused the model to assign significantly more weight to the satellite image condition while deprioritizing the environmental description branch. Over time, this imbalance led to the degradation of the model's ability to utilize text prompts effectively.

In summary, while CFG was employed for multi-condition generation, the stronger and more precise information from satellite images overshadowed the less-detailed text descriptions, undermining the influence of text prompts during training.

Q3. Details of Cross-View Conditioning Mechanism

Apologize for the confusion we made in the initial submission. The mechanism indeed leverages a query to extract the features from the satellite image. We have modified Sec.4.1 to make it clearer, and a brief introduction is below.

Specifically, we assume a set of candidate heights for each pixel in the ground image, and use the height information to find corresponding pixels, $P(u_g, v_g, h_i + \Delta h_i)$ from the satellite image, collectting corresponding features by $V \otimes P(u_g, v_g, h_i + \Delta h_i)$, where $V$ denote the satellite feature maps, $\otimes$ denotes a sampling operation that extracts features from $V$ according to the projected satellite image coordinates.

The collected satellite feature maps at different heights assumptions are then aggregated via an attention matrix $A_i$ (Eq.(5) in the paper), which has a shape of $HW \times N$, where $H$ and $W$ denotes the height and width of the ground features, and $N$ is the number of different heights utilized. The attention weights $A_i$ are normalized via a softmax function across the $N$ height planes, determining the confidence associated with each plane.

(1) Visualization of the "Cross-View Conditioning Mechanism".

In Fig.12, Sec.A.6.1 of the Appendix, we provide visualizations of the collected features from satellite images under varying height assumptions, the corresponding weights in the attention matrix $A_i$, and the progressively refined ground feature maps achieved using the GCA mechanism.

In the middle of Fig.12, the top row illustrates the ground feature maps collected from satellite features at different candidate heights $h_i + \Delta h_i$, while the bottom row shows the corresponding attention maps $\\{A_i\\}_{i=1}^N$. During the initial GCA iteration (step 1), the projection plane accurately samples features based on the height assumption, with the attention being evenly distributed across various locations, forming the initial solution. By the second GCA iteration (step 2), the offset $\Delta h_i$ induces positional shifts in the projection plane, as highlighted in the red box. These shifts allow the projection to capture more detailed ground-level features, such as pathways, while also refining representations of other scene elements.

As the GCA iterations progress, the attention maps evolve to focus on salient regions. For planes where $h<0$, attention focuses primarily on the ground, while for planes where $h>0$, it shifts towards higher elements such as trees. This adaptive refinement underscores the ability of GCA to progressively enhance feature representation.

(2) Advantages compared to a simple cross-attention mechanism.

The proposed GCA module offers the following advantages over a naive cross-attention mechanism.

• Flexibility in Image Generation. By leveraging relative pose between satellite and ground images, the proposed GCA enables the generation of ground images at arbitrary locations on the satellite map. In contrast, the simple cross-attention mechanism cannot handle the relative pose information and requires additional modules to process the relative pose information (e.g., zero123[CVPR2023]). Our method is more flexible and can handle various relative pose differences.

• Avoid redundancy of information. The proposed GCA restricts attention to regions that are geometrically likely to correspond—each ground-view pixel attends only to satellite image regions along its camera ray. This focused attention minimizes noise from irrelevant regions, unlike simple cross-attention, which indiscriminately considers the entire image. This targeted approach improves generation quality, as evidenced by the results in Tab.3, where our method consistently outperforms simple cross-attention.

• Improved computational efficiency and reduced GPU memory usage. GCA achieves significant reductions in computational complexity by employing sparse sampling. For satellite image features of size $S \times S$ and ground image features of size $H \times W$, the complexity of naive cross-attention is $O(S \times S \times H \times W)$. In contrast, our algorithm samples N planes (N=8), with the complexity of sampling being $O(N \times H \times W)$, computing horizontal and vertical coordinate offsets at $O(2N \times H \times W)$, and attention calculation at $O(N \times H \times W)$. Overall, our complexity is significantly reduced to $O(4 \times N \times H \times W)$ compared to naive cross-attention. In practical experiments, for inference on one example, simple cross-attention consumes 250MB of GPU memory, whereas using GCA reduces this to only 184MB.

Dear Reviewer KCsS,

We would like to thank you very much for your insightful review, and we hope that our response addresses your previous concerns regarding this paper.

Regarding the issue you are concerned about: in Appendix (Sec. A.2), we have supplemented data augmentation experiments to emphasize the motivation of our paper; in the Appendix (Sec.A.4), we have added formula derivations and based on this, we thoroughly revised Sec.3 and Sec.4 to make the correspondence between the formula and model design clearer; in Sec. 4.1, we have revised and explained the GCA module; in Appendix (Sec. A.6.1), we have included visual experiments of the GCA module; in Appendix (Sec. A.6.2), we have outlined the advantages of GCA; in Fig. 9, Fig. 10, and Fig. 12, we have demonstrated sample operations; in Eq. 13, we have explained the reason for adjusting $\epsilon$ in ZoEC; in Tab. 4 and Appendix (Sec. A.7), we have explained the reasons for the failure of the CFG scheme in our application scenario.

As the discussion period is coming to a close, we kindly invite you to share any additional comments or concerns about our work. We would be more than happy to address them. In the meantime, we hope you might consider revisiting your evaluation and potentially increasing your rating.

Thank you for your thoughtful feedback and consideration! We really appreciate it!

Best regards,

The Authors

How is the generated image in Q1 response comes from? What is the detail of data augmentation strategy? Furthermore, could the authors compare your method with other methods on the performance of data augmentation? Such as BEVGen and BEVControl? Could you conduct data augmentation experiments on KITTI or other autonomous driving datasets?

We thank Reviewer-vmd3 for the valuable comments and suggestions. Please find our response below.

Q1. To refine English expression and standardize academic phrases, such as "pose correction" and "pose alignment".

We have standardized the terminology to "pose alignment" and made corrections to other terms accordingly.

Q2. Image citation, caption elucidation, and Explanation of Formulas.

We have corrected all these mistakes in the main text, marked as blue. The parameter τ in Eq. 9 is a temperature hyperparameter, which has now been revised to Eq. 6.

Q3: Lack of discussion on reconstruction results related to roads and buildings in the CVUSA dataset.

We provide additional visualizations of generated results by various methods on complex scenes from the CVUSA dataset in Fig.11, Sec.A.5 of the Appendix. These complex scenes comprise a small portion of the dataset, posing significant challenges for faithful ground-view image recovery.

Among the comparison algorithms, S2S and Sat2Den struggle to capture the unique features of rural scenes, producing blurry and indistinct images. ControlNet shows noticeable geometric misalignments. In contrast, our method, leveraging the proposed Geometric Cross-Attention (GCA) mechanism and Iterative Homography Adjustment (IHA) during inference, demonstrates superior recovery of road structures, closely resembling the ground-truth images.

Buildings, which occupy only a small region in the target ground-view images, present an additional challenge as their facades are not explicitly encoded in the conditioning satellite images. Despite this, our method—guided by the geometric reasoning capabilities of GCA and IHA—recovers buildings in the appropriate regions of the ground-view images and synthesizes plausible facade appearances (as shown in the third example in Fig.11).

As a result, our generated images not only retain strong structural consistency but also exhibit a promising ability to represent buildings and other challenging scene elements.

Q4. How does ZoEC perform under different languages and textual descriptions? Can it adapt to multilingual scenarios?

Thank you for this insightful question. The proposed ZoEC framework is capable of handling various languages and textual descriptions, provided there exists a multimodal model capable of mapping these languages to the same latent space as images. In our implementation, we use a frozen CLIP model to handle the language of English and compute losses to guide the generative process. However, our framework is flexible and can accommodate other multimodal models for other languages without requiring additional training. For example, Chinese-CLIP can be integrated to support Chinese text prompts, leveraging its ability to align Chinese text with corresponding image features. We demonstrate this in Fig.14, Sec.A.8 of the Appendix, where the model successfully generates corresponding images following the Chinese language guidance.

Dear Reviewer vmd3,

We would like to thank you very much for your insightful review, and we hope that our response addresses your previous concerns regarding this paper.

Regarding the issue you are concerned about: We have standardized the use of academic terms, formulas, images, and tables; in Appendix (Sec.A.5), we have supplemented the visual analysis of complex scenes in CVUSA; and in Appendix (Sec.A.8), we have supplemented the performance of ZoEC in multilingual scenarios.

As the discussion period is coming to a close, we kindly invite you to share any additional comments or concerns about our work. We would be more than happy to address them. In the meantime, we hope you might consider revisiting your evaluation and potentially increasing your rating.

Thank you for your thoughtful feedback and consideration! We really appreciate it!

Best regards,

The Authors

Thank you for your detailed response. I will revise my rating accordingly. It is worth noting that, to the best of my knowledge, the citation for your CrossviewDiff is incorrect. I suggest updating it to the correct version.

We thank Reviewer-jsaR for the valuable comments. Please find our response below.

Q1. Amendments to the Related Works section are needed to include DreamFusion, Zero-1-to-3, IM-3D, and CAT3D.

Thank you for pointing out the important works. We have restructured our related work section and included detailed discussions of these papers, marked as blue in Lines 121-124. In short, all these works aim to generate novel view images from given camera poses. The difference is that these works are object-centric, while our work focuses on scene-level and target for challenges where target and source views have significant elevation differences.

Q2. The improvements in the results displayed in Tab.1 and Tab.2 are not significant.

Due to the minimal overlap between satellite and ground images, the appearance of generated images might significantly differ from the GT ground images, although the geometric layout might be consistent, making significant improvement difficult. To facilitate comprehensive comparisons, we further incorporate semantic consistency between generated and GT images for evaluating different methods' performance, in addition to the evaluation metrics in the initial submission. The semantic consistency measures the similarity of deep features extracted by DINO[CVPR2021] and SegmentAnything[CVPR2023]. It can be seen that although the superiority of our method on SSIM, PSNR, and RMSE is insignificant, the perceptual, semantic, and depth consistency achieved by our method is much better than the state-of-the-art approaches.

Q3. Why choose the affine transformation to describe relative pose shift?

Thank you for pointing out this important question, and we apologize that this wasn't made clear in our initial submission. When there is only a rotation difference between two cameras, the Homogrphy transform is generally applicable to describe the correspondence relationships between pixels in two images. When there are both rotation and translation differences, the Homography transform can only be used to depict pixel correspondences for scene points on the same plane. In practice, the scene depth exhibits complex variations and thus the pixel correspondences between two images are hard to be described by one single Homography. Thus, we propose to iteratively apply Homography on the noisy latent vectors during the denoising sampling process, and expect that a combination of multiple different Homographies collaborates with the denoising UNet to approximate the complex pixel correspondences caused by relative pose shifts.

We have made this clearer in the paper, marked as blue in Lines 264-267, Appendix (Sec.A.3.1), and have changed the mistakenly used description of "affine" to "Homography".

Q4. Is Clip Score enough it measure text-based generation ability?

CLIP has a certain capability to evaluate the similarity between generated images and the given environmental prompts. To obtain more comprehensive results, we supplement the experiments with evaluations using BLIP. We query the image content with the question "What season is it there?" to determine the season depicted in each image. Subsequently, we evaluate the recall rate that aligns with the environmental prompts we provide, as shown in Tab.4. The results demonstrate that our method consistently outperforms other methods across all evaluation metrics.

Dear Reviewer jsaR,

We would like to thank you very much for your insightful review, and we hope that our response addresses your previous concerns regarding this paper.

Regarding the issue you are concerned about: in related work, we have included more important works; in Tab1 and Tab2, we have conducted more comprehensive comparisons to demonstrate the advantages of the proposed algorithm; in Line 264-267 and Appendix (Sec.A.3.1), we have explained the reason for choosing the Homography transformation; and in Tab.4, we have supplemented the Blip score to more fully evaluate the text-based generation ability.

As the discussion period is coming to a close, we kindly invite you to share any additional comments or concerns about our work. We would be more than happy to address them. In the meantime, we hope you might consider revisiting your evaluation and potentially increasing your rating.

Thank you for your thoughtful feedback and consideration! We really appreciate it!

Best regards,

The Authors

We thank Reviewer-7YpL for the valuable comments. Please find our response below.

Q1. How Iterative Homography Adjustment (IHA) affects inference speed.

The Iterative Homography Adjustment (IHA) mechanism operates entirely in the latent space, where the feature map resolution is low, rather than mapping it back to image space, avoiding the need for a heavy ground-to-satellite localization network that operates on the original image resolution. As a result, the computational overhead introduced by IHA is small. We conducted evaluations with a batch size of 1, and the results are summarized in the table below.

When comparing memory usage, the baseline model without IHA (Wo.IHA) requires 20,126MB, while the inclusion of IHA (W.IHA) increases memory usage slightly to 21,022MB (896MB increase) due to the addition of a lightweight localization network. In terms of time cost, the baseline model (Wo.IHA) requires 5.406 seconds per image, while the model with IHA (W.IHA) increases this slightly to 5.513 seconds per image—an additional cost of only 0.107 seconds per image. We believe that IHA strikes a favorable balance between performance and computational efficiency, offering promising practical value.

We have added corresponding explanations in the Method Section (Line 295-299) and provide the computation comparison in Appendix (Sec.A.3.2).

Q2. Ablation study on Geometric Cross-Attention (GCA).

Thank you for this suggestion. We have incorporated the ablation study results of GCA in Tab.3 of the paper. Similar to the ablation study experiments on IHA, we measure the pose consistency between the generated images and the given pose by leveraging a cross-view localization model. The results demonstrate that both the proposed IHA and GCA improve the pose consistency of the generated images.

Q3. Performance on RMSE, PSNR, and SD.

We supply the comparisons in terms of RMSE, PSNR, and SD in the paper (Tab.1 & Tab.2). The results demonstrate that our method consistently outperforms previous state-of-the-art across all evaluation metrics on the CVUSA dataset.

For the VIGOR dataset, our approach does not surpass the GAN-based Sat2Density in pixel metrics such as RMSE and PSNR. This discrepancy arises because pixel-based metrics are more tolerant of blurry images. Our approach, based on a diffusion model, aims to recover finer details in images, often leading to larger pixel differences. Nevertheless, our method demonstrates clear advantages in structural, perceptual, semantic, and depth consistency metrics, showcasing its ability to generate high-quality results that align more closely with human perception and practical requirements.

Q4. Referencing other relevant works such as SCP-Diff [ECCV 2024].

Thank you for recommending this relevant and interesting work. We have discussed it in the related work section (Lines 115-118). The proposed Noise Prior in SCP-Diff is innovative and efficient. We would like to make further exploration along this direction in our task.

Dear Reviewer 7YpL,

We would like to thank you very much for your insightful review, and we hope that our response addresses your previous concerns regarding this paper.

Regarding the issues you are concerned about: in Line 295-299 and Appendix (Sec.A.3.2), we have supplemented the discussion on the impact of IHA on reasoning speed; in Tab.3, we have added ablation experiments of the GCA mechanism; in Tab.1 and Tab.2, we have included evaluation results of RMSE, PSNR, and SD; and in the related work section, we have discussed the SCP-Diff as you recommended.

As the discussion period is coming to a close, we kindly invite you to share any additional comments or concerns about our work. We would be more than happy to address them. In the meantime, we hope you might consider revisiting your evaluation and potentially increasing your rating.

Thank you for your thoughtful feedback and consideration! We really appreciate it!

Best regards,

The Authors