CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation

We thank Reviewer pTZW for their constructive feedback and appreciation, particularly for recognizing CymbaDiff's novelty and robustness, the clarity of our model, dataset, and experiments, and the significance of our sketch-based approach for 3D scene synthesis in applications like autonomous driving and urban simulation. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:

Real-world and cross-domain tests are needed to assess generalizability and scalability to diverse scenes.

We have demonstrated the effectiveness of our approach on SketchSem3D dataset, which is curated using the widely adopted SemanticKITTI and KITTI-360 datasets.

To further address the concern, we now also validate the efficacy of CymbaDiff by evaluating our model trained on Sketch-based SemanticKITTI on the challenging real-world nuScenes dataset. CymbaDiff achieves a 3D FID of 183.37 and a 3D MMD of 0.09, outperforming the widely used UrbanDiff model, which achieves a 3D FID of 291.4 and a 3D MMD of 0.11 (lower is better). This ascertains the effectiveness of our approach.

The paper lacks analysis of failure cases; discussing challenging scenes would offer a more balanced evaluation.

Observing the results of CymbaDiff on the proposed SketchSem3D dataset, it is apparent that it demonstrates strong performance, effectively handling challenges such as semantic misalignment caused by noisy pseudo-labels, e.g., due to confusion between vegetation and buildings. Nevertheless, it does occasionally fail to accurately reconstruct small or occluded objects that are underrepresented in the training data or sparsely encoded in the sketch and PSA inputs. Although CymbaDiff mitigates this issue to some extent through the use of the Cross-Scale Contextual Block and Cylinder Mamba Block, which capture multi-scale contextual information, its performance could be further enhanced by increasing the representation of small objects in the dataset. We will add more discussions in the camera-ready (in supplementary if it is not permitted by space in the main paper).

The paper lacks user studies or human evaluations.

While our study primarily relies on quantitative metrics such as 3D FID and MMD to evaluate the proposed approach, these widely adopted measures in generative tasks, ranging from 2D image synthesis to 3D object generation, offer objective, reproducible, and scalable assessments of fidelity and diversity.

We acknowledge that human evaluation can offer complementary insights into semantic coherence, structural plausibility, and perceptual realism. However, its subjectivity, high resource demands, limited scalability, and variability in user interpretation make it less practical for consistent and large-scale benchmarking.

In future work, we will consider incorporating structured human evaluation alongside metric-based evaluation to qualitatively assess the perceptual quality of generated 3D scenes, complementing our current quantitative analysis for a more comprehensive evaluation.

The paper uses CLIP and SAM for pseudo-labeling. How does the pipeline handle inaccurate or ambiguous labels from them?

To reduce noise in Pseudo-labeled Satellite Image Annotations (PSA), we restrict CLIP predictions to semantic classes present in the 3D ground truth, ensuring alignment with the SketchSem3D taxonomy and preventing irrelevant or out-of-distribution labels.

Second, we perform manual spot-checks by comparing predicted class distributions with BEV projections of the 3D ground truth to identify and remove mislabeled samples. Third, incorporating sketches provides structural cues that help the model compensate for PSA imperfections during training.

Beyond these safeguards, the architecture enhances robustness through a pre-trained VAE that encodes inputs into a stable latent space with accurate spatial priors. The latent diffusion process then iteratively denoises within this space, making it resilient to moderate input noise. Combined with sketch-based guidance, this design ensures semantically coherent 3D scene generation under weak or noisy supervision.

Provide more quantitative comparisons with state-of-the-art methods, and include broader results on generation quality, efficiency, and generalization to strengthen the paper.

We selected 3D Latent Diffusion and 3D-DiT as baselines due to the limited availability of open-source 3D scene generation models. Although not originally designed for semantic voxel generation, both models have shown strong performance in image and 3D object synthesis, and their general architectures can be effectively adapted to the SketchSem3D dataset to establish strong baselines.

We acknowledge the relevance of methods like UrbanDiff and SemCity; however, the lack of public implementations prevents direct comparison. We address this by adding BlockFusion (required by other reviewers) as a baseline on SketchSem3D. However, since it was originally designed as a 2D tri-plane denoiser, adapting it to our 3D latent space led to unstable training and suboptimal performance, with the loss failing to converge. Despite these challenges, we still cite these works and will include a discussion of their contributions in the camera-ready.

Moreover, we provide quantitative comparisons in the following table across methods in terms of parameter count and runtime performance. These results demonstrate that CymbaDiff achieves a favorable trade-off between model efficiency and computational cost, offering competitive performance with significantly fewer parameters compared to these two generative models.

Table: The efficiency comparison. M: Million, and S: seconds

These additions will be included in the camera-ready to improve clarity and support a more comprehensive evaluation of CymbaDiff. We will also provide qualitative comparisons on the SketchSem3D dataset to visually demonstrate the model's effectiveness.

What are CymbaDiff's key innovations compared to prior methods? and Clarify how Cylinder Mamba enhances diffusion and differs from models like UrbanDiff or SemCity.

The key innovations of CymbaDiff lie in its use of sketches and pseudo-labeled satellite annotations (PSA) as input modalities for 3D semantic voxel scene generation, and the incorporation of Cylinder Mamba blocks into the denoising diffusion process, facilitating contextual modeling across both cylindrical $(\theta, r, z)$ and Cartesian $(x, y, z)$ coordinate systems.

The former enables generation from abstract and lightweight inputs, which supports more accessible supervision and is particularly suitable for large-scale or remote scenarios where high-fidelity sensor data is unavailable or impractical. The latter leverages the semantic alignment and radial awareness of cylindrical encoding with the computational efficiency and regularity of Cartesian grids. This dual representation, together with a triple-axis decomposition strategy, enables effective modeling of radial, vertical, and angular dependencies in complex outdoor scenes.

However, methods such as UrbanDiff and SemCity rely on dense sensory inputs, including RGB images or BEV representations. UrbanDiff operates solely within the Cartesian coordinate system, while SemCity projects 3D data onto three orthogonal 2D planes, leading to the loss of important volumetric structural information. The experimental results demonstrate that CymbaDiff achieves superior performance, particularly in terms of 3D Fréchet Inception Distance (FID), highlighting its effectiveness in capturing complex spatial dependencies. We will add more discussions in the camera-ready (in supplementary if it is not permitted by space in the main paper).

Thank you for your reply. This has solved most of my problems.

1. Real-world and cross-domain tests are needed to assess generalizability and scalability to diverse scenes. Could your make a certain comparison with SemCity or citydreamer? This paper has made the code available. Meanwhile, could your method generate scenes similar to those of CityDreamer?
2. In Table 4 of the paper, the "car" category for SOTA is not bolded. I would like to know what the main reasons are for these categories not reaching SOTA, and if there could be a specific analysis provided.

Looking forward to your reply.

We are grateful to the reviewer for acknowledging that many of their previous concerns are addressed. Please see our responses to the remaining queries below.

CymbaDiff’s applicability in real-world simulation or planning.

We appreciate the reviewer’s concern. However, we emphasize that CymbaDiff is the first-of-its-kind method for outdoor 3D voxel scene generation with SK+PSA conditioning, setting a strong baseline. It provides structurally and semantically meaningful representations (rather than photorealistic appearances) that are helpful and fundamental to planning, perception, and decision-making. While hyperrealism has its own importance, it is not the objective of CymbaDiff. CymbaDiff and CityDreamer pursue different objectives. CityDreamer prioritizes visual realism to generate 2D photorealistic scenes with only 6 categories. CymbaDiff generates 3D voxel scenes with 18 categories. Different objectives/number of categories make comparison of the two methods not straightforward.

CymbaDiff generates 3D voxel scenes with 18 semantic categories more accurately than any existing generative method. We agree that for simulation, photorealism is important, but for making autonomous decisions directly on the data, semantic and structural accuracy has its own value. Hence, CityDreamer and CymbaDiff are both import in their own way.

Questions the FID > 700 results of SemCity and CityDreamer, and requests proof they were fairly trained on SketchSem3D.

Regarding the high FID scores of SemCity and CityDreamer, we confirm that both models were trained under the exact same conditions as our CymbaDiff. We used the same pre-trained VAE, the same voxel resolution, the same SK+PSA conditioning, the same loss functions, and the same hyperparameters. Despite using multiple random seeds and repeated training runs, both models consistently achieved high FID scores. While images or codes cannot be shared during this phase, we will provide detailed training logs, loss curves, and code in the final submission to ensure transparency and reproducibility.

The reason why Semcity and CityDreamer do not perform well in our experiments is their denoisers (provided in their official GitHub repositories). The denoiser in SemCity only has convolutional and linear layers, whereas that in CityDreamer relies on a simple stacking of Transformer layers. Although Transformers can model long-range dependencies, such simplified designs may be suboptimal for large-scale 3D voxel scene generation, where sparse and irregular data demand specialized mechanisms to effectively capture both local geometry and relevant global context. We will explicitly note these details in the camera-ready when comparing our method with SemCity and CityDreamer.

Points out that nearly half of the categories, including key ones like “truck” and “road,” do not reach SOTA, indicating limited category-level generalization.

In Table 4, our method achieves the best overall mIoU and the best performance in 10 out of the 18 evaluated categories, clearly demonstrating its strong overall effectiveness. Moreover, our approach provides the first evidence that sketches (SK) and pseudo-labeled satellite image annotations (PSA) are not only viable but also effective alternatives, particularly in domains like remote sensing, where RGB/stereo data is often unavailable. Enabling the flexible use of SK and PSA introduces unique challenges absent in RGB or stereo-based methods.

Note that some methods are exploiting stereo (which contains 3D information), while our technique uses SK and PSA (2D inputs) outperforming those methods on most categories. We will further clarify these distinctions in the final version to avoid any misinterpretation of the method’s scope and contributions.

We thank Reviewer Ws1Y for their constructive feedback and appreciation, especially for recognizing the effectiveness of Cylinder Mamba blocks in enhancing spatial coherence and the contribution of SketchSem3D as the first large-scale sketch-based 3D scene generation benchmark. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:

Provides data on parameters, memory use, or runtime to support reducing memory cost.

Thank you for the valuable suggestion. We provide the following table to compare the memory cost of our method with 3D DiT and 3D Latent Diffusion. To ensure a fair comparison, we replaced our latent-space denoiser with their respective architectures, and the results reflect only the denoiser parameters. Minor code modifications were applied to both baselines to enable consistency with our method's runtime environment. The table demonstrates that our architecture (CymbaDiff) offers a clear memory gain. Over DiT, this gain is about 8.5x, which can be critical in practical deployment.

Table: The efficiency comparison. M: Million, and S: seconds

The benefit of cylindrical coordinates is unclear for datasets like SemanticKITTI.

Although datasets such as SemanticKITTI do not exhibit strict rotational symmetry, they often display approximate bilateral symmetry along the driving path, for example, with sidewalks and buildings typically appearing on both sides of the road. To capture this structure, we adopt cylindrical coordinates $(\theta, r, z)$, which align well with the egocentric and radial layout of street-view environments.

Cylindrical coordinates offer a vehicle-centric, geometrically coherent representation, enabling semantic tokenization along angular and radial dimensions and facilitating long-range context modeling with Mamba. In contrast, Cartesian coordinates provide uniform, axis-aligned voxels suitable for dense computation and 3D convolutions but lack egocentric awareness and may underrepresent radial continuity.

To exploit the strengths of both representations, our framework integrates cylindrical and Cartesian coordinate systems. This hybrid design combines the semantic coherence of cylindrical encoding with the computational efficiency of Cartesian grids, enhancing overall 3D scene modeling. We will further clarify this in the camera-ready.

Clarify how Triple Mamba (Cartesian) and C-Mamba (cylindrical) features are aligned before fusion.

Triple Mamba and C-Mamba operate in distinct coordinate systems, Cartesian and cylindrical, respectively, which necessitates careful alignment to enable effective feature fusion. To achieve this, we apply an inverse coordinate transformation to convert the cylindrical features produced by C-Mamba back into Cartesian voxel space. Specifically, for each token in the $(\theta, r, z)$ grid, we compute its corresponding $(x,y,z)$ location using the predefined voxel resolution and center alignment parameters.

This transformation allows the C-Mamba outputs to be reprojected into a dense 3D tensor that is spatially aligned with the Cartesian grid used by Triple Mamba. Once both feature representations reside in the same $(x,y,z)$ coordinate space, we apply element-wise fusion, ensuring spatial correspondence is preserved during integration. This approach enables the coherent fusion of features across modalities, enhancing the model's ability to capture complementary information from both coordinate systems. We will release the code to enhance the clarity for readers.

The paper claims a different evaluation metric from SemCity, yet reports the same FID value as in the SemCity paper. Please clarify.

Apologies for the misunderstanding. The evaluation metrics used in our study differ from those adopted in SemCity. SemCity relies on the 2D Fréchet Inception Distance (FID) computed from rendered images. Our approach employs more comprehensive 3D evaluation metrics, specifically 3D FID and Maximum Mean Discrepancy (MMD), which directly measure geometric fidelity and semantic consistency within the voxel space. Although this distinction is noted in Lines 279–282 of the manuscript, we will further emphasize it in Table 2 to avoid potential misinterpretation and to ensure a fair comparison between the methods.

The 3D convolution decomposition (Sec. 4.3) should trade quality for efficiency, yet FID improves with DDCB. Please explain.

To balance reconstruction quality and efficiency in decomposed 3D convolutions, the Dilated Decomposed Convolution Block (DDCB) stacks multiple Dimensional Decomposition Residual (DDR) units with varying dilation rates. This design captures multi-scale spatial features while maintaining low computational cost.

Serially structuring decomposed convolutions compensates for the reduced joint receptive field in dimension-wise decomposition, preserving representational capacity and enhancing directional context modeling. This contributes to the improved FID performance observed in our experiments.

Enhance Fig. 4 with structural consistency analysis (e.g., seam artifacts, voxel coherence) as in NeRF-Synthetic for better transparency.

While fine-grained analysis of structural consistency could offer insights into model behavior, the Cylinder Mamba block operates in 3D latent space, where features represent abstract embeddings rather than explicit geometry, making spatial interpretation challenging. Nonetheless, we acknowledge its value and will consider such interpretability strategies in future work.

Clearer design details.

Thank you for your valuable suggestions. We will improve Sec. 4 to provide clearer descriptions of each core module in the final version, and also augment the appendix with illustrative figures to visually support the architectural details. Furthermore, we plan to release the source code alongside the paper, which will ensure covering all details.

We sincerely thank Reviewer fZuf for their comments and appreciation of our work, particularly highlighting the well-designed dataset and model as the main strengths, which enable effective 3D scene generation from simple 2D sketch inputs. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:

Section 4 lacks details on the Scene Structure Estimation and Latent Mapping networks.

In Section 4.1, the Scene Structure Estimation Network (SSEN) incorporates multi-scale feature extraction modules along with several Dimensional Decomposition Residual (DDR) blocks with various dilations to effectively capture spatial structures and rich contextual information across varying receptive fields.

Specifically, multi-scale feature extraction modules capture hierarchical contextual information by aggregating features across multiple receptive fields. It employs parallel branches of $3 \times 3 \times 3$ convolutions to replace $5 \times 5 \times 5$ and $7 \times 7 \times 7$ convolutions, which are progressively stacked and merged at multiple levels. Residual connections are integrated throughout the block to facilitate feature reuse and gradient flow.

The Dimensional Decomposition Residual (DDR) blocks begin with a $1 \times 1 \times 1$ convolution for channel adjustment, followed by a series of three directional convolutions: $1 \times 1 \times 3$, $1 \times 3 \times 1$, and $3 \times 1 \times 1$, which capture contextual dependencies along the depth, height, and width dimensions, respectively. The outputs of these directional convolutions are then fused through an additional $1 \times 1 \times 1$ convolution. A residual connection is incorporated by adding the input feature map to the fused output.

The Latent Mapping Network in Section 4.2 adopts the same architecture as the Encoder of the Variational Autoencoder (VAE). The VAE encoder consists of two down-sampling blocks, each comprising four consecutive convolutional layers. Every pair of convolutional layers is followed by a Batch Normalization layer and a ReLU activation function. Following these operations, a downsampling convolutional layer is applied, which is also followed by Batch Normalization and ReLU. We will add this architectural information in the camera-ready version and provide a corresponding figure to clearly depict the structure of the SSEN and Latent Mapping Network. We would also like to emphasize that we intend to make our code public to ensure a comprehensive understanding of our technique.

Section 4.4 introduces x(t), y(t), and h(t), but does not link them to z_TMB and f_TMB; a brief explanation and connection are needed.

Thank you for the constructive comment. Advanced state space models (SSMs), such as the Structured State Space Sequence Model (S4) and Mamba, represent a class of systems that map a one-dimensional continuous input sequence $x(t) \in \mathbb{R}$ to an output $y(t) \in \mathbb{R}$ via a hidden state $h(t) \in \mathbb{R}^{N}$. These models are typically formulated using linear ordinary differential equations (ODEs), defined as:

$h^{\prime}(t) = Ah(t) + Bx(t), \quad y(t) = Ch(t),$

where $A\in \mathbb{R}^{N \times N}$ and$B\in \mathbb{R}^{N \times 1}$, $C\in \mathbb{R}^{1 \times N}$ denote the state matrix, input matrix, and output matrix, respectively. Since deriving the analytical solution for $h(t)$ is often intractable and real-world data is typically discrete, we discretize the system as follows:

$h(t) = \overline{A}h(t-1) + \overline{B} x(t), \quad y(t) = Ch(t),$

where $\overline{A}= exp\left (\triangle A \right )$ and $\overline{B}=\left ( \triangle A \right )^{-1} \left ( exp\left ( \triangle A \right ) - I \right ) \cdot \triangle B$ are the discretized state parameters and $\triangle$ is the discretization step size. In the context of our Triple Mamba layer, the input $x(t)$ corresponds to a residual-enhanced representation:

$z_{TMB}(t) =LN(f_{TMB}(t)) + f_{TMB}(t),$

where $f_{TMB}(t)$ is the raw input features prior to residual layer normalization $LN$. The temporal dynamics of the Mamba input are thus governed by:

$h(t) = \overline{A}h(t-1) + \overline{B} z_{TMB}(t), \quad y(t) = Ch(t),$

To capture multi-directional dependencies, the Triple Mamba layer applies three separate Mamba modules, each operating on the same input $z_{TMB}(t)$ but with distinct ordering strategies: forward, backward, and random inter-slice directions. The output of the $i^{th}$ Triple Mamba layer is computed as:

$\psi_{i}(z_{TMB}(t)) = \psi_{i}^{f}(z_{TMB}(t)) + \psi_{i}^{b}(z_{TMB}(t)) + \psi_{i}^{u}(z_{TMB}(t)),$

where $\psi_{i}^{f}$, $\psi_{i}^{b}$, $\psi_{i}^{u}$ represent the outputs from the forward, backward, and random inter-slice directions, respectively. This formulation can be succinctly expressed as:

$Triplemamba_{i}(z_{TMB}(t)) = Mamba_{i}^{f}(z_{TMB}(t)) + Mamba_{i}^{b}(z_{TMB}(t)) + Mamba_{i}^{u}(z_{TMB}(t)).$

We will include these details in the camera-ready version (supplementary material if the space does not permit it in the main paper).

Lines 286–299 lack a clear reference to the corresponding table.

Thank you for the comments. We will add the corresponding table number in the text.

The sketch and pseudolabels in Fig. 4 often appear to come from different, unrelated scenes.

The difference is due to a temporal discrepancy between the data sources: the 3D ground truth annotations in SemanticKITTI were collected around 2013, whereas the satellite imagery used for Pseudo-labeled Satellite Image Annotations (PSA) was captured approximately in 2025. This temporal gap may lead to minor inconsistencies in spatial structures. Also, the PSA are generated through a weakly supervised pipeline that aligns satellite imagery with text embeddings, which enables scalable semantic supervision but can introduce semantic or geometric inaccuracies. In contrast, the sketches are derived directly from the 3D ground truth, ensuring temporal consistency with the underlying scenes. As a result, sketches serve as a reliable spatial reference that helps mitigate the domain gap introduced by the satellite imagery. We will explicitly note this in the camera-ready.

Consider citing recent methods like DynamicCity and Towards Generating Realistic 3D Semantic Training Data for completeness.

We will cite and discuss DynamicCity [1] and 3DiSS [2] in the final version. These works are indeed relevant to our study. We sincerely thank the reviewer for mentioning them.

[1] DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes

[2] Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving.

Are 3D GT classes used to filter incorrect CLIP pseudolabels during generation?

During pseudo-label generation, we do not directly filter the CLIP outputs using the 3D ground truth semantic classes. Instead, we utilize the set of semantic labels present in the 3D ground truth scenes as candidate classes for predicting labels in the satellite imagery. After the initial predictions are made, we conduct a manual validation step, where the predicted class distributions are compared against the bird's-eye view (BEV) projections of the 3D ground truth. This post-hoc verification ensures that the retained pseudo-labels are semantically consistent with the underlying scene context, thereby enhancing the accuracy and reliability of the annotation process.

Are the textual descriptions in Fig. 1 included in the dataset?

Yes, we include textual descriptions used in Fig.1 in the dataset. It will be released to support further research.

The sketches are too precise to be human-drawn.

We agree that human-drawn sketches are an important direction. Thanks for acknowledging our SketchSem3D dataset as a first step towards generating a 3D scene from human-drawn sketches. Based on our benchmark, future work will incorporate freehand or user-generated sketches to better approximate real-world input conditions and further enhance the model's generalization and practical usability.

The authors have addressed and clarified my concerns in their rebuttal. Still, the mathematical notations and explanations in their rebuttal should be added to the main paper to enhance the clarity of the methodology section. I suggest adding it to the main paper, and in case space is needed, the authors can consider moving Figure 3 (the architecture) to the supplement. I see the mathematical explanation as more relevant to be in the main paper rather than the architecture figure. Therefore, I have raised my rating since the rebuttal clarified my concerns.

We sincerely thank Reviewer 4gZF for their comments and appreciation of our work, especially noting our dataset's size, flexibility, and strong baseline performance. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:

Supplementary architectural figures for CSCB and DDR blocks are needed. Details on the Cylinder Mamba, including the triple Mamba mechanism and its ordering process, are also insufficient.

Thank you for the constructive comment. As shown in Figure 3, the Cylinder Mamba block integrates the Triple Mamba and C-Mamba layers to enhance representational capacity by combining Cartesian and cylindrical spatial representations. The Triple Mamba layer models feature dependencies along forward, backward, and inter-slice directions in Cartesian space. Although architecturally identical to the C-Mamba layer, the detailed structure of C-Mamba is explicitly illustrated in Figure 3 for clarity. The key distinction lies in the input domain: Triple Mamba operates on Cartesian grids, whereas C-Mamba processes data in cylindrical coordinates.

In the C-Mamba layer, inputs are transformed into cylindrical coordinates and sorted by angular, radial, and vertical indices $(\theta, r, z)$ to align with the egocentric geometry of street-view scenes, enabling directional context modeling. After processing, features are mapped back to Cartesian space (sorted according to $(x, y, z)$) and fused with those from the Triple Mamba layer, allowing the model to jointly exploit radial and axis-aligned spatial cues.

We will further improve clarity by adding architectural figures and more details on the triple Mamba mechanism and its ordering process in the final version. Please note, our source code will also be released to facilitate comprehensive understanding of our method.

The necessity of incorporating sketches remains unclear; PSA can be predicted from sketches. Why not directly utilize the BEV projections of semantic voxels?

Thank you for the comment. Sketches offer clear and structured spatial layouts that effectively complement pseudo-labeled satellite annotations (PSA). Since PSA is generated by aligning recent satellite imagery with text prompts, it is susceptible to noise and spatial inconsistencies, particularly due to the temporal discrepancy between contemporary satellite data (e.g., from 2025) and the older LiDAR-based ground truth in SemanticKITTI (2013). Incorporating sketches mitigates these issues by providing reliable spatial priors, and jointly using both inputs enables more flexible and robust conditioning, thereby enhancing spatial accuracy.

We conducted an ablation study by training CymbaDiff using only PSA inputs, which required code adjustments and excluded sketch cues. As shown in the table below, the resulting performance drop underscores the essential role of sketch guidance in enhancing 3D scene generation quality and fidelity.

Predicting PSA from sketches remains an underexplored direction, which we plan to investigate in future work using the annotations provided in our benchmark. Notably, we deliberately refrain from using BEV projections of semantic voxels, as dense 3D annotations are often unavailable in practical settings. In contrast, satellite imagery serves as a scalable and accessible source of weak supervision, making our approach more broadly applicable under realistic data constraints.

The method could be tested on BEV-based NuScenes to compare with UrbanDiff and assess task difficulty, and provide BlockFusion results on SketchSem3D.

We selected 3D Latent Diffusion and 3D-DiT as baselines due to the limited availability of open-source 3D scene generation models. Although not originally designed for semantic voxel generation, both models have shown strong performance in image and 3D object synthesis, and their general architectures can be effectively adapted to the SketchSem3D dataset to establish strong baselines.

The pre-processed BEV-conditioned NuScenes dataset used in UrbanDiff is not publicly available, so we cannot train the proposed CymbaDiff on this dataset. To address the reviewer's concern, we directly applied our model, trained solely on the Sketch-based SemanticKITTI dataset, to the Sketch-based nuScenes validation dataset without any fine-tuning. Evaluation on the nine overlapping semantic classes shared between the two datasets shows that our CymbaDiff achieves a 3D FID of 183.37 and a 3D MMD of 0.09, outperforming the popular UrbanDiff model on the BEV-conditioned NuScenes dataset that achieves 3D FID and 3D MMD scores of 291.4 and 0.11 (lower is better). In addition, the proposed SketchSem3D dataset offers a more practical alternative to BEV-conditioned NuScenes. We will make this dataset publicly available to facilitate future research.

BlockFusion serves as a strong baseline for 3D scene generation. However, since it was originally designed as a 2D tri-plane denoiser, adapting it to our 3D latent space led to unstable training and suboptimal performance, with the loss failing to converge.

Table 4 is less informative, as SK+PSA offers an easier setting than raw-image-based methods.

Our work explores a new research direction, and currently, there are no directly comparable methods using the same input modalities. For this reason, we compare CymbaDiff with existing methods that use monocular or stereo RGB inputs, which are common in semantic scene completion. These comparisons highlight the differences, gaps, and challenges between input modalities, as our approach is based on sketches (SK) and pseudo-labeled satellite image annotations (PSA) rather than RGB images. While stereo images can provide geometric depth cues for 3D reconstruction, our results demonstrate that SK and PSA inputs offer a flexible and effective alternative, particularly in scenarios where conventional RGB inputs are unavailable or impractical, such as remote sensing. In the final version, we will provide further explanation to clarify these distinctions and prevent possible misinterpretation.

We appreciate the reviewer mentioning their willingness to raise the rating, and hope that the above responses help in that regard. We look forward to discuss further in the discussion phase, if needed.