TRIM: Scalable 3D Gaussian Diffusion Inference with Temporal and Spatial Trimming

We sincerely appreciate the reviewer's thoughtful and detailed feedback. Your constructive suggestions are invaluable to improving our work, and we are encouraged by your recognition of our contributions. Below, we provide detailed responses to your concerns. As some of the weaknesses and questions are closely related, we merge their responses and address them together for coherence.

Weaknesses:

W1: The paper frequently describes the method as training-free, but this characterization is somewhat misleading since a latent selector is trained to guide trajectory pruning.

Our claim of "training-free" refers specifically to that we do not require any re-training or fine-tuning on diffusion models, meaning that weights of the off-the-shelf DiffSplat model remain completely unchanged in our framework. We acknowledge that our method introduces a lightweight latent selector, which is trained to adapt to the DiffSplat model and guide trajectory pruning. The selector is the only trainable component in our framework, and training it on our synthetic dataset takes less than 10 minutes, incurring only minimal training computational cost, as shown in training details in Appendix A. We will revise the paper to clarify this statement more rigorously in the next version.

W2, W3.1, Q2: The explanation of the latent selector's training process is occasionally unclear. For instance, lines 115–120 and Equation (1) describe a training approach that the authors later state is not used, without sufficient explanation for its inclusion.

Thanks for your attention to our latent selector training process. We introduce the distillation objective in Equation (1) to set up the training goal, which is to align the selector's output with the Decoder-Renderer-Evaluator's score, which is the ground truth of the synthetic dataset. However, in the experiments of directly distiling the Decoder-Renderer-Evaluator joint model into the selector, we notice that the selector training is not stable and hard to converge. We attribute this training challenge to two reasons: 1) The variance of score distribution is very small, which makes it difficult for the selector to learn the slight score difference among multiple trajectories. 2) The lightweight selector is too weak to distill the complex knowledge in the Decoder-Renderer-Evaluator.

Considering the above challenges on the direct distillation, we further propose a pairwise comparison training strategy to perform an indirect distillation, as shown in Line 143-158. The selector is changed to predict the relative performance among trajectories, which is more robust and stable to train. Compared to the direct distillation, the pairwise comparison training strategy is under the same offline distillation setting with the same training data and a similar training objective.

We acknowledge that the explanation of the latent selector's training process is not clear in the submission. We appreciate your suggestion and will improve the method description to make the transition from direct model distillation to indirect pairwise comparison distillation smoother in the revised version.

W3.2, Q2: Equation (2) is employed for training the latent selector, but key details—such as how the timestep ( t ) is sampled—are missing. Clarifying whether the selector is trained across all timesteps or at a fixed one would improve both the interpretability and reproducibility of the method.

We thank the reviewer for pointing out that Equation (2) does not mention the timestep. We clarify that Equation (2) reflects the final training objective used in our experiments, and the corresponding timestep setting is earlier described in Lines 144–145. Specifically, we use $\mathbf{f}= \mathrm{CNN}(\mathbf{z}_t)$, the CNN embeddings of the intermediate latent features at timestep $t$, as inputs to the MLP, and train the selector with the loss in Equation (2).

We train the selector at a fixed timestep, and our ablation study in Figure 8 investigates the impact of different $t$ values. The performance gain plateaus after $t = 50\% T$, so we adopt this as the default timestep in our main experiments.

We appreciate the reviewer's suggestion of training the selector across all timesteps. We agree that incorporating timestep as an additional input for the selector could turn to a more elegant and robust architecture without manually tuning the hyperparameter $t$. The timestep-aware selector might enable more flexible temporal trimming during inference. We plan to explore this advanced selector model in the future work.

W4: Pruning all but one trajectory during inference may limit output diversity, as it commits early to a single generation path. Addressing how this approach impacts diversity, particularly in multi-modal generation tasks, would be valuable.

Thank you for your comment regarding output diversity. It is generally accepted that the diversity of diffusion models is primarily driven by the initial noise, i.e., different random seeds result in diverse generations. However, the quality of these diverse outputs can vary significantly.

Our method is motivated by the observation that among multiple sampled trajectories, some yield substantially better results than others. As shown in the left subfigure of Figure 3, instead of choosing a random path, we employ a pairwise tournament selection strategy to identify and retain the trajectory with the most promising potential. This does not reduce the inherent diversity from random noise, but selects a higher-quality sample from the diverse pool. Thus, our approach balances maintaining diversity and improving output quality.

Moreover, we can invert the usage of the selector to identify the "loser" trajectories, those predicted to have lower performance potential, offering a way to analyze the failure cases and lower-bound quality of the model's outputs. This dual capability of the well-trained selector enables more effective investigation of output diversity from both best-case and worst-case perspectives. We plan to explore this direction in future work.

W5, Q1: Some baseline results in Table 2 reported differ noticeably from those in the original papers, without any explanation or discussion.

Thank you for pointing out the discrepancies. There are two main reasons for the differences between our reported baseline results and those in the original papers: 1）Data selection inconsistency: DiffSplat reports results on a randomly selected subset of 300 images from the GSO dataset, but does not disclose the its selection list. In our experiments, we randomly sample 300 GSO objects, matching the number used in DiffSplat, but it is not an identical subset to DiffSplat's. 2） Viewpoint rendering mismatch: Both DiffSplat and InstantMesh do not provide multi-view rendering camera parameters of the GSO evaluation dataset. Therefore, we follow the official GSO dataset's rendering protocol as an alternative, using four orthogonal views (front, back, left, right) rendered at a fixed elevation angle of 0°. This setting offers a standard and reproducible test set that is beneficial for future research. However, these four orthogonal views have lower overlap compared to six- or eight-view settings used in other works, making the reconstruction task more challenging. This explains why our reproduced baseline results are mismatched and lower than those reported in the original papers.

We have included detailed settings about the GSO benchmark in Appendix A to ensure reproducibility. The results in Table 2 are conducted under these consistent settings for a fair comparison. We will provide additional explanation to discuss the differences in the GSO dataset settings in the revised version.

W6: The ablation study focuses primarily on variations in the latent selector architecture. It would be beneficial to also analyze the impact of individual trimming components—such as the Temporal Trimming Scheme, Progressive Mask Expansion Scheduler, Token Merging and Padding, and Post-denoising Correction—to better understand their contributions to overall performance.

Thank you for the valuable suggestion to clarify the contribution of individual components. We provide an ablation study analyzing the contributions of Trajectory Reduction and Instance Masking in Figure 1 and Figure 2 of Appendix B. For the internal components of Instance Masking, we offer detailed discussion on their design and function in Section 3.3. Specifically, Instance Mask Detection and Token Merging & Padding are core components for spatial trimming and are indispensable to the overall pipeline. And we conducted additional ablations on the remaining two components: Progressive Mask Expansion Scheduler and Post-denoising Correction. The results in the table below show that removing these two components hurts the model output quality, demonstrating that both components contribute positively to the model's performance.

Specifically, Progressive Mask Expansion Scheduler introduces a smooth and gradual transition from unmasked to masked regions, facilitating more stable and effective denoising. Post-denoising Correction removes Gaussian primitives located in transparent background regions, which are typically associated with [BG] tokens that are not optimally denoised, thereby reducing visual artifacts in the final rendered images. We will add visual comparisons of rendered outputs with and without Post-denoising Correction in the revised version to better illustrate its effect.

We sincerely appreciate the time and effort you dedicated to reviewing our work and your valuable comments. We provide the following clarifications and responses to address your concerns in detail.

Weaknesses:

W1： The latent selector seems to rely heavily on the data synthesis and metrics score. What's the off-the-shelf image metric evaluator? Does it lead to overfitting and having to re-train the selector when changing the dataset?

Thank you for the question. We provide the implementation details of selector training in Appendix A. Specifically, we construct a synthetic dataset using ChatGPT-generated prompts and 64 trajectories per prompt, and use CLIP score as the evaluation metric to supervise the selector. The well-trained selector is integrated into the TRIM pipeline to perform trajectory reduction (temporal trimming). It is noteworthy that this synthetic dataset is entirely independent of any downstream evaluation datasets, and no test sample is used during selector training, avoiding any test data leakage. In our text-to-3D generation experiments, the selector is trained using ChatGPT-generated prompts describing the single object, and it generalizes effectively to single-object with surrounding and multi-object generation tasks, as shown in Table 1. This indicates that the selector does not require re-training when the evaluation dataset changes.

Regarding concerns about overfitting to the CLIP score, we empirically observe that the trajectories selected by our model also gain improvements under another ImageReward metric. This suggests that the selector, although trained with a single metric, generalizes well and provides improvements among other metrics.

W2. The authors mention one of the significant improvements is the inference speed due to the Instance Masking (IM). Can the authors include the inference time comparison with other works?

Thank you for the suggestion. We have added inference time comparisons with two more baselines, LGM and InstantMesh, in the table below. Note that both LGM and InstantMesh adopt a different 3D generation pipeline that involves multi-view image diffusion followed by image-to-3D reconstruction. This pipeline uses a simpler U-Net architecture for image generation, resulting in faster runtimes than our Gaussian Splatting-based 3D diffusion model.

Specifically, LGM uses 30 inference steps, while InstantMesh uses 75 steps, leading to InstantMesh having the longest runtime. In contrast, our TRIM uses the default 28 denoising steps but still achieves a significantly faster runtime than DiffSplat and InstantMesh, while being only slightly slower than LGM. We also provide the major performance PSNR results in the table below, which shows that TRIM achieves the best performance among all baselines and LGM performs not well on PSNR even though it has the fastest runtime. We will include the discussion of runtime differences between various 3D generation pipelines and baselines in the revised version.

Questions:

Q1. How many trajectories do you have for TRIM? Does the Inference Step Scaling mean increasing denoising steps in inference?

We discuss the number of trajectories used in TRIM in Section 4.4. The ablation experiments with 2,4,8 trajectories is shown in Figure 7, and the performance gain increases with more sampled trajectories. Based on this, we adopt 8 trajectories in our main experiments in Table 1 and Table 2.

Regarding inference step scaling, as analyzed in Section 3.2, denoising N trajectories over T steps leads to a total of N×T denoising steps. Our temporal trimming scheme reduces denoising steps to N×T−(N−1)×t, where t is the early-stop trimming timestep. Thus, we save (N−1)×t steps overall.

In implementation, we generate all N trajectories in parallel, so the total runtime is primarily determined by the longest (selected) trajectory, plus a small overhead from the pairwise tournament selection used in temporal trimming. As reported in Table 1 of Appendix A, this overhead is minimal and has little impact on overall runtime.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We are also pleased to see your recognition of the contributions of our work. We provide our clarifications in response to your concerns as follows. Due to the similarity between your provided Weakness 2 and Question 1, we address them together in the response to the weaknesses.

Weaknesses:

W1: The masking strategy relies on a corner-based attention heuristic, which assumes that objects are centered in the scene. This design limits generalizability to more diverse or complex layouts where foreground and background regions are not spatially separable.

Thank you for pointing out the potential limitation of our masking strategy based on the corner-based attention heuristic in complex layouts where foreground and background regions are not spatially separable.

We would like to clarify that the center-object assumption is a widely adopted setting in the 3D object generation field. Specifically, 1) 3D datasets such as Objaverse and GSO (Google Scanned Objects) construct their coordinate systems with the target object centered, and both rendered and captured views are oriented towards the object center. 2) Many recent models, including LGM, InstantMesh, and DiffSplat, also train with object-centered renderings as input. Our approach follows these widely used settings for dataset preparation and model training to ensure comparability and relevance to standard benchmarks.

We clarify that our method can also be applied in scenarios where foreground and background regions are not spatially separable. Notably, the "background" identified by our method refers to invalid transparent regions, rather than semantically meaningful backgrounds. Specifically, our approach detects regions where the Gaussian Splatting primitive's opacity attribute is zero or close to zero, which are effectively marked as transparent and uninformative. These regions are regarded as redundant primitives and then removed during the Transformer-based denoising process.

We agree that generating complex layouts (e.g., multiple objects or objects with background context) is more challenging. To evaluate this, we have included evaluation results on such scenarios in Table 1 (T$^3$Bench's Single Object w/ Sur and Multiple Objects). Although performance drops slightly in these complex settings compared to single-object cases, our method still outperforms baselines, showing its effectiveness under complex layout settings. We appreciate your suggestion and will explore more robust strategies for complex layout generation in future work, especially for multi-object and background-aware scenes.

W2, Q1: Although the selector is described as lightweight, the paper does not provide a quantitative analysis of its memory or computational overhead.

Thank you for the suggestion. We provide the computational cost of the selector in Table 1 of the appendix, as shown in the table below. Leveraging our latent selector introduces around 0.29 GB of additional GPU memory usage and 0.1 seconds of latency during inference. Overall, the marginal cost of the temporal selector is outweighed by the substantial efficiency improvements introduced by spatial trimming, making the combined trimming strategy both effective and resource-efficient.

W3: The method, as explained in the paper, is not as effective on image-to-3D to 3D scenarios.

Thank you for the observation. Compared to text-to-3D generation, the image-to-3D task inherently exhibits lower output diversity, which reduces the gains brought by our temporal trimming. We attribute this to two main reasons: 1) Stronger supervision from input reference image: The reference image in image-to-3D provides a strong conditioning signal, making the sampled trajectories across different random seeds more consistent. This diminishes the benefit of pruning less useful trajectories since every trajectory may be similar. 2) Evaluation metric bias: The evaluation is based on accuracy compared to ground-truth views, which inherently penalizes plausible outputs that deviate from the ground-truth. Thus, the diversity encouraged by temporal trimming is under-reflected in the current reconstruction metric, since many generated details look plausible but are not considered correct under strict ground-truth comparisons.

As a potential solution, we will incorporate diversity-aware metrics or human preference studies, like CLIP and aesthetic scorer, in future benchmarks to better evaluate the image-to-3D generation task.

We sincerely appreciate the reviewer's insightful and valuable comments. Below are our clarifications regarding your concerns. Due to the high similarity between your feedback in the weaknesses and questions, we merge some questions into the reply to the weaknesses and address them together.

Weaknesses:

W1, Q1. The writing lacks clarity in some areas.

• The definitions of Trajectory Scaling and Inference Step Scaling should be provided earlier in the paper

Thank you for the insightful suggestion. We agree that introducing Trajectory Scaling and Inference Step Scaling earlier in the paper would improve clarity and strengthen the motivation for our method. Our approach is driven by two key observations: 1）Increasing inference steps in 3D generation leads to diminishing gains, and even degradation when the number of steps becomes excessive. 2）Among multiple sampled trajectories, certain ones yield significantly better results than others.

These findings demonstrate that the trajectory is a more effective axis to scale than the inference step, motivating us to propose Trajectory Scaling and the corresponding Trajectory Reduction methods. Therefore, moving the definition and analysis of these two kinds of scaling strategies to Sections 2 and 3, which are more closely associated with the core motivation, would help clarify the intuition behind our method and enhance the paper's clarity. We appreciate this helpful suggestion and will reorganize this motivation in the revised version.

W1, Q2. The writing lacks clarity in some areas.

• Figure 6 is difficult to understand.

Thank you for your careful examination of our experimental results. We acknowledge that Figure 6 may be unclear in its current form. Figure 6 compares two different scaling strategies, Trajectory Scaling and Inference Step Scaling, each varying along different axes. Specifically, all experiments start from a baseline of 10 denoising steps and 1 trajectory (the leftmost bars in Figure 6):

• Trajectory Scaling: We fix the number of denoising steps at 10 and increase the number of trajectories to {2, 4, 8}.
• Inference Step Scaling: We fix the number of trajectories at 1 and increase the number of denoising steps to {20, 40, 80}.

According to the above scaling settings, the total inference budgets (steps) are #steps $\times$ #trajectories, {10, 20, 40, 80}, shown as the x-axis of Figure 6. This comparison reveals the two observations discussed in our response to [W1, Q1]: 1) increased inference steps do not necessarily lead to better performance, and 2) sampling more trajectories offers more potential for selecting better outputs. These two observations show that the trajectory is a more effective axis to scale rather than the inference step. In Figure 6, the trajectory scaling strategy gradually outperforms the inference step scaling strategy and has more potential for better performance as the inference budgets increase.

We acknowledge that the current legend and caption in Figure 6 are a bit confusing. In the revision, we will 1) clarify the axis labels and legends to match the original scaling strategies (TRIM $\rightarrow$ Trajectory Scaling, DiffSplat $\rightarrow$ Inference Step Scaling). 2) Rephrase the figure description in the paragraph for easier understanding. We appreciate your feedback.

W2: The experimental metrics are not sufficiently comprehensive. The paper uses CLIP similarity, CLIP R-Precision, ImageReward, PSNR, SSIM, and LPIPS to demonstrate the alignment between 3D objects and text/images, as well as human aesthetic preferences. However, these metrics are not exhaustive. A more comprehensive evaluation could be conducted by referring to the metrics used in 3DGen-Bench or by conducting a user study, which would provide a more thorough and nuanced assessment.

We appreciate the reviewer's suggestion of using 3DGen-Bench and conducting a user study. Here, we use the CLIP-based 3DGen-Score model in 3DGen-Bench as an additional metric to enhance our evaluation. The 3DGen-Score model requires multi-view RGB images, normal maps, and a text prompt as input. The evaluation output contains five criteria: Geometry Plausibility, Geometry Details, Texture Quality, Geometry-Texture Coherence, and Prompt-Asset Alignment, each with different value scopes. We provide results on the T$^3$-Single text-to-3D benchmark in the table below, which is evaluated by the 3DGen-Score metric. The 3DGen-Score results show that TRIM achieves better performance than the baseline DiffSplat on all five criteria.

For a user study, we have included detailed visualization comparisons and utilized an aesthetic scorer to approximate human preference. As suggested, we plan to conduct a formal user study and include the results in future work.

W3, Q5: The paper claims to be compatible with various Transformer-based 3D diffusion models without requiring retraining, but it lacks sufficient comparisons to show how the performance changes when TRIM is applied to different backbones.

We thank the reviewer for pointing out the importance of evaluating different backbones. Our proposed trajectory reduction strategy is designed to be applicable to most diffusion-based architectures, while the instance mask strategy specifically leverages the token structure of Transformer-based backbones. In our main experiments, we choose Stable Diffusion 3.5, the latest diffusion model in the Stable Diffusion series, as the backbone. To further demonstrate generalization, we also apply TRIM to PixArt-Sigma with a different diffusion backbone, and present the results on the T$^3$-Single text-to-3D benchmark in the table below. These results show that TRIM improves the performance of DiffSplat with the backbone of PixArt-Sigma, demonstrating that TRIM consistently improves both the generation quality and efficiency across various Transformer-based backbones.

W4, Q6: There is a lack of adequate qualitative comparisons; the paper only provides a direct comparison with DiffSplat, which is not enough for a thorough evaluation.

We appreciate the reviewer's suggestion of providing more qualitative comparisons with other state-of-the-art methods. In the experiment section, we choose GVGEN, DIRECT-3D, LGM, InstantMesh, and DiffSplat as the baseline methods and show quantitative comparison results in Table 1 and Table 2. Since DiffSplat is the state-of-the-art method beyond the other three approaches, we present the qualitative comparisons with DiffSplat to highlight the improvement of TRIM. While the rebuttal does not allow us to insert the images, we will provide qualitative comparisons with more baselines, such as LGM and InstantMesh, in the updated submission.

Questions:

Q3: More Details: Could you provide more detailed information on the experimental parameters, such as the optimal settings and the network architecture parameters?

Thank you for your inquiry for more implementation details. We present our main experimental parameters in Appendix A. The ablation settings with custom parameter values are noted alongside each ablation experiment for clarity.

For the 3D generation inference, we follow DiffSplat's default settings to ensure a fair comparison and reproducibility. Our 3D generation pipeline is built on the Stable-Diffusion-3.5-Medium backbone and uses the Flow Matching Euler ODE solver with 28 denoising steps and 8 sampled trajectories in the main experiments. The classifier-free guidance scale is set to 7. Input images are center-cropped and resized to 256×256 resolution.

For the selector training, we adopt the Conv1-FC2 architecture for the selector, which achieves the best performance in the ablation study, shown as Table 3. We train the selector using the AdamW optimizer with a learning rate of 0.001, weight decay of 0.01, and a cosine decay scheduler. We set the training batch size to 64 and the epochs to 20.

In addition, we detail the pairwise trajectory data synthesis procedure used to generate samples for training the selector in Appendix A. We appreciate your question and will provide more detailed implementation details in the revised version.