Fourier Token Merging: Understanding and Capitalizing Frequency Domain for Efficient Image Generation

Dear Reviewer,

Thank you for the insightful comments. We address the following concerns:

Comment 1: I think the amount of experiments in this work is not enough, especially more diffussion methods should be used for experiments instead of just one SD model.

Response: We applied our method to more models as follows. We experimented our Fourier Token Merging in ViT models. Results show that our method generalizes well on image classification tasks. For ImageNet val-1k dataset on the 4090 machine, with the same level of throughput (e.g. around 2000 im/s), our method increases the accuracy by 0.7 point (from 83.0% to 83.79%). With the same level of accuracy, our method achieves 1.2$\times$ speedups.

Aside from the image classfication experiment, we also include results for Diffusion Transformer (DiT) models. Specifically, we have done experiments on DiT models, tailoring FTM to their specific transformer blocks. Results are promising: even with a large merging ratio (0.75), we maintain high image quality (1.83 FID increase) while achieving over 20% latency reduction. This suggests FTM's strong generalization capabilities to diverse diffusion models, including DiT-like architectures, distinguishing our approach and highlighting its potential for current foundation models.

We will add those results into the final version.

Comment 2: Figure 1 is not clear, and it is difficult to intuitively see the execution process of the Fourier Merging Module. It would be better to have a figure to explain the merging process in detail.

Response: We will make clear the merging process through image illustration as detailed in Section 3.1.

Comment 3: In line 84, ``only the lowest $\tau d$ frequency", what is $\tau d$? It seems that there is a typo here.

Response: As indicated earlier in the paragraph, $d$ is the dimension of the token x and $\tau$ is the truncation ratio, so $\tau d$ is $\tau \times d$ which refers to the dimension we keep after truncating on the frequency domain. We will add the clarification into the text to avoid any confusions.

Comment 4: What is the connection between 3.2 and 3.1? A clear expression may be needed here.

Response: Token Merging itself is an approximation computing method and brings in error for the image generation process. Section 3.1 discusses our main idea of introducing the Fourier Merging Module into the Token Merging pipeline. A theoretical interpretation and intuition behind the Fourier Token Merging is then explained in Section 3.2 and show the potential of achieving smaller approximation error compared to the original Token Merging.

Comment 5: The writing of the Modular Integration section seemed confusing, and although there were descriptions of the advantages of the approach, I thought they were just common sight and did not help me understand how the proposed approach was effective.

Response: Our idea is to cluster (merging) tokens in the frequency domain so as to maximize its decorrelation ability. We show both theoretically that our method gives smaller approximation errors compared to the original method (Section 3.2) and empirically that our method merges tokens that have higher similarity scores (Section 3.3). Section 4 shows that Fourier-based clustering produces more structured and semantically coherent clusters. We further evaluate end-to-end image generation and find our method not only increases the image generation quality but also achieves speedup compared to the original Token Merging method.

Dear Reviewer,

Thank you for the insightful comments. We address the following concerns:

Comment 1: The current method is evaluated only on diffusion models with U-Net architecture. Applicability to up-to-date architectures like Diffusion Transformer is not demonstrated.

Response: While we primarily evaluated on Stable Diffusion, our Fourier Token Merging method w for e applicability. Existing token merging methods often struggle with DiT's specialized architecture and RoPE embeddings. We have done experiments on DiT models, tailoring FTM to their specific transformer blocks. Results are promising: even with a large merging ratio (0.75), we maintain high image quality (1.83 FID increase) while achieving over 20% latency reduction. This suggests FTM's strong generalization capabilities to diverse diffusion models, including DiT-like architectures, distinguishing our approach and highlighting its potential for current foundation models. We will add the results into the final version.

Comment 2: The method introduces more hyper-parameters and the token merging process is highly affected by this hyper-parameters which may require more tuning during the training process.

Response: Our Fourier Token Merging method uses more hyperparameters, but this provides crucial control over the efficiency-quality trade-off, letting users customize for their needs (e.g., merging ratio). Optimal values for parameters like truncation ratios will be included in the config file. Crucially, as Fourier Token Merging is an off-the-shelf, training-free method, parameter tuning is highly efficient, allowing quick experimentation and rapid optimization for applications.

Comment 3: The improvement is not significant. In Figure 5, The FID and latency improvement is not significant when compared to the baseline.

Response: Our method not only reduces latency but also increases the image quality. Figure 5 clearly illustrates that for the same level of image quality (e.g., around 41.5 FID), our FTM method achieves approximately 1.5$\times$ speedup compared to ToMeSD. Conversely, if we target the same inference speed (e.g., around 1.38 seconds per image), our method yields a 0.5 lower FID score, indicating superior image quality.

Dear Reviewer,

Thank you for the insightful comments. We address the following concerns:

Comment 1: Figure 1 needs to be refined. The clarity and aesthetics of this figure could be improved.

Response: We will enhance the figure.

Comment 2: The paper does not provide any visual comparison of generated images between the proposed Fourier Token Merging and the baseline method. Since the primary goal is to preserve image quality while reducing computational cost, qualitative results are crucial to verify that the visual fidelity and semantics are indeed retained. Including such examples would strengthen the empirical claims.

Response: We'll address this by adding a dedicated section with visual comparisons of images generated using our method at various merging ratios, along with those from the original Stable Diffusion and the ToMeSD adapted model. It will clearly illustrate the quality retention of our method, even with higher merging ratios.

Comment 3: he Fourier Token Merging indicates a plug-and-play solution, but the experiment is insufficient. It only conducts experiments based on stable diffusion model. In my opinion, if a method clarifies that the method is a plug-and-play module, it needs to verify the proposed module into different backbones for a wider evaluation of the effectiveness.

Response: We applied our method to more models as follows. We experimented our Fourier Token Merging in ViT models. Results show that our method generalizes well on image classification tasks. For ImageNet val-1k dataset on the 4090 machine, with the same level of throughput (e.g. around 2000 im/s), our method increases the accuracy by 0.7 point (from 83.0% to 83.79%). With the same level of accuracy, our method achieves 1.2$\times$ speedups.

Aside from the image classfication experiment, we also include results for Diffusion Transformer (DiT) models. Specifically, we have done experiments on DiT models, tailoring FTM to their specific transformer blocks. Results are promising: even with a large merging ratio (0.75), we maintain high image quality (1.83 FID increase) while achieving over 20% latency reduction. This suggests FTM's strong generalization capabilities to diverse diffusion models, including DiT-like architectures, distinguishing our approach and highlighting its potential for current foundation models.

We will add those results into the final version.

Comment 4: Will the code for this paper be open-sourced? It is recommended to do so, as it would greatly benefit and advance the community.

Response: Yes. We will open source the code.

Dear Reviewer,

Thank you for the insightful comments. We address the following concerns:

Comment 1: The method is evaluated on Stable Diffusion, which is relatively outdated. Based on prior experience, token merging techniques often do not transfer well to more recent diffusion transformer models such as DiT, PixArt, FLUX, or modern video generation models. These newer models tend to suffer significant accuracy drops under small merging ratios, casting doubt on the applicability of this line of work to current foundation models. Demonstrating that FTM generalizes effectively to DiTs would constitute a meaningful contribution; otherwise, the method appears to offer only incremental gains over existing token merging approaches.

Response: While we primarily evaluated on Stable Diffusion, our Fourier Token Merging method aims for broader applicability. Existing token merging methods often struggle with DiT's specialized architecture and RoPE embeddings. We have done experiments on DiT models, tailoring FTM to their specific transformer blocks. Results are promising: even with a large merging ratio (0.75), we maintain high image quality (1.83 FID increase) while achieving over 20% latency reduction. This suggests FTM's strong generalization capabilities to diverse diffusion models, including DiT-like architectures, distinguishing our approach and highlighting its potential for current foundation models. We will add the results into the final version.

Comment 2: The use of frequency-domain operations, although efficient, may not be well-optimized or readily supported on all deployment hardware, potentially limiting practical benefits.

Response: Thanks for highlighting the hardware optimization aspect of frequency-domain operations. Our method relies on Fast Fourier Transform (FFT), which is highly optimized on GPUs: Modern GPUs excel at parallel FFTs. Libraries like cuFFT (NVIDIA) provide massive speedups, making our operations highly efficient and well-supported on GPU-accelerated platforms. Even constrained MCUs with DSP extensions or modern cores often have hardware accelerators or optimized DSP libraries (e.g., CMSIS-DSP) for efficient FFTs.

Comment 3: The merging strategy is static and hand-crafted, lacking the adaptability of learnable or data-driven approaches that can tailor token reduction more effectively to specific tasks or inputs.

Response: Currently we support adaptive token merging by scheduling merging rates for different layers and diffusion steps. We agree that incorporating more adaptability could be a future step to take to further develop this line of work. One way is to allow dynamic merging ratios based on features of the input image (e.g., complexity, content density) or intermediate activations, similar to how some adaptive token pruning methods operate.

Comment 4: Missing SOTA baselines such as token pruning or token downsampling works.

Response: Thanks for suggesting comparisons with SOTA token pruning and downsampling methods. %We've included the baselines. We included the results of those baselines as follows.

Token pruning: AT-EDM offers 1.13$\times$ speedup over ToMe but loses quality. Our method, at a similar 1.1$\times$ speedup, actually decreases FID (40.75 vs. 41.31), showing better quality-speed trade-off.

Token Downsampling: This method helps only at very high merging ratios (e.g., 0.89), leading to longer latency at lower ratios (e.g., 0.75). In contrast, our method consistently achieves speedups across various ratios, e.g., 1.3$\times$ at 41.8 FID.

The comparisons suggest that our method offers competitive or superior performance, especially in maintaining quality while providing consistent speedups. We'll include a comprehensive comparison in the revised manuscript.

The reviewer thanks the authors for the rebuttal and new results.

The DiT experiments and SOTA comparisons are helpful. Which DiT model was used, and how does the method compare to standard token merging? It’s surprising that the model performs well under 75% sparsity—was sparsity applied to all layers or specific ones? What is the end-to-end latency/memory improvement? The current results lack coverage of key settings, and contradict prior findings that token merging often struggles with diffusion transformers, suggesting the need for more thorough benchmarking and clearer reporting of settings. The paper could make a significant contribution if it convincingly addresses the challenges of applying previous token merging techniques to diffusion transformers. However, based on the current limited results and unclear experimental settings, it is difficult to draw a definitive conclusion. More comprehensive evaluations and clearer descriptions are needed to support the claims.

While FFT efficiency on GPUs is acknowledged, deployment efficiency remains unclear. Without empirical overhead data, claims of high efficiency are unconvincing. The merging strategy remains heuristic; a more adaptive or learnable approach would strengthen the method.

Comment 5: The DiT experiments and SOTA comparisons are helpful. Which DiT model was used, and how does the method compare to standard token merging? It’s surprising that the model performs well under 75% sparsity—was sparsity applied to all layers or specific ones? What is the end-to-end latency/memory improvement? The current results lack coverage of key settings, and contradict prior findings that token merging often struggles with diffusion transformers, suggesting the need for more thorough benchmarking and clearer reporting of settings. The paper could make a significant contribution if it convincingly addresses the challenges of applying previous token merging techniques to diffusion transformers. However, based on the current limited results and unclear experimental settings, it is difficult to draw a definitive conclusion. More comprehensive evaluations and clearer descriptions are needed to support the claims.

Response: Our method was benchmarked on Flux.1-dev. To overcome the challenges of applying token merging to Diffusion Transformers, makes several key innovations. First, we bypass the initial 10 text-image fusion blocks to prevent the loss of critical cross-modal features. Second, we handle the two distinct DiT block types within the Flux architecture separately: For JointTransformer blocks, where text and image tokens are processed in separate streams, we merge each modality independently before they are concatenated. The Rotary Position Embedding (RoPE) indices are then gathered accordingly to maintain positional integrity. For SingleTransformer blocks, where tokens are already fused, we first split the hidden state back into its constituent text and image parts, merge each part individually, and then re-concatenate them.

The end-to-end gains on the DiT-based Flux.1-dev using the Diffusers framework (1024x1024 images, 35 steps) are summarized below. Our method achieves a 23.4% speedup and a 50% memory reduction with a minimal impact on FID.

Comment 6: While FFT efficiency on GPUs is acknowledged, deployment efficiency remains unclear. Without empirical overhead data, claims of high efficiency are unconvincing.

Response: Regarding deployment efficiency, our empirical measurements confirm the overhead is minimal. The wall-clock time for the FFT is under 0.1 ms on average, which accounts for only 0.25% to 0.4% of the total inference time.

Dear Reviewer,

Thank you for the insightful comments. We address the following concerns:

Comment 1: In the experimental result section is unclear whether the baseline in Figure 5 is the original Stable Diffusion v1.5 or the ToMeSD adapted model. This makes it hard to correctly judge the effectiveness of the method. It is also unclear why the latency of the plotted points differ across the three subplots of Figure 5.

Response: To clarify, the baseline in Figure 5 is the ToMeSD adapted model, not the original Stable Diffusion v1.5. Our goal was to directly compare our Fourier Token Merging (FTM) method against ToMeSD, which is a state-of-the-art token merging technique for Stable Diffusion.

Regarding the different latencies across the three subplots in Figure 5, these variations are intentional. Each subplot represents a different merging and/or truncation ratio applied by our Fourier Merging method. The latency ranges from 2 s/im (with no merging) to as low as 1.3 s/im and we selects those Pareto optimals to give a direct comparison. Larger merging or truncation ratios inherently lead to lower latencies due to the reduction in computational load. By showcasing these different ratios, we demonstrate the trade-off between inference speed and image quality (FID score) for both our method and the ToMeSD baseline. This allows for a more comprehensive comparison across a range of operational points.

For instance, Figure 5 clearly illustrates that for the same level of image quality (e.g., around 41.5 FID), our FTM method achieves approximately 1.5$\times$ speedup compared to ToMeSD. Conversely, if we target the same inference speed (e.g., around 1.38 seconds per image), our method yields a 0.5 lower FID score, indicating superior image quality.

Comment 2: There is a stark lack of visual examples of the generated images. This would in my opinion be an essential part of a paper focused on image generation.

Response: We'll address this by adding a dedicated section with visual comparisons of images generated using our method at various merging ratios, along with those from the original Stable Diffusion and the ToMeSD adapted model. It will clearly illustrate the quality retention of our method, even with higher merging ratios.

Comment 3: Has the authors considered using the proposed methods on other image recognition tasks such as image classification? A direct replica of the ToMe off-the-shelf experiments using SWAV, MAE, and ViT could demonstrate how well the proposed method generalizes.

Response: We experimented our Fourier Token Merging in ViT models. Results show that our method generalizes well on image classification tasks. For ImageNet val-1k dataset on the 4090 machine, with the same level of throughput (e.g. around 2000 im/s), our method increases the accuracy by 0.7 point (from 83.0% to 83.79%). With the same level of accuracy, our method achieves 1.2$\times$ speedups.

Comment 4: Does the authors have any idea of why the oscillation in Figure 2 happens? I cannot determine if this is related to the network depth, or other factors. Any insights into this would be beneficial in better understanding the comparison.

Response: These oscillations directly relate to the network depth of the Stable Diffusion model. As network depth increases, token dimensions decrease, leading to greater feature abstraction. Both ToMe and our method show this. The UNet's architecture, with its varying representational changes at different depths, causes these fluctuations in similarity scores, reflecting the changing redundancy and information content of tokens throughout the network.

Comment 5: If the authors can provide justification for only using the Real-coefficients of the Fourier tokens or a comparison with the full complex tokens, this would in my opinion strengthen the methodology significantly.

Response: Since input tokens are real-valued, their Fourier Transform has conjugate symmetry, meaning the imaginary part contains redundant information. We hypothesized the real part alone could adequately capture essential frequency-domain information for token merging. Furthermore, discarding the imaginary part significantly boosts speed and simplifies implementation by reducing computational and memory demands. Our experiments also confirmed this choice. Using only the real part consistently yielded superior or comparable performance. For instance, it achieved 84.33% accuracy, slightly outperforming the absolute value (84.29%) and full complex representation (84%).

Comment 6: It is unclear why the assumption of linear attention is made on line 117. To me this seems like a major assumption which does not correspond to the actual operations of Tome. If a justification of this can be provided the proof would in my opinion be stronger.

Response: While ToMe uses softmax attention, we made this simplification for tractable and interpretable approximation error analysis. Softmax's non-linearity makes error decomposition difficult. Linear attention allows us to directly expose second-order token interaction influence. It also clearly decomposes approximation error after token clustering, offering theoretical insight into how merging impacts self-attention. Our goal isn't exact ToMe characterization, but to provide theoretical understanding through an analyzable surrogate, a common approach in approximate attention research (e.g., Performer, Linformer).

Comment 7: The authors state on line 306 that ``One limitation is that it is currently limited to transformer-based diffusion models". I believe this limitation is incorrect, as the proposed method is not inherently limited to diffusion models. Instead I would suggest that the manuscript is currently limited in its evaluation due to only comparing on a single method (Stable Diffusion v1.5.

Response: Aside from the image classfication experiment, we will also include results for Diffusion Transformer (DiT) models in the revision. Specifically, we have done experiments on DiT models, tailoring FTM to their specific transformer blocks. Results are promising: even with a large merging ratio (0.75), we maintain high image quality (1.83 FID increase) while achieving over 20% latency reduction. This suggests FTM's strong generalization capabilities to diverse diffusion models, including DiT-like architectures, distinguishing our approach and highlighting its potential for current foundation models.

Comment 8: Paper Formatting Concerns.

Response: We're grateful for these precise suggestions, as they significantly improve the accuracy and clarity of our paper.