FIPER: Factorized Features for Robust Image Super-Resolution and Compression

We sincerely thank the reviewer for recognizing our solid experiments and novel pipeline. We apologize for the clarity issues and address each question comprehensively below:

Q1. Notation in Equation (9)

We apologize for any confusion caused by our notation. We mainly follow the notations and definitions in [1]. In Equation (9): $L = R(\hat{y}) + R(\hat{z}) + \lambda\,D(x, \hat{x})$

• $\hat{z}$ refers to the hyperprior, an auxiliary latent used to estimate the scale parameters for entropy coding, following the standard variational hyperprior framework.
• $R(\cdot)$ denotes the bit-rate cost of encoding a quantized variable, representing the rate term in the rate–distortion objective.

Thus, $R(\hat{y}) + R(\hat{z})$ measures the total bit cost for both the primary latents and the hyperprior, and $\lambda\, D(x, \hat{x})$ balances this cost against reconstruction quality. We will clarify these definitions in the revised manuscript.

[1] Ballé, Johannes, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. "Variational image compression with a scale hyperprior." ICLR 2018.

Q2. Pixel-wise vs Global Transform

Thank you for raising this point. In the classical Fourier representation, each frequency component is applied uniformly across the whole image: a single scalar coefficient multiplies a fixed sinusoidal pattern, so every pixel “sees” the same globally. This is powerful for capturing overall periodicity, but it cannot adapt to local variations in texture or detail.

In our Factorized Features framework, we generalize this in two key ways, trading pure global consistency for content-aware flexibility while retaining frequency modeling:

1. Learnable, pixel-wise basis maps Instead of a fixed sinusoid, each basis becomes a full-resolution map that can vary from one pixel to the next, so that regions with different patterns (e.g., smooth sky vs. brick wall) get different frequency accommodations.

2. Spatially-varying coefficient maps Rather than a single scalar per frequency, we produce a coefficient map for each basis. This allows each location to choose how strongly it uses each basis, adapting dynamically to local edge strength, texture complexity, or noise.

Together, these two changes turn the rigid global transform into a hybrid: - It still captures multi-frequency structure (low and high frequencies) across the image, - But it now automatically adapts to fine details and local content via learnable maps.

Why this matters

• Local Detail Preservation: Regions that need more high-frequency emphasis (e.g., sharp edges) can boost those basis maps without affecting smooth regions.
• Task Generality: This flexible decomposition works not only for super-resolution but also for compression (and, in principle, deblurring or denoising), since all these tasks benefit from recovering both broad structures and subtle textures.

In short, we move from “one size fits all” sinusoidal bases and scalar coefficients to a rich, pixel-wise basis–coefficient decomposition, which preserves the spirit of Fourier analysis while unlocking powerful local adaptation.

Q3. Figure 2(b) Clarification

We thank the reviewer for this question. In Fig. 2(b), “trainable basis settings” means that we enable gradient updates on the coordinate basis by setting require_grad=True for the (x,y) tensors. In other words, although initialized with fixed grid values, these coordinates become learnable parameters during training (see lines 170–171). We will clarify this point in the revised manuscript.

Q4. Alpha (α) in Figure 4

We can see in Figure 4 with sawtooth transformation $\gamma$ the feature is prompted to learn pattern from different scale in a patch-like manner, e.g. $\gamma_2$ makes the same basis repeat four times in the grid. However, only $\gamma$ is not enough for very fine details; therefore, we introduce $\alpha$ and $\psi$. Each basis is forced to accommodate both high-frequency (large $\alpha$) and low-frequency (small $\alpha$) components together with periodic functions $\psi$, i.e., larger $\alpha$ will make $\psi$ oscillate sharper. To be more general, sawtooth transformation is to form a patch-like pattern, and $\alpha$ is to model pixel-wise high-frequency details.

Q5. High-frequency Feature Visualization

Note that due to rebuttal guidelines, we can not include images or external links here. We will include visual examples of the extracted high-frequency feature maps in the revised version to illustrate what the network learns.

Q6. Learning k

• $k$: Pure scalar hyperparameter in sawtooth transformation in line 162, where we use $\{256, 128, 64, 32,16,8\}$. It is not the output of convolutional layers, found empirically using a validation set and fixed throughout training.
• $K$: The number of $\alpha$. We conduct experiments to validate the setting.

We will make this explicit in the revised manuscript.

Q7. Sequential vs Parallel Processing

Although the coefficient and basis branches operate in parallel, the main feature $X_{coeff}$ is extracted via the backbone network. The basis transformer then downsamples those features to construct the basis. We will rephrase the caption and accompanying text to clearly separate these two stages.

Q8. Alternative Coordinate Transformations

L1. Computational Limitations

Thank you for asking us to clarify this point. By “computation-limited scenarios,” we refer to environments such as on-device real-time decoding where both latency and memory are severely constrained. Although our Factorized Features design already reduces parameters, FLOPs, and peak memory compared to prior methods, the extra steps of basis generation and coefficient fusion still introduce non-negligible overhead under real-time constraints.

However, to demonstrate our method’s effectiveness even under strict computational constraints, we integrated the Factorized Features framework into state-of-the-art lightweight super-resolution models and observed consistent performance gains.

S&W3. Figure 5 Caption

Thank you for highlighting this point. We agree that the presentation around Figure 5 can be made more coherent, and that explicit captions will help readers follow the workflow. In the revised manuscript, we will:

• Clarify the narrative: Reorganize the text in Sections 4.2 and 4.3 to present the coefficient and basis extraction steps in a clear, sequential manner.
• Add detailed captions:

This figure illustrates how Factorized Features are used in Super-Resolution and Image Compression. (a) Given a low-resolution input image, we first extract $X_{coeff}$ feature with Coefficient Backbone. Next, we generate the basis and coefficient with Basis Swin Transformer and convolution layers, respectively, from the same $X_{coeff}$. Finally, the prediction is reconstructed by Factorized Features Reconstruction. (b) To decrease distortion in image compression, we replace the synthesis transform of the traditional learned image compression pipeline with (a) by aligning spatial resolution and latent channels.

These changes will ensure the description is coherent and that Figure 5 stands on its own as a self-contained illustration of our method.

Dear Reviewer evQ6,

Thank you for your effort in the review process, including the response to authors' feedback. If you believe the discussion with authors can be concluded, please submit the Mandatory Acknowledgement as required by the program of the conference.

Thank you,

AC

Thank you again for your detailed feedback and for confirming that all concerns have been addressed. We appreciate your time and judgment.

We sincerely thank the reviewer for recognizing our work as "excellent" in quality and clarity, and for acknowledging our "significant and novel contribution." We address all concerns comprehensively below:

W1. Learned Non-uniform Basis Mechanism

In lines 89-100 of the Supplement, we describe implementation details of how the basis is generated. To be more clear, given an image, we extract a feature $X_{coeff}$ by Coefficient Backbone, and then the Basis Transformer is applied to refine the feature with downsampling for final basis generation. After the reconstruction with the coefficient and basis, the prediction is used to compute the loss and subsequently optimize the network. With gradient descent, when the Basis Transformer is optimized, the basis is optimized simultaneously, i.e., the models are trained to adapt to content-aware basis, so the basis is not calculated by heuristic rules. This design enables content-aware basis generation: smooth regions get low-frequency bases while textured areas receive high-frequency ones. We will expand this explanation with architectural diagrams in the revision.

W2. Mergeable-basis Property Explanation

Thank you for this valuable suggestion. We will expand the discussion around Eq. 10 in the main text to clarify both the benefits and limitations of our mergeable-basis approach. Specifically:

As detailed in lines 133–141 of the Supplementary, our merged basis exploits the observation that different images often share common frequency components. By learning a single, shared set of basis functions, we can leverage the frequently appearing similar or repetitive patterns across images to greatly reduce coding redundancy in multi-image compression. In practice, this means fewer total parameters and lower computational cost required when compressing an image collection. We also note the trade-off: when images are highly diverse, forcing them into one shared basis can increase reconstruction error (as seen by the quality drop for larger M in Fig. 7). Still, this content-aware information redundancy exploitation enables a significant reduction in resource usage, and we will make these points explicit in the revised manuscript.

Q1. Evaluation on More Tasks

We acknowledge this limitation and have extended our evaluation:

GoPro Dataset (Motion Deblurring)

Dense-Haze Dataset (Dehazing)

Our framework clearly generalizes across diverse low-level vision tasks. Due to time constraints in the rebuttal phase, we have focused on these two additional benchmarks, but we will include further evaluations on denoising, deraining, and other tasks in the revised manuscript.

Futhermore, to demonstrate our method’s broader effectiveness, we integrated the Factorized Features framework into state-of-the-art lightweight super-resolution models and observed consistent performance gains.

[1] Liu, X., Liu, J., Tang, J., & Wu, G. (2025). CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 17902-17912).

[2] Lee, D., Yun, S., & Ro, Y. (2025). Emulating Self-attention with Convolution for Efficient Image Super-Resolution. arXiv preprint arXiv:2503.06671.

Q2. Comprehensive Component Ablation

The experiment is conducted with ESC-F as in Q1.

Comparison of multi-frequency modulation and the number of frequency components

Comparison of Coordinate Transformation

We choose $N=6$ for better quality and training efficiency ($N=10$ converges slower). Note that due to time and resource constraint we omit spatial variant coefficients since it is quite intuitive and also learned non-uniform bases since it is discussed in Figure 2.

Thank you for your kind feedback! We are happy to hear that our responses addressed most of your concerns. Please do not hesitate to point out any remaining questions we might have missed.

We sincerely thank the reviewer for their positive assessment and constructive feedback. We are encouraged that they find our frequency-aware decomposition "well motivated and novel" and acknowledge our significant improvements (21% BD-Rate gain). We address each concern below:

W1. Lightweight Backbone Evaluation

Thank you for this important suggestion. We have conducted additional experiments with lightweight backbones and observed that Factorized Features provides consistent gains even in resource-constrained settings: lightweight models can also benefit from our Factorized Features.

Due to time and resource constraints, we have focused on these initial evaluations on super-resolution; comparisons with lightweight image compression methods[3][4] will be included in the final version. We will add this analysis to the paper.

[1] Liu, X., Liu, J., Tang, J., & Wu, G. (2025). CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 17902-17912).

[2] Lee, D., Yun, S., & Ro, Y. (2025). Emulating Self-attention with Convolution for Efficient Image Super-Resolution. arXiv preprint arXiv:2503.06671.

[3] Bao, Y., Tan, W., Jia, C., Li, M., Liang, Y., & Tian, Y. (2025). ShiftLIC: Lightweight Learned Image Compression with Spatial-Channel Shift Operations. IEEE Transactions on Circuits and Systems for Video Technology.

[4] Wang, S., Cheng, Z., Feng, D., Lu, G., Song, L., & Zhang, W. (2024, December). Asymllic: Asymmetric lightweight learned image compression. In 2024 IEEE International Conference on Visual Communications and Image Processing (VCIP) (pp. 1-5). IEEE.

W2. Limited Task Coverage

We acknowledge this limitation and have extended our evaluation:

GoPro Dataset (Motion Deblurring)

Dense-Haze Dataset (Dehazing)

Our framework clearly generalizes across diverse low-level vision tasks. Due to time constraints in the rebuttal phase, we have focused on these two additional benchmarks, but we will include further evaluations on denoising, deraining, and other tasks in the revised manuscript.

Q1. Runtime with Lightweight Backbones

Please see W1. Note that ESC is convolution-based while CATANet is a hybrid transformer.

Q2. Hyperparameter Choices (N=6, α∈{1,4,16,64})

We choose $N=6$ for better quality and training efficiency ($N=10$ converges slower).

Q3. Figure 3 Clarity

Figure 3 shows the difference between vanilla fields, coordinate transformation, and multi-frequency modulation. Specifically, 3.a is a vanilla basis-coefficient field, and 3.b adds sawtooth transformation. We can see in 3.b that the coordinate transformation explicitly models a patch-like pattern, e.g., the second plot from the top is divided into two periods. By enforcing such frequency components, the models can decompose the signal effectively.

Next, 3.d is our full formulation, while 3.c has no $\alpha$. Intuitively, from 3.b to 3.c, applying $\psi$ (we use $cos$ here) should not have much effect on the performance since we do not have any constraint on these learnable bases. With the introduction of $\alpha$, the models are forced to attend to signal components of different frequencies. Note that due to rebuttal guidelines, we can not include images or external links here; we will adjust the graph layout and font for better clarity.

Q4. Alternative Transformations

The experiment is conducted with ESC-F as in W1.

Dear Reviewer Wt2E,

This is a reminder that the author-reviewer discussion period is ending soon on Aug. 6 (AOE), and you have not yet responded to authors' rebuttal. Please read authors' rebuttal as soon as possible, and engage in any necessary discussions, and consider if you would like to update your review and score. Please at least submit the Mandatory Acknowledgement as a sign that you have completed this task.

Thank you for your service in the review process.

AC

We sincerely thank the reviewer for their thorough and constructive feedback. We are encouraged that they find our paper "interesting" and acknowledge our contributions, including unified representation, explicit frequency modeling, optimized compression pipeline, and strong empirical performance. Below, we address each concern:

W1. Computational Efficiency

We acknowledge this important practical concern. In Table 1, only the parameter count of our HAT-F is slightly more than its counterpart, HAT-L; the MAC and Memory footprint of all our models are lower than their counterparts. In Table 2, our TCM-HAT-F-multi, though with slightly more parameters, actually achieves competitive encoding time and even the lowest decoding time of all models. Furthermore, to demonstrate the effectiveness of our method even under computation-constrained scenarios, we integrate our Factorized Features framework to state-of-the-art lightweight super-resolution models and observe consistent improvement.

[1] Liu, X., Liu, J., Tang, J., & Wu, G. (2025). CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 17902-17912).

[2] Lee, D., Yun, S., & Ro, Y. (2025). Emulating Self-attention with Convolution for Efficient Image Super-Resolution. arXiv preprint arXiv:2503.06671.

W2. Lack of Semantic Information

Good catch! Our current focus is on frequency-domain structural modeling rather than semantic understanding. However, our framework is extensible - for example, we can insert the CLIP feature into the Coefficient Backbone [3] or condition attention on human instructions [4] in the Basis Transformer for explicit guidance. We will leave this to future work.

[3] Lu, Z., Xia, Q., Wang, W., & Wang, F. (2025). CLIP-aware domain-adaptive super-resolution. Multimedia Systems, 31(3), 257.

[4] Conde, M. V., Geigle, G., & Timofte, R. (2024, September). Instructir: High-quality image restoration following human instructions. In European Conference on Computer Vision (pp. 1-21). Cham: Springer Nature Switzerland.

W3. Scalability in Multi-Frame Compression

As shown in Figure 7, our method demonstrates robust scalability. The PSNR remains high even at M=24, with only a minor degradation of approximately 0.1 dB when increasing M from 8 to 24. This slight quality loss, resulting from more frames sharing common bases, is significantly outweighed by the twofold improvement in compression efficiency. This trade-off is more favorable than those offered by existing methods.

Regarding Table 2, the high BD-Rate observed at small values of M (e.g., M=1) is attributed to the overhead from the additional basis transmission branch, as detailed in Figure 10 of the Supplement.

W4. Backbone Architecture Dependence

We integrate our Factorized Features framework into state-of-the-art lightweight super-resolution models and witness consistent improvement as shown in the table of W1.

Q1. Failure Cases

Failure cases typically involve extremely fine textures—details that are absent in the low-resolution (LR) input but appear in the super-resolved (SR) output. However, this has been a shared inherent limitation of super-resolution methods to date. For example, in the top row of Figure 6, some white structures are present in HR but are completely absent in LR.

Note that due to rebuttal guidelines, we can not include images or external links here; visual examples of these failure cases will be provided in the revised manuscript.

L1. Task-Specific Overfitting

We thank the reviewer for this question. We understand it to ask whether our Factorized Features overfit to a single task or dataset. In fact, they exhibit strong cross-task and cross-domain generalization. For example, the same Factorized Features learned for super-resolution transfer seamlessly to image compression (as described in Section 4.3) and other low-level vision tasks such as image restoration (see below).

Even within super-resolution, a model trained exclusively on the DF2K dataset achieves excellent performance on held-out benchmarks (the lightweight SR comparison Table in Question 1 is trained with DF2K). If this does not address your concern, could you please clarify further?

GoPro Dataset (Motion Deblurring)

Dense-Haze Dataset (Dehazing)

Our framework clearly generalizes across diverse low-level vision tasks. Due to time constraints in the rebuttal phase, we have focused on these two additional benchmarks, but we will include further evaluations on denoising, deraining, and other tasks in the revised manuscript.

S&W1. Generalization

We can indeed extend our network to other tasks. For image generation, for example, we can integrate our Factorized Features into the VAE decoder of a diffusion model. For video generation, we can employ our mergeable basis and leverage its properties to enhance structural coherence. We will discuss these extensions in the Future Work / Directions section.

We hope these clarifications address the reviewer's concerns. We believe the strong empirical results, unified framework, and extensibility of our approach constitute significant contributions that merit acceptance. We will incorporate all suggested improvements in the final version.

Dear Reviewer yir9,

This is a reminder that the author-reviewer discussion period is ending soon on Aug. 6 (AOE), and you have not yet responded to authors' rebuttal. Please read authors' rebuttal as soon as possible, and engage in any necessary discussions, and consider if you would like to update your review and score. Please at least submit the Mandatory Acknowledgement as a sign that you have completed this task.

Thank you for your service in the review process.

AC