BiMaCoSR: Binary One-Step Diffusion Model Leveraging Flexible Matrix Compression for Real Super-Resolution

1. The author applies their method solely to the SinSR model. It would be beneficial to conduct experiments with additional diffusion models to evaluate the method's generalizability.

2. The author does not provide real-time speedup results, which are crucial, particularly for ultra-low bit quantization. I am curious to know whether the proposed method leads to actual computational reductions.

No theoretical claims and proofs.

1. It is quite surprising that in Table 2, the proposed method requires fewer FLOPs than ReSTE and XNOR, which do not include any additional computational branches.

2. Why does the author focus exclusively on one-step diffusion models? I haven't found any specific design or rationale for choosing this type of model.

3. It is essential to compare the proposed method with the following works [1, 2, 3] to validate its contribution, as all of them also employ additional computational branches (e.g., LoRA or sparse matrix), similar to the approach proposed here.

[1] Huang W, Liu Y, Qin H, et al. Billm: Pushing the limit of post-training quantization for llms[J]. arXiv preprint arXiv:2402.04291, 2024.

[2] Zhang Y, Qin H, Zhao Z, et al. Flexible residual binarization for image super-resolution[C]//Forty-first International Conference on Machine Learning. 2024.

[3] Li Z, Ni B, Zhang W, et al. Performance guaranteed network acceleration via high-order residual quantization[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2584-2592.

I believe the proposed methods can be applied to any model architecture.

Adding new computation branches is very common in the binarization area. All of the following related studies are missing:

1. The author proposes SBM, but these techniques are widely used in low-bit quantization [5, 6] and binarization [1, 4], and should have been discussed in more detail.

2. Quantization-aware LoRA fine-tuning methods [7, 8] appear to be similar to LRBM, assuming they do not merge LoRA during inference.

3. SVD initialization [7] and magnitude-based selection [5] are also common practices, but related works in this area have not been cited.

[4] Li Z, Yan X, Zhang T, et al. Arb-llm: Alternating refined binarizations for large language models[J]. arXiv preprint arXiv:2410.03129, 2024.

[5] Kim S, Hooper C, Gholami A, et al. Squeezellm: Dense-and-sparse quantization[J]. arXiv preprint arXiv:2306.07629, 2023

[6] Dettmers T, Svirschevski R, Egiazarian V, et al. Spqr: A sparse-quantized representation for near-lossless llm weight compression[J]. arXiv preprint arXiv:2306.03078, 2023.

[7] Guo H, Greengard P, Xing E P, et al. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning[J]. arXiv preprint arXiv:2311.12023, 2023.

[8] Dettmers T, Pagnoni A, Holtzman A, et al. Qlora: Efficient finetuning of quantized llms[J]. Advances in neural information processing systems, 2023, 36: 10088-10115.

The methodology is well-aligned with the paper’s objectives. The goal was to drastically compress a diffusion-based SR model while preserving its high-fidelity output; to that end, the authors combined two complementary strategies: model binarization for compression and one-step distillation for fast inference. This joint approach directly targets both memory and speed objectives, and the method is executed thoughtfully. In particular, the introduction of LRMB and SMB is a clever design choice to meet the quality goal – these branches explicitly mitigate the known weaknesses of binarization (loss of information from small weights and rare large weights). The method leverages known techniques (low-rank approximation, sparse outlier capture, and XNOR-Net binarization) in a novel combination tailored for diffusion SR. The evaluation setup also matches the objectives: since the task is Real-World SR, the authors test on multiple Real-SR benchmarks (RealSR​, DRealSR, and a third dataset, likely a synthetic or DIV2K-based set) covering a variety of real degradations. They report a comprehensive set of 9 evaluation metrics – including standard fidelity metrics (PSNR, SSIM), perceptual distances (LPIPS, DISTS), and no-reference image quality scores (e.g. CLIP-IQA, MANIQA, NIQE/FID). This broad evaluation criterion is appropriate, as it captures both the fidelity and perceptual quality aspects of SR, aligning with the objective of producing realistic high-quality images.

The paper’s theoretical claims and derivations are mostly straightforward and correct. Rather than introducing new fundamental theory, the authors apply existing theoretical constructs (low-rank matrix factorization, binary convolution via XNOR) to their problem and provide derivations to justify design choices. For example, they express each full-precision weight matrix W as a sum of a low-rank component (matrices B and A from SVD) and a residual to be binarized. This decomposition is mathematically sound and allows them to claim a separation of “low-frequency” vs “high-frequency” information between LRMB and the binarized branch. While the terms “low-frequency” and “high-frequency” are used somewhat intuitively (referring to the magnitude/content of singular values rather than literal spatial frequency), the logic is reasonable: large singular values capture dominant image structures, and retaining them in FP (LRMB) should help reconstruct smooth components, whereas the binary residual can focus on fine textures. This claim is supported by the observed reduction in initial quantization error: after subtracting the SVD-based low-rank part, the norm of the remaining weight error ∥W_res∥²_F drops to 0.1855 from 1.1275 (a substantial reduction). This quantitative evidence backs the theoretical argument that their initialization decouples the weight information effectively.

They also provide complexity analyses to support the “negligible overhead” claim. The paper derives formulas for the storage cost of LRMB: O_s = (mr + rn)B (with B=32 bits for FP) versus the binarized weight cost mn*B' (B'=1 bit). Given r ≪ m,n, they show that even if stored in 32-bit, the LRMB adds only a tiny fraction of what the full matrix would, confirming the overhead is minimal. Similar reasoning is applied to the sparse branch: only k (top 0.1%) entries of each weight matrix are kept, so the extra cost is trivial. All these derivations are mathematically correct. There are no complex new proofs in the paper; rather, the authors ensure that each design element is backed by a clear explanation or formula. I did not find any algebraic errors or logical gaps in these derivations. The use of the straight-through estimator (STE) for binarization is referenced (e.g., ReSTE [1]) and the binary convolution operation is formulated via XNOR and bit-count equations – these are standard in binary neural networks and are presented correctly. One minor observation is that the paper doesn’t formally prove why the combination of branches yields optimal retention of information (that would be a very difficult theoretical guarantee). Instead, it relies on intuitive reasoning (e.g., outlier weights are rare but crucial​, so capturing them in SMB is beneficial) which is then validated experimentally. This approach is acceptable for an applied paper. In summary, the theoretical basis of the method is solid and internally consistent.

[1] Wu, Xiao-Ming, et al. "Estimator meets equilibrium perspective: A rectified straight through estimator for binary neural networks training." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

The experimental design is comprehensive and sound, giving credibility to the results. The authors conduct evaluations on three different SR datasets, covering both real-world degradations and (likely) a standard benchmark, which strengthens the generality of their claims. For each dataset, they report a wide range of metrics (nine in total), ensuring that no single metric bias (e.g., PSNR vs perceptual quality) dominates the assessment​. The comparison includes multiple baselines: (a) the full-precision diffusion models (ResShift and its one-step distilled version SinSR), and (b) several state-of-the-art binarization approaches adapted to the same one-step model (ReActNet, BBCU, ReSTE, BiDM).

The ablation studies further bolster the experimental rigor. The paper includes a breakdown ablation (BMB only vs +LRMB vs +LRMB+SMB), loss function ablation, different ranks for LRMB, and different initialization strategies for the branches. These experiments are well-designed to answer key questions about why the method works. For example, the breakdown ablation clearly shows each component’s effect on performance and verifies that the final design (with both branches) is needed for the best balance of quality and efficiency.

The paper does an excellent job positioning itself in the context of prior work. In the introduction and related work, the authors survey two key areas: diffusion-based super-resolution and network binarization/acceleration. They cite the foundational and latest works in diffusion models for SR, such as SR3 (first iterative SR diffusion), DiffBIR, SinSR (one-step diffusion). On the binarization side, they reference seminal works like XNOR-Net for classification, as well as more recent binarization techniques and benchmarks (ReActNet, Qin et al. 2020/2022 for improved accuracy​, etc.). Crucially, they cite very recent papers that apply quantization to diffusion models: Binary Latent Diffusion (Wang et al. 2023), BiDM (Zheng et al. 2024), and a NeurIPS 2024 work “BI-DiffSR” (Chen et al. 2024)​.

The paper cites most of the crucial prior work, but there are a couple of references that should have been mentioned explicitly:

1. LoRA (Low-Rank Adaptation) – Hu et al., 2021. The idea of using a low-rank decomposition to inject or preserve information is directly inspired by LoRA (as acknowledged), but the original LoRA paper does not appear in the reference list. Citing it would credit the source of the low-rank approach and contextualize LRMB within the broader use of low-rank matrices in neural network compression.
2. Knowledge Distillation for Diffusion Models – While the authors cite SinSR and ResShift for one-step distillation, they might have also referenced the general concept of knowledge distillation (Hinton et al., 2015) or earlier works on distilling iterative generative models. However, this is a minor point since they did cite the specific SR diffusion distillation methods they used.

The method in the paper seems to be a general binarization approach for diffusion models, yet the rationale for validating it on the Sr task is unclear. While the evaluation for the SR task is reasonable.

There are no relevant theoretical proofs in the main text, but rank-related parts are in the supplementary materials. I checked the logic there and found no errors

The experiment includes effect comparisons, key speed tests, and ablation studies on each module. However, it has flaws: no actual - device running time was designed for speed-related aspects; Table 3 of the ablation study lacks BMB + SMB results, even though SMB's sparse matrix might be low-rank.

No relevant papers were found.

No relevant papers were found.