Spectral Compressive Imaging via Chromaticity-Intensity Decomposition

Thanks for your valuable comments, we address your concerns one by one.

Q4-1: Utility of chromaticity in downstream tasks where intensity is also important. So why not reconstruct the hyperspectral cube and then, if needed, decompose it?

A4-1: We agree that both chromaticity and intensity are important for downstream applications such as material identification and surgical guidance. However, in this work, we specifically focus on chromaticity reconstruction, due to the following considerations:

(1) Task difficulty and reconstruction fidelity. Chromaticity encodes fine-grained texture and material information and is generally harder to reconstruct, whereas intensity primarily reflects low-frequency lighting patterns, which are relatively easier to recover. Therefore, we directly supervise and reconstruct the chromaticity component, which proves to be more effective than reconstructing the full hyperspectral cube followed by decomposition (see Table 2). The direct supervision helps preserve high-frequency chromatic information crucial for tasks like object fingerprinting.

(2) Joint ambiguity in decomposition Jointly reconstructing both chromaticity and intensity introduces ambiguity, especially in regions where the product of the two can yield similar measurements but from different chromaticity-intensity combinations. Our current pipeline does not explicitly resolve this ambiguity. However, we observe that a stage-wise strategy—where the first stage reconstructs intensity (with frozen weights) and the subsequent stages focus on chromaticity—still achieves competitive performance. This code will be released.

Q4-2: Utility of chromaticity in downstream tasks where intensity is also important, still needs calibration.

A4-2: We agree that absolute chromaticity and illumination intensity can be obtained through pre-calibration using a spectrally-flat target. In fact, we also plan to incorporate this calibration-based strategy in future work to estimate physically accurate lighting conditions. However, in our current experiments, we adopt a normalized dual-path PAN image as a proxy for intensity. This is not a physically absolute lighting measurement, but a relative intensity estimate. This choice is motivated by two practical considerations:

(1) Stability during training. Using normalized intensity improves data consistency and helps stabilize model training, especially under variations in exposure, gain, and scene content.

(2) Focus on spatial variation rather than absolute magnitude. Our aim is to leverage the spatial structure of intensity changes (e.g., shading, highlights) rather than the absolute illumination levels. The relative intensity still captures rich cues about lighting distribution, which is sufficient for our chromaticity reconstruction pipeline.

Q4-3: About noise amplification in dark region.

A4-3: This is a valid and commonly recognized issue in chromaticity-based representations, especially in low-light or shadowed regions where signal-to-noise ratio is inherently low. Such issue are also extensively discussed in the Retinex and intrinsic image literature (see [1] and [2]). In our case, we agree that this effect exists and is hard to eliminate completely. There are two practical strategies to mitigate it:

(1) Retraining on real-world datasets containing paired chromaticity-intensity hyperspectral data under diverse lighting conditions. This would allow the model to learn better noise-aware priors in low-light regions. Unfortunately, such datasets are currently unavailable to the best of our knowledge.

(2) Post-processing on the reconstructed chromaticity, such as applying TV-denoising or bilateral filtering to suppress artifacts in dark areas. While feasible, this approach introduces additional hyperparameters and lacks ground-truth supervision, making quantitative evaluation difficult.

Q4-4: Real-world experiment difficulties due to misalignment.

A4-4: In our current real-world experiments, however, we use publicly available real CASSI system datasets. That said, we fully acknowledge that in a true dual-camera configuration, such issues can impact the quality of the estimated intensity image, and in turn affect chromaticity reconstruction. We will consider the suggestions.

Q4-5: Noise simulation in simulation and real-data condition.

A4-5: In simulation experiment, both CASSI measurements and the PAN images are noise-free (for fair comparison with the baseline algorithm). In real experiment, follow the setting of TSA-Net [3], we injected 11-bit, quantum efficiency 0.4 shot noise into both CASSI measurements and PAN images.

Q4-6: About adding more real-world dataset testing.

A4-6: We agree that validation on more complex real scenes is important. Due to platform and cost constraints, we have not yet been able to collect a large-scale real dataset. To date, the only publicly available dual‑camera compressive hyperspectral imaging (DCCHI) dataset appears in the paper [4], which provides exactly two real‑scene captures: one ninja scene and one doll scene.

Q4-7: About KAIST dataset selection.

A4-7: The KAIST benchmark for spectral imaging was first proposed in TSA-Net [3], these 10 scenes are randomly chosen, and this benchmark was used in a series of subsequent papers focus on spectral reconstruction like MST[CVPR 2022], DAUHST[NeurIPS 2022], SSR[CVPR 2024], In2SET[CVPR 2024].

Q4-8: Table 1 not highlighted best result.

A4-8: Thank you for pointing this out. We sincerely apologize for the confusion. Few metrics in one baseline work PIDS [5] are incorrect, causing the misunderstanding. The reconstructed results are available in the original benchmark from [5] (which is a dual-camera CASSI baseline), and we have now revised the table accordingly. By the way, to illustrate the effectiveness of our Chroma-Intensity Decomposition method. We adapted PIDS into PIDS-CIDS, where we formulate the optimizationproblem as：

$$𝑐̂ = argmin_c (1/2)‖y − Φ(c ⊙ i′)‖² + τ·TV(c − i'), s.t. i' = interpolate(i_{rgb})，$$

where TV denotes Total Variation and $\text{interpolate}(\mathbf{i}_\text{rgb})$ means to interpolate the RGB image captured by the second camera to spectral channels (28 channels in our case). This is to align with the PIDS setting (the second camera is RGB camera).

It can be found that PIDS-CID better results compared with pure original one. Some visilization results are presented in supplement Figure 7 (denoted as PIDS-RGB).

Q4-9: Suggestions on Writing fixes.

A4-9: Thank you for pointing this out. We will correct the writing accordingly.

References:

[1] Wei et al., Deep Retinex Decomposition for Low-Light Enhancement, BMVC 2018.

[2] Bell, Sean, et al., Intrinsic images in the wild, ACM TOG, 2014.

[3] Meng, et al., End-to-end low cost compressive spectral imaging with spatial-spectral self-attention, ECCV, 2020.

[4] Zhang, et al., Fast hyperspectral image recovery of dual-camera compressive hyperspectral imaging via non-iterative subspace-based fusion, IEEE TIP, 2021.

[5] Chen et al., Prior image guided snapshot compressive spectral imaging, IEEE TPAMI 2023.

Thanks for your valuable comments, we address your concerns one by one.

Q3-1: On the justification of using PAN image as intensity.

A3-1: The idea of using PAN image as intensity is inspired by Retinex theory [1] widely used for low-light image enhancement, where a RGB image is decomposesed into reflectance and illumination. This is typically formulated as:

$$\mathbf{X}(u,v) = \mathbf{C}(u,v) \cdot \mathbf{I}(u,v)$$

where $\mathbf{C}$ denotes chromaticity (color) and $\mathbf{I}$ denotes intensity (illumination). We generalize this concept to hyperspectral images as:

$$\mathbf{X}(u,v,\lambda) = \mathbf{C}(u,v,\lambda) \cdot \mathbf{I}(u,v),$$

where chromaticity and intensity are defined as:

$$\mathbf{C}(u,v,\lambda) = \frac{\mathbf{X}(u,v,\lambda)}{\int \mathbf{X}(u,v,\lambda') d\lambda'}, \quad \mathbf{I}(u,v) = \int \mathbf{X}(u,v,\lambda) d\lambda.$$

In a dual-camera setup with a CASSI sensor and a grayscale PAN camera, both exposed under the same illumination $L(\lambda)$, $s(\lambda)$ is the spectral camera response. we model the PAN image as the integral of three terms:

$$\mathbf{I}_{\text{PAN}}(u,v) = \int s(\lambda) \cdot L(\lambda) \cdot \mathbf{X}(u,v,\lambda) \cdot d\lambda.$$

Substituting the decomposition of $\mathbf{X}$ , we obtain:

$$\mathbf{I}_{\text{PAN}}(u,v) = \mathbf{I}(u,v) \cdot \int s(\lambda) L(\lambda) \mathbf{C}(u,v,\lambda) \, d\lambda.$$

The key observation is that $\mathbf{C}(u,v,\lambda)$ is normalized (the integral of $\mathbf{C}(u,v,\lambda)$ over wavelength equals 1) and typically smooth or slowly varying in $\lambda$, while $s(\lambda)L(\lambda)$ acts as a broadband low-pass kernel. This enables the approximation:

$$\int s(\lambda) L(\lambda) \mathbf{C}(u,v,\lambda) d\lambda \approx k,$$

where $k$ is a scalar constant across spatial positions. Thus, we have:

$$\mathbf{I}_{\text{PAN}}(u,v) \approx k \cdot \mathbf{I}(u,v).$$

To resolve the scale ambiguity, we normalize PAN to [0,1] during both training and inference. This justifies using PAN as a relative intensity estimate in our chromaticity–intensity framework, and we will add this proof into our supplement.

Q3-2: Concerns on network module design and the gain of chromatic-intensity decomposition.

A3-2: This decomposition is the core contribution of our paper and brings several conceptual and practical advantages that go beyond simple numerical gains in PSNR or SSIM. We demonstrate this in two aspects:

(1) This decomposition allows us to separate illumination material-dependent reflectance (chromaticity) and intensity from single measurement, providing a representation that is more physically meaningful and invariant to lighting changes. We believe this will significantly benefit CASSI-based reconstruction and its downstream applications such as remote sensing, water quality detection, or hyperspectral material classification, where chromaticity serves as a more precise fingerprint of material properties.

(2) Secondly, by explicitly modeling intensity, we enable applications that target illumination manipulation. For instance, we can apply gamma correction to the intensity map to achieve low-light enhancement, which would be infeasible if illumination and reflectance were entangled.

About the network design, although our network components (e.g., spatial–spectral transformers) are not entirely novel, their design is carefully adapted to suit the properties of chromaticity (sparse spectra and high-frequency details). We also propose a dual noise estimation module to handle anisotropic noise across different reconstruction stages. This is backed by theoretical modeling and empirical evidence, and improves convergence and visual quality.

The gains of this decomposition is our main contribution, which is manifest in two aspects: spectral reconstruction (Table 1) and chromaticity reconstruction (Table 2). Ours outperform single-camera (DAUHST[NeurIPS 2022], SSR[CVPR 2024]) and dual-camera schemes (PIDS[TPAMI 2023], In2SET[CVPR 2024]).

In summary, our goal is not only to improve metrics but to reformulate the problem in a more interpretable and physically grounded way, which we believe is a valuable step for the CASSI research community.

Q3-3: Manuscript check.

A3-3: Thanks for the suggestion, we will correct the errors accordingly.

References:

[1] Wei et al., Deep Retinex Decomposition for Low-Light Enhancement, BMVC 2018.

Thanks for your recognition and valuable comments, we address your concerns one by one.

Q2-1: Exposure mismatch in dual-camera chromaticity-intensity decomposition.

A2-1: This is a valid concern. However, in practice, we apply normalization to the intensity map to mitigate exposure differences both during training and testing. During training (simulated data), the intensity map is normalized by averaging across all spectral channels: $I(x, y) = \frac{1}{L} \sum_{\lambda=1}^L X(x, y, \lambda)$. During real testing (dual-camera system), we normalize the captured PAN image by dividing by its maximum value: $I_{\text{norm}}(x, y) = \frac{I_{\text{PAN}}(x, y)}{\max(I_{\text{PAN}})}$.

As a result, both the intensity and chromaticity maps are treated as relative quantities. While the absolute radiometric scale may be lost, the relative spatial distribution of intensity is preserved, which is sufficient to support meaningful chromaticity estimation and reconstruction. Moreover this formulation is widely accepted in related literature on (RGB) low-light imaging and intrinsic image decomposition, where illumination is often treated as a relative quantity and gamma correction is used to simulate different lighting conditions (see [1,2]). For example, we can enhance the perceived brightness via: $I_{\text{gamma}}(x, y) = I_{\text{norm}}(x, y)^{\gamma}, \quad \gamma \in (0.5, 2.0)$.

Q2-2: Concerns about Dual Noise-Estimation Module (DNEM).

A2-2: We thank the reviewer for the interest in Dual Noise-Estimation Module (DNEM). We explain the concerns in two aspects.

(1) Positive-definiteness of $\Sigma^{(k)}$. We argue that $\Sigma^{(k)}$ is not a full covariance matrix but a diagonal positive definite matrix representing spatially varying noise variance (see [3]). Specifically, $\Sigma^{(k)} = \operatorname{diag}(\sigma_1^{2}, \dots, \sigma_M^{2})$. To ensure all entries are strictly positive and avoid numerical instability during inversion, we apply the Softplus activation to the output of the CNN branch predicting log-variances and add a small constant $\varepsilon$ for numerical stability: $\sigma_i^{2} = \log(1 + \exp(\hat{\sigma}_i)) + \varepsilon$. where $\hat{\sigma}_i$ is the raw output from the network. This guarantees each entry in $\Sigma^{(k)}$ remains positive-definite by design.

(2) Gradient interference and stability analysis. We employ a shared CNN encoder, followed by two lightweight separate heads to predict $\Sigma^{(k)}$ and ω(k), respectively. This modular design alleviates potential gradient interference. Results (Table 3 and SM Figure 4) show that using learned $\Sigma^{(k)}$ improves reconstruction PSNR and SSIM, highlighting the importance of noise-adaptive spatial modeling in CASSI. We will revise the manuscript to clearly state that $\Sigma^{(k)}$ is a diagonal variance map and provide details about softplus activation and the dual-head structure to improve clarity.

Q2-3: Single-camera models adapted to the dual-camera setting.

A2-3: In the original submission, we compared our proposed CIDNet with In2SET (CVPR-2024) and PIDS (TPAMI-2023), which is the state-of-the-art method for dual-camera compressive hyperspectral reconstruction. Our results showed that CIDNet outperforms In2SET and PIDS significantly in simulated and real scenarios. To futher demonstrate the effectivesss of our decomposition method, we have also adapted popular single-camera reconstruction models (unfolding framework GAP-Net [4] and an E2E model MST [5]) to the dual-camera setting. The detailed results are now summarized as follows. The expriments are compared with 9-stage unfolding framework and it is shown that with Chroma-Intensity Decomposition greatly improve the reconstruction quality.

Q2-4: Additional baseline that directly predicts the HSI cube without Chro-Intensity Decomposition.

A2-4: We have added a new baseline that directly predicts the HSI cube without Chro-Intensity Decomposition. The performance of CIDNet improved significantly after adding C-I decomposition. We compared the MST [5] with our CID-Net. At the same time, without C-I decomposition (i.e., direct reconstructing hyperspectral images without input PAN images), our network still has better performance with lower parameters and FLOPs.

References:

[1] Chen, Chen, et al. "Learning to See in the Dark." CVPR, 2018.

[2] Bell, Sean, et al. "Intrinsic images in the wild." ACM TOG, 2014.

[3] Glaubitz, Jan, et al. "Generalized sparse Bayesian learning and application to image reconstruction." SIAM/ASA Journal on Uncertainty Quantification 11.1 (2023): 262-284.

[4] Meng, et al. "Gap-net for snapshot compressive imaging". *arXiv *, 2020.

[5] Cai, et al. "Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition". CVPR, 2022.

Thanks for your recognition and valuable comments, we address your concerns one by one.

Q1-1: Some equations are confusing, signs and brackets need to be checked.

A1-1: Thanks for your kindly suggestions. We will carefully correct the writting errors in the final manuscript, especially your mentioned

$\hat{\mathbf{c}} = \arg min_{\mathbf{c}} \left(-\frac{1}{2} (\mathbf{y} - \mathbf{H}\mathbf{c})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{y} - \mathbf{H}\mathbf{c})\right) + \tau R(\mathbf{c})$

$\textstyle \hat{\mathbf{c}} = \arg min_{\mathbf{c}, \mathbf{z}} \left(-\frac{1}{2} (\mathbf{y} - \mathbf{H}\mathbf{c})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{y} - \mathbf{H}\mathbf{c})\right) + \tau R(\mathbf{z}), \quad \text{s.t.} \quad \mathbf{c} = \mathbf{z}.$

would be corrected as

$\hat{\mathbf{c}} = \arg min_{\mathbf{c}} \left(\frac{1}{2} (\mathbf{y} - \mathbf{H}\mathbf{c})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{y} - \mathbf{H}\mathbf{c})\right) + \tau R(\mathbf{c})$

$\textstyle \mathbf{\hat{c}},\mathbf{\hat{z}} = \arg min_{\mathbf{c}, \mathbf{z}} \left(-\frac{1}{2} (\mathbf{y} - \mathbf{H}\mathbf{c})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{y} - \mathbf{H}\mathbf{c})\right) + \tau R(\mathbf{z}), \quad \text{s.t.} \quad \mathbf{c} = \mathbf{z}.$

Q1-2: LWSA and TKSA are not explained.

A1-2: We provide motivation and technical details of our hybrid transformer module: LWSA (Local Window Spatial Attention) and TKSA (Top-K Spectral Attention).

Motivation: Chromaticity exhibits rich high-frequency details in the spatial domain (see Figure 1), but its spectral signatures tend to be sparse and low-rank (see Figure 5). To effectively model this structure, we use LWSA (a window-based spatial attention) to capture local spatial textures. We design a Top-K Spectral Attention (TKSA) to focus on the most informative spectral dimensions for each spatial location. While LWSA is a typical window transformer, we focus on the explanation of TKSA.

Techinical details of TKSA: Top-K Spectral Attention (TKSA): Instead of applying global attention across all spectral bands—which is both computationally expensive and often dominated by noisy or redundant information—we propose to:

• Divide the image into non-overlapping spatial windows.
• For each window, we first calculate the attention map in the spectral dimension, this attention map size is C×C, for each line in the attention map, we keep the maximum K value, and the other values are set to -∞, so that after the softmax operation, these values will become 0, and we finally use this attention map to weight the key and get the output.

Q1-3: Ablation study explanation.

A1-3: Below, we clarify the motivation and significance of each experiment by highlighting the core novelties of our work. Our main contribution is to reformulate spectral reconstruction as a chromaticity prediction problem by decomposing the spectral image into intensity × chromaticity. The intensity is directly obtained from the PAN image captured by the dual-camera CASSI system, while chromaticity contains the essential spectral variation. Based on this, we design targeted modules for effective chromaticity reconstruction.

• Table 3 presents the contribution of three components:

(1) Int.: our decomposition-based framework that incorporates PAN-guided intensity;

(2) HSST: a hybrid spatial-spectral transformer tailored for local spatial and sparse spectral patterns in chromaticity;

(3) DNEM: a dual-noise estimation module that models spatially-varying degradation across unfolding stages.

• Table 4 compares different attention types within the encoder-decoder. The hybrid spatial-spectral attention (TopK) yields the best performance, validating the importance of our design.
• Table 5 further verifies that our intensity-chromaticity decomposition generalizes well across different types of frameworks (iterative, end-to-end, and unfolding), demonstrating its broad applicability and core novelty.

Q1-4: Is prox and proj implemented as the proposed transformers?

Unfolding network typically alternates between a gradient projection and a proximal mapping step to enforce measurement consistency and data prior updating respectively. Hereby we use $\mathbf{proj}$ and $\mathbf{prox}$ to denote the gradient projection and proximal mapping respectively, where $\mathbf{proj}$ is some linear operators and $\mathbf{prox}$ contains a learnable network with our hybrid transformer.