COSMIC: Compress Satellite Image Efficiently via Diffusion Compensation

Thank you for your careful review and valuable feedback.

Q1. More performance comparison tests of various models need to be included.

Please see our general response 2 above.

Q2. The process of handling metadata is not clearly explained.

Sorry for the confusion in writing. First, we normalize each metadata. And then, inspired by the way that the timestep $t$ processed by diffusion model, we use sinusoidal embedding to encode each metadata. For each metadata embedding, we use its corresponding linear function to map it to CLIP space. Finally, we use CLIP to align it with the satellite image.

Q3. Spatiotemporal complexity comparison.

In this paper, we use two lightweight modules, LCB and CAM, while other methods use ordinary convolution. We list their time and space complexity as follows. Assuming that the number of input channels is $c_{in}$, the number of output channels is $c_{out}$, the input feature map size is $n^2$ and the convolution kernel size is $k$.

Please allow us to thank you again for your careful review and valuable feedback, and in particular for recognizing the strengths of our paper in terms of clear writing, novel topic and practical significance.

Kindly let us know if our response and the new experiments have properly addressed your concerns. We are more than happy to answer any additional questions during the post-rebuttal period. Your feedback will be greatly appreciated.

Dear Reviewer AGki,

Thank you very much again for your initial comments. They are extremely valuable for improving our work. We hope our response has adequately addressed your concerns. We shall be grateful if you could kindly give any feedback to our rebuttal.

Best Regard,

#Paper9518 Author(s)

Q1. Encoder Degradation and Training Specificity.

These two questions are just repeating our last section, i.e. Limitations & Future work. Moreover, we are frustrated to find two detectors (GPT-zero and Scribbr) have 100% confidence of AI-generation on your review. It's clearly doubtful if the review can really help this paper's decision process.

Anyway, let us to explain these two questions again.

1. We never claim COSMIC is perfect. In experiments, we notice the performance of COSMIC is slightly degraded at extremely low bit rate and we report this limitation honestly. The reason is simple since the less image content the image encoder provides, the more compensation from diffusion is needed at decompression. It is hard to rely only on diffusion to decompress.
2. We believe that training a diffusion model specifically for satellite images, which has sufficient prior knowledge of satellite images, can solve this problem to some extent. However, this is not the scope of this paper.

Q2. Limited Power Supply Considerations & Real-World Application Scenarios.

Please see our general response 1 above.

Q3. Satellite images typically consider using pixel-level distortion metrics instead of perceptual metrics.

In the paper, we considered both distortion metrics and perceptual metrics, and experimental results show that COSMIC can achieve SOTA performance in both metrics.

Thanks for the reply.

First, thanks for acknowledging our work can ensure that the distribution of the generated image is as close as possible to the distribution of the original image which is the general goal of any lossy image compression methods.

The history of satellites' photography is much longer than image compression algorithms. Of course there are pixel-level image compression methods used by Mars rover Viking [1] launched in 1975, way before the invention of JPEG. Other pixel-level methods are mostly used for lossless compression, which is neither within our scope nor used by satellites launched since 2000. Today, most satellites are using lossy compression methods like JPEG which is not coded and decoded in a pixel level . We list satellites using JPEG as follows: Soloar-B[2], BILSAT-1[3], Cartosat-2[4], TacSat-2[5], TEAMSAT[6], SPOT-5 [7], Cartosat-1[8], CartoSat-2E[9], TurkSat-3USat[10], SAC-C[11], etc.).

Moreover, JPEG also results in uncontrollable distortion but is still widely used by various satellites. JPEG uses different quantization table only to tune the compression ratio which cannot control the distortion performance which depends on the image contents. In our paper, we have demonstrated that COSMIC surpasses JPEG2000 across both distortion metrics and perceptual metrics.

Reference

[1] The Martian Landscape, https://www.nasa.gov/wp-content/uploads/2024/01/sp-425-the-martian-landscape.pdf

[2] Soloar-B, https://www.eoportal.org/satellite-missions/solar-b#spacecraft

[3] Bradford A, Gomes L M, Sweeting M, et al. BILSAT-1: A low-cost, agile, earth observation microsatellite for Turkey[J]. Acta astronautica, 2003, 53(4-10): 761-769.

[4] Cartosat-2, https://space.skyrocket.de/doc_sdat/cartosat-2.htm

[5] TacSat-2, https://space.skyrocket.de/doc_sdat/tacsat-2.htm

[6] TEAMSAT, https://www.eoportal.org/satellite-missions/teamsat#overview

[7] SPOT-5, https://www.eoportal.org/satellite-missions/spot-5#launch

[8] Cartosat-1, https://earth.esa.int/eogateway/missions/irs-p5

[9] CartoSat-2E, https://www.eoportal.org/satellite-missions/cartosat-2e#spacecraft

[10] TurkSat-3USat, https://www.eoportal.org/satellite-missions/turksat-3usat#transponder

[11] SAC-C, https://www.eoportal.org/satellite-missions/sac-c#mmrs-multispectral-medium-resolution-scanner

Thanks for the very detailed review and suggestions. Fig.S2 can be found in global response PDF.

Q1. Please rewrite the text and the figures to clarify the distinction between training and testing stages.

Sorry for the confusion in writing. The training is divided into two stages.

In the first stage, we train the compression model. Since the Image decoder $\mathcal{D}$ needs two parts of information for decoding, they correspond to $y\prime$ in Fig.S2, which represents the feature map extracted by the on-board image compression encoder, and $z_0$, which represents the compensation information. Therefore, we introduce another Image encoder, corresponding to the Image encoder in the Compensation Model part of Fig.S2, to extract compensation information $z_0$ from the original image. In the first stage, $\mathcal{E}$, $\tilde{\mathcal{E}}$ and $\mathcal{D}$ are trained together.

In the second stage of training, we freeze the parameters of $\mathcal{E}$, $\tilde{\mathcal{E}}$ and $\mathcal{D}$, and train the noise prediction network, with the goal of making the information generated by the diffusion model as close to $z_0$ as possible, denoted as $z_0\prime$, so as to generate the compensation information required by the decoder.

In the testing stage, the trained diffusion model can generate compensation information $z_0\prime$. Therefore, we no longer need $\tilde{\mathcal{E}}$. The $z_0\prime$ generated by the diffusion model replaces the $z_0$ extracted by $\tilde{\mathcal{E}}$ to help the image decoder decompress the image.

We will improve the text and figures to guarantee a better reading experience.

Q2. Update the literature review with models used for compression on-board satellites.

We will add the following content to the Background section.

There are many methods for remote sensing image compression, but most of them don't focus on onboard deployment. There are also some works for compressing data on satellites. [1] used the CAE model to extract image features and reduce the image dimension to achieve compression. However, this method only considers the reduction of image dimension and does not consider the arithmetic coding process in the actual transmission process, resulting in the image compression rate being fixed at 8 and the bpp not being able to be flexibly adjusted. [2] proposed the complexity-reduced VAE, which reduces the amount of calculation by reducing the number of model channels and the entropy model structure. However, violently reducing the number of model channels will lead to a significant decline in model performance.

Q3. No comparisons with efficient compression algorithms used on satellites or edge devices.

On terrain, there have been some efficient compression algorithms for edge devices. However, on terrain, edge devices are usually used to decompress images at the receiving end [3][4][5], for example, receiving a picture on a smartphone. Therefore, existing efficient compression algorithms for ground-edge devices usually focus on decoder efficiency, which is not applicable to satellite scenarios. Few works focus on encoder efficiency on satellites for image compression. We choose the European Space Agency (ESA)'s paper [1] as a baseline and list comparison as follows and will add them in revision. [1] conducts experiments on three platforms, respectively NVIDIA GeForce GTX 1650, Intel Myriad 2 VPU, and Intel CoreTM i7-6700 Processor. However, the power consumption of GPU and CPU shown in the article is 83W and 45.5W respectively (the results are shown in Table V in [1]), which is unpractical for power-constrained satellites and no satellites deploy GeForce GTX 1650 or Intel i7-6700 as payloads. For VPU, according to Table V, taking 10.92 seconds to process 2048 $256\times256$ patches, we can infer that the throughput onboard of [1] is 98.36Mbps, while COSMIC can achieve 507.37Mbps within satellite supported power range [6].

Q4. Authors claim that the advantages of the proposed method are particularly visible on image seams (page 7); elsewhere in the paper, it is mentioned that the proposed method deals with image patches. How can the proposed method show advantages on image seams while it is working on the individual patches as input and not the stitched image?

Earth-observation satellites take large photos, for example, the swath width of WorldView-3 is 13.1km with 1m GSD [7] which will lead to several GBs raw data per photo. The constrained computing resources on the satellites cannot support to compress an entire photo. Therefore, satellite photos are typically compressed by tiling to small images and compression onboard. After receiving on the ground, they are decompressed and stitched together. COSMIC and all baselines all follow this process and do not specially process seams. Generally speaking, without any special processing on the seams, the higher the decompression image fidelity, the less noticeable the seams will be. Moreover, stitching satellite image algorithms is orthogonal to compressing each tile, and we’ll discuss them in revision.

Reference

[1] Artificial intelligence based on-board image compression for the Φ-Sat-2 mission. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023.

[2] Reduced-complexity end-to-end variational autoencoder for on board satellite image compression. Remote Sensing, 2021.

[3] Computationally efficient neural image compression. arXiv:1912.08771, 2019.

[4] Computationally-efficient neural image compression with shallow decoders. ICCV 2023.

[5] Complexity-guided slimmable decoder for efficient deep video compression. CVPR 2023.

[6] Wildfires From Space. https://blogs.nvidia.com/blog/ororatech-wildfires-from-space/, 2021.

[7]WorldView-3 DG_WorldView3_DS_2014.pdf (spaceimagingme.com)

This is the response to Q3.

Q3. Add a baseline to compare with efficient compression algorithms used on satellites

To further alleviate your concerns, we select a representative work [4] of ESA for onboard compression for comparison. We use the same model structure as in [4] and triple the number of channels to adapt RGB image as input. We retrain the models with fMoW dataset for a fair comparison with Adam optimizer with $lr = 1 × 10^{−4}$ for 100 epochs with a batchsize of 64.

• COSMIC can still achieve SOTA results on distortion and perception metrics. The results show that under similar PSNR, COSMIC achieve higher MS-SSIM, lower LPIPS and FID at lower bpp. As [4] only considers the reduction of image dimension, only some fixed bpps can be achieved. And due to the serious reduction of image dimension, a large amount of information is lost, and the image reconstruction quality is poor.

• For efficiency comparison, COSMIC can reduce FLOPs by $3\times$ and increase throughput by $5\times$, as shown in the following table. [4] conducts experiments on VPU platform, and we use the same level computing power edge devices, Jetson Xavier NX. For detailed information, please refer to Global response Q1 and Rebuttal Q3.

Reference

[1] Remote sensing image compression based on high-frequency and low-frequency components. IEEE Transactions on Geoscience and Remote Sensing, 2024.

[2] Remote sensing image compression based on the multiple prior information. Remote Sensing, 15(8):2211, 2023.

[3] Global priors with anchored-stripe attention and multiscale convolution for remote sensing images compression. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023.

[4] Artificial intelligence based on-board image compression for the Φ-Sat-2 mission. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023.

This is the response to Q2. We’ll make the revision as follow:

Q2. Update the literature review in Background section

We revise the Background section as follow:

1. Remove the claim that no algorithm for compressing data on satellite exist.
2. Update the literature review of onboard image compression algorithm.

The final revision in Sec2.1 from L112 to L119 is as follow (strike-through means the removed content and Italic+Bold indicates newly added):

Although there are some works on remote sensing image compression, none of the previous work is targeted at on-board computing scenarios. There are some compression methods specifically for remote sensing images [1,2,3]. [1] uses discrete wavelet transform to divide image features into high-frequency features and low-frequency features, and design a frequency domain encoding-decoding module to preserve high-frequency information, thereby improving the compression performance. [2] explore local and non-local redundancy through a mixed hyperprior network to improve entropy model estimation accuracy. Few of these work focus on onboard deployment. [4] use the CAE model to extract image features and reduce the image dimension to achieve compression, and deploy the model on VPU. However, this method only considers the reduction of image dimension and does not consider the arithmetic coding process in the actual transmission process, resulting in the image compression rate can only be adjusted by changing the model architecture.

Reference

[1] Remote sensing image compression based on high-frequency and low-frequency components. IEEE Transactions on Geoscience and Remote Sensing, 2024.

[2] Remote sensing image compression based on the multiple prior information. Remote Sensing, 15(8):2211, 2023.

[3] Global priors with anchored-stripe attention and multiscale convolution for remote sensing images compression. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023.

[4] Artificial intelligence based on-board image compression for the Φ-Sat-2 mission. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023.

Thank you for the feedback.

Given the character limit (5000), we have to response each question separately in three comment. This is the response to Q1. We’ll make the revision as follow:

Q1. Rewrite the training and testing stage

We revise the Method section as follow:

1. Add more detailed training process and rearrange the order for clearer expression.
2. Clarify the distinction between training and testing stages.

The final revision in Sec4.3 from L198 to L205 is as follow (strike-through means the removed content and Italic+Bold indicates newly added):

We first determine what details are lost in $\lfloor \mathbf{y} \rceil$. The training is divided into two stages. In the first stage, we train the compression model. Here, the image decoder receives two parts of information, one of which is the latent representation $\lfloor \mathbf{y} \rceil$ extracted by the encoder $\mathcal{E}$ on the satellite, and the other part is the information $\mathrm{z}_0$ extracted from the original image $\mathbf{x}_0$ by the image encoder $\tilde{\mathcal{E}}$ on the ground, which is used as compensation to $\lfloor \mathbf{y} \rceil$.Since the Image decoder $\mathcal{D}$ needs two parts of information (i.e. $y\prime$ and $z_0$ in Figure 2) for decoding, we introduce another image encoder $\tilde{\mathcal{E}}$ to extract compensation information $z_0$ from the original image. In the first stage, $\mathcal{E}$, $\tilde{\mathcal{E}}$ and $\mathcal{D}$ are trained together. In the second stage of training, we freeze the parameters of $\mathcal{E}$, $\tilde{\mathcal{E}}$ and $\mathcal{D}$, and train the noise prediction network, with the goal of making the information generated by the diffusion model as close to $z_0$ as possible, denoted as $z_0\prime$, so as to generate the compensation information required by the decoder. During the inference phase, $\mathrm{z}_0$ is replaced by $\mathrm{z}_0\prime$ generated from Gaussian noise by diffusion under the guidance of specific conditions. the trained diffusion model can generate compensation information $z_0\prime$. Therefore, we no longer need $\tilde{\mathcal{E}}$. The $z_0\prime$ generated by the diffusion model replaces the $z_0$ extracted by $\tilde{\mathcal{E}}$ to help the image decoder decompress the image.

We thank the reviewer for this thoughtful review and we are glad to see their positive assessment.

Note that Fig.S2 can be found in the PDF attached to the global response.

Q1. More in-depth discussion on power challenges.

Please see our general response 1 above.

Q2. Issues in the writing.

Sorry for the confusion. The image compression encoder is deployed on the satellite, which is lightweight and used to extract satellite image features. In Fig.S2, it corresponds to the image encoder on the satellite in the Compression Model part. The image encoder corresponds to the Image Encoder module in the Compensation Model part in Fig.S2. This module is only used during training to extract the information lost in the lossy compression process to provide compensation for the decompression process, and help finetune the diffusion to learn how to generate the compensation information as a target. During the inference, this module is not used, and the compensation information is generated by diffusion. Image Decoder corresponds to the Image Decoder module in the Compression Model part of Fig.S2, which is used for the image decompression process. We make detailed annotations in Fig.S2, and will further explain and modify them in revision.

Q3. Different satellites may carry different sensors, leading to variations in the type of metadata (m), which may affect the method's performance.

Good question. We demonstrate that COSMIC has SOTA results with only three common metadata (i.e. location, timestamp, and GSD) on satellites. Different satellites carry different sensors but there are several sensors on almost all satellites. Take LANDSAT8 launched by NASA in 2013 [1] and Sentinel-2 launched by ESA in 2015 [2] as examples, they can all collect location, timestamp, and GSD. If we use these three metadata and take a bpp of 0.46 to experiment with COSMIS, PSNR and MS-SSIM decrease slightly, respectively from 27.31 to 27.20 and from 0.969 to 0.968， which can still guarantee the SOTA results. We believe that more metadata can achieve better results, and COSMIC can still achieve SOTA performance even with commonly used metadata. We’ll add them in revision.

Reference

[1] Landsat 8 (L8) Data Users Handbook Version 5.0 https://www.usgs.gov/landsat-missions/landsat-8-data-users-handbook

[2]Sentinel-2 User Handbook sentinel.esa.int/documents/247904/685211/Sentinel-2_User_Handbook

Please allow us to thank you again for reviewing our paper and the insightful comments, and in particular for recognizing the strengths of our paper in terms of novel method, highly feasible, good soundness, and good contribution.

Kindly let us know if our response and the new experiments have properly addressed your concerns. We are more than happy to answer any additional questions during the post-rebuttal period. Your feedback will be greatly appreciated.