AstroCompress: A benchmark dataset for multi-purpose compression of astronomical data

We appreciate the reviewer’s compliments regarding our diverse dataset and benchmarks, as well as the feedback. We address the concerns point-by-point:

Two different methods for handling 16-bit images are discussed and tested. It is recommended that the performance differences between these methods be highlighted.

The performance difference between the two formats (2-channel 8-bit v.s. 1-channel 16-bit) seems to be dataset-dependent and mostly an empirical question. For each experiment, we reported whichever approach worked better. We include the compression ratios under both approaches in Table 5, Appendix C.1 of our updated submission.

The proposed dataset’s effectiveness, especially in comparison to existing datasets such as Maireles-González et al. (2023), could be explored more thoroughly.

We would like to emphasize the significant differences between our dataset and the existing [Maireles-González, 2023] dataset:

1. Size: The existing dataset was 1-2 GB, while our additional contribution datasets have a total size of 320 GB.
2. Diversity of telescope location: The existing dataset contains a small number of images from five different ground-based telescopes. Our datasets span across ground-based and space-based. Lossless compression is significantly more important for space-based compression, where bandwidth is strictly and severely limited.
3. Curation of telescope parameters: In our study, we purposefully restricted our scope to imagery data of a single wavelength filter for many of our datasets, but allowed for a wide range of exposure times, readout patterns, and other parameters. In other words, we have intentionally built a diverse dataset in certain parameters, and a constrained dataset in other parameters. The larger size of our dataset allows us and future scientists to study differences in compression performance as a function of these parameters.
4. Train/test split and prevention of data leakage: existing datasets are from unknown locations in the sky that may contain overlapping images. In contrast, we have used sky coordinates to carefully assemble train/test splits that are ready for use by ML practitioners with no overlap between the images in the train and test splits.

The performance of RGB-pretrained models on the AstroCompress dataset are suggested to be examined and reported.

This is an interesting idea; however, an RGB-pretrained model would not be compatible with the AstroCompress dataset. Specifically, RGB images have three 8-bit channels whereas our dataset entails either a single 16-bit channel or two 8-bit channels, so the models need to be redesigned/retrained for each kind of data and won't work across the two kinds of data. In addition, our data distribution is significantly different from natural images: we have background regions that have very low pixel values, and spherical/ellipsoidal objects that represent stars and galaxies which are many orders of magnitude brighter than the background.

---

References

[Maireles-González, 2023] Maireles-González, Òscar, et al. "Efficient Lossless Compression of Integer Astronomical Data." Publications of the Astronomical Society of the Pacific 135.1051 (2023): 094502.

We are glad we were able to address some of your concerns. We will try to answer the ones that remain:

It is foreseeable that an RGB-pretrained model would be incompatible with the AstroCompress dataset. However, it is still valuable to evaluate how significantly the performance is degraded. Conventional methods, such as JPEG2000, generally perform well across most datasets, but learned methods may not.

We think the question of generalization is an interesting one that deserves more attention and research in the neural compression community. We followed the reviewer’s suggestion and did preliminary exploration on how well a neural compression method trained on RGB images generalizes to our astronomy data. As we mentioned earlier, compression models designed for RGB images cannot directly handle our 16-bit data; therefore, we implement a workaround by reformatting each 16-bit pixel as consisting of 3 channels of 8 bits like [MSB, MSB, LSB], where we duplicate the most significant bits (MSB) across the first two channels and fill the last channel with the least significant bits (LSB). As an alternative method, we tried [MSB, LSB, 0]. We will call these workarounds “duplicated” and “zero-padded,” respectively.

We pretrained the L3C model on RGB images from OpenImages (generally around 500 pixels in at least one dimension) and ImageNet64 datasets and evaluated on our single-frame datasets. We also trained IDF on RGB images from Cifar10 dataset and evaluate in the same manner. Overall, we found our RGB-pretrained model to give surprisingly robust performance on astronomy data. When compared to the L3C model pretrained on the respective astronomy datasets, the RGB-pretrained model did worse by 15% - 25% on KECK, HST, and JWST-2D datasets, but actually performed better by 10% - 28% on LCO and SDSS datasets, in terms of compression ratio. The detailed results are given in the following table, with a few values missing due to time constraints:

Given that the deadline for updating the submission has passed, we will incorporate the complete table of results in the next version of our manuscript.

Besides, I am not really sure about how important astronomical data compression is to ICLR community. From the BACKGROUND AND RELATED WORK section in this paper, I find that most papers of astronomical data compression are published in remote sensing and compression related venues, not machine learning related venues (NIPS, ICLR, ICML, CVPR, ICCV, ECCV, etc.). I am not really sure about:

How many people in ICLR community is interested in astronomical data compression?

We acknowledge that astro compression may be “under the radar” for most ML researchers; most existing compression algorithms have been non-neural, and thus have been published in non-AI venues. However, the community is strongly interested in ML applications for science, and we firmly believe our initial foray into ML-based astro data compression demonstrates the promise of this approach and is likely to inspire more interest in the wider ML and astronomy community. We outline the following three-step argument:

1. At ICLR and other AI venues, there is significant interest in both ML-based compression and more broadly ML for scientific applications. Workshops and tutorials on neural compression have been consistently held at top AI/ML venues in the past few years (e.g., ICLR 2021, NeurIPS 2022, ICML/AAAI/UAI 2023, NeurIPS 2024; see https://neuralcompression.github.io/), and an even larger number of events have been held on the applications of ML/AI to the sciences (see, e.g., https://ai4sciencecommunity.github.io/).

2. Scientific data volumes are at an inflection point of exponential growth, from astronomical imaging to biological imaging to genome sequencing, as cited earlier in this rebuttal and in our paper’s introduction. There is a new significant need for advanced codec development, and ML-based methods are currently under-explored. The compression tasks across scientific domains share many similarities:

   • Need for fast encoding on-device and allowance for slow decoding on-cluster, the opposite of internet compression
   • Strict requirements for little to no reconstruction error, unlike the more extensive work on neural lossy compression meant for internet media
   • Highly noisy data

3. We believe astronomy is an ideal domain for spurring the development of future ML methods because it differs from other scientific data sources like biology in key ways:

   • Publicly available
   • Nearly limitless in quantity
   • Unique financial implications – where percentage improvements in compression can translate to hundreds of millions of dollars in mission value, due to limited storage and the remoteness of space. Backlogged data cannot be “rescued” and thus must be deleted.

How many people in the astronomical data compression community will read ICLR?

In recent years, there has been a surge of interest in AI for science. Successes in ML being used in weather forecasting and protein folding have served as inspiring examples for other sciences, especially where volumes of data are large. Astronomical data are publicly accessible and vast in quantity. In this context, it is only natural that astronomers are increasingly turning to neural methods to enhance the processing and analysis of astronomical data.

We are encouraged by the reviewer’s positive feedback on our dataset size and diversity as well as our careful choice of neural codecs.

We assure the reviewer that our evaluation reflects the SOTA performance in non-neural compression methods for lossless astronomical data compression and that there is recent interest in this field. We address the reviewer’s concerns below, point by point:

One most evident weakness is that the authors evaluate 7 general lossless codec and only 1 astronomical codec (published in 2009). The authors's evaluation is not thorough, many astronomical codecs are omitted.

We received a similar question from Reviewer iEQb, so we point to our response there, under the question “If there also exists some codecs specially designed for astro data, the authors should also provide some evaluation.” In summary, we chose to evaluate RICE due to its superior compression performance and widespread adoption in the astronomy field, particularly compared to alternative astronomy codecs like HCOMPRESS. Since the recent astronomy codec by [Maireles-González, 2023] demonstrated performance similar to JPEG-2000, we chose to evaluate JPEG-2000 because it is well-documented and serves as a representative standard non-neural codec from the Maireles-González paper.

Astronomical data compression is not an active research area, and the only reasonable baseline was published 15 years ago.

While the most representative baseline was published 15 years ago, there has been recent interest in astronomical data compression, for example, the pair of in-depth papers on lossless and lossy astro compression by [Maireles-González, 2023]. We believe there is a strong motivation for renewed interest in this field, given that we are at the cusp of a data explosion “step change” in astronomy. The field is transforming from an era of reasonable data volumes to overwhelming, unmanageable ones.

NASA telescope projects operate on long time horizons. JWST, currently the latest and most advanced space telescope, was planned more than 20 years ago. For this reason, the hardware onboard is very old—only 65 GB of data can be stored. In addition, the CPU onboard is highly limited. Due to the small scale of data and compute onboard space telescopes until now, there has been little capacity for advanced compression codecs.

With that being said, the scale is growing dramatically going forward.

In 2027, NASA will launch the $4.3 billion Nancy Grace Roman Space Telescope. A recent audit of this project published months ago by the Office of the Inspector General (OIG) for NASA [NASA OIG, 2024] emphasizes that the single greatest concern for the Roman project is data transmission issues due to data scale and lack of bandwidth: “Roman is expected to produce an unprecedented amount of data, and there are significant risks with its current plans for transmitting this information back to Earth. The primary network that NASA will use to downlink the data does not have adequate capacity to support Roman’s data requirements.”

Also in 2027, the $3 billion ground-based Square Kilometer Array “is expected to produce 62 exabytes of data each year… this data flood will be transmitted from each telescope at a blisteringly fast 8 terabits per second, signaling the need for extreme data processing.” [Pool, 2020]

These examples underscore the urgent need for new astro compression algorithms. Billion-dollar costs imply that a few percent in improved data compression could mean a gain of hundreds of millions of dollars in value.

---

References:

[NASA OIG, 2024] “Audit of the Nancy Grace Roman Space Telescope Project.” NASA Office of the Inspector General, 31 July 2024, oig.nasa.gov/wp-content/uploads/2024/07/ig-24-014.pdf.

[Pool, 2020] Pool, Rebecca. “Drowning in Data.” SPIE, The International Society for Optics and Photonics, 1 May 2020, spie.org/news/photonics-focus/mayjun-2020/square-kilometer-array-big-data.

We are highly grateful for the reviewer’s time spent carefully examining our paper and its appendix. We address any concerns below:

The only thing which I would have liked to see more discussion on was the point about runtime/compression ratio. Is there any actionable recommendation here? Also given that JPEG-XL is mostly a reference implementation right now and hasn't been optimized while the neural methods require a GPU, is it fair comparison to say that it is slower?

This field of neural scientific data compression is still in its earliest stages. Our actionable recommendation is to prioritize the development of lossless neural methods that consistently outperform traditional codecs in compression ratio, even if they are initially slower and GPU-intensive. Once these methods are mature, the next step would be optimizing them for deployment on specialized hardware, including space-grade systems.

We acknowledge that both our neural methods and JPEG-XL are reference implementations, and neither has been optimized for speed. Therefore, direct runtime comparisons should be interpreted with caution; however, given advances in technology, it is likely that telescopes in the future will be equipped with hardware that supports massively parallel computation and can run neural compression methods. Finally, we point out that compression is also very relevant for ground-based telescopes, which can easily host multi-GPU compute clusters.

We updated our manuscript to reflect these considerations.

Any actionable recommendation around neural/classical compression based on results?

We have the following recommendations and directions that we are extremely excited to work on for future papers. We outline three key directions inspired by our findings and observations:

1. Autoregressive models, especially those built on the successful Transformer architecture, have strong potential as the best neural codec for lossless astronomy data compression. These models are inherently faster at encoding due to parallelizability, but slower at decoding, which fits well within the astronomy context where the data needs to be captured and encoded efficiently (e.g., from a space telescope) but has a high compute/time budget for decoding and analysis (e.g., with supercomputers on Earth). Given that many Transformer models are state-of-the-art likelihood estimators [Child et al., 2019; Roy et al., 2021], such a model is likely to match or even beat our newly added VDM performance.
2. Our results highlight the potential for near-lossless compression to strike a balance between storage efficiency and data fidelity. This approach involves quantizing data into bins to ensure a strict per-pixel maximum reconstruction error, which we anticipate will be acceptable to astronomers for certain use cases.. We have updated the discussion section of our manuscript to reflect this new focus.
3. As shown in Figure 4 of our paper, there are substantial differences in the characteristics of foreground (stars, galaxies, etc.) and background (dark pixels) data. Future methods should exploit these differences by applying distinct compression strategies to each.

Are resource requirements (GPU, memory) factored into speed results?

Given that running-time efficiency results are very much dependent on hardware and the implementation, we only reported the wall clock time (“speed”) to provide a rough comparison between the different methods run on standard hardware. For reference, our neural methods were run on a single RTX 6000 Ada GPU with 48GB of memory, whereas non-neural methods were run on a single Intel(R) Xeon(R) Silver 4112 CPU @ 2.60GHz. We’ve included these configuration details in section 4.6. Although neither the neural methods (which require GPUs) nor JPEG-XL (which requires high-performance CPUs) can be immediately deployable onboard today’s space telescopes, future efforts on optimizing computational efficiency can soon make them feasible.

---

References

[Child et al., 2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.

[Roy et al., 2021] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53–68, 2021.

The cross-dataset evaluation can be improved. In this paper, only brief conclusions about the generalization of different codecs are provided. A deeper evaluation of the similarity between different datasets and the reason for performing differently for different codecs is suggested.

We organize our response in three points:

1. Why are some datasets easier to compress than others?

We completed a new analysis below to help answer this question. As seen in our new Figure 5 in Appendix D, we show that background noise levels, which account for some of the variation between our datasets, are a significant predictor of compressibility. We follow the prior analysis by [Pence, 2009], outlined as follows:

1. Shannon’s source coding theorem establishes that the entropy of a data source defines the lower bound for the optimal bitrate of lossless compression.
2. Most of the pixels in our images are “background” pixels, or not part of an identifiable star or galaxy. To a first approximation, these background pixels are Gaussian noise-dominated.
3. [Pence, 2009] show that if we assume that background pixels are drawn from a Gaussian distribution $\mathcal{N}(\mu, \sigma^2)$, then the corresponding per-pixel entropy in bits is proportional to $\log_2(\sigma)$. We plot this quantity against the bit-rate of JPEG-LS in Figure 5 in our updated manuscript; the figure can also be found here.

While background pixels are not exactly drawn from a Gaussian, and JPEG-LS is not a perfect codec, the linear relationship on this plot suggests that background noise levels significantly influence an image's compressibility. Moreover, Figure 2 in our paper reinforces this insight: as exposure time increases, the number of noise bits in background pixels rises, thereby reducing the image's compression potential.

Hubble data points have the weakest linear relationship, and we verified that Hubble’s background pixels do appear Gaussian, suggesting that Hubble images have more “source” pixels than other datasets that cause the apparent variability. This corresponds well with intuition, as Hubble is known to sometimes take deep, long exposures of dense stellar fields. JWST images are outliers in the upper right corner, as JWST uses both different detector technology and wavelength filters: HgCdTe and infrared, compared to the others’ CMOS and visible light.

2. Why do some train datasets allow our models to generalize better than others?

We refer the reviewer to Table 2 of our manuscript for experimental results on the generalization performance of IDF, where we train and test on all pairwise combinations of our single-frame datasets. The results show that the cross-data generalization performance heavily depends on the training data used, and the performance difference across different datasets can be quite high and will ultimately depend on the statistical characteristics of the datasets. The Keck train set generalizes exceptionally well.

Referring to the scatterplot above (Figure 5 in our updated manuscript): Keck has the highest background pixel variance of all datasets, the widest range of background pixel variances amongst its images, and a very tight linear correlation between background variance and compressibility. From an astronomer’s standpoint, the Keck images are also the most diverse, as they involve several different wavelength bands.

These facts suggest that a neural model might learn through Keck to focus on the background pixels and generalize out-of-distribution due to the diversity present within the dataset. The JWST-2D and Hubble-2D datasets are ~5x and ~50x larger than Keck, suggesting that the stronger generalization performance of training on Keck is not a result of a larger training data size.

3. New experiment on cross-data generalizability

We agree with the reviewer that cross-data evaluation would provide further insight into the behavior of different algorithms on different datasets. To contribute further insight, we conducted a new experiment where we trained IDF on all of the training datasets and evaluated it on each individual dataset. The results in our updated Table 2 show that this model generalizes best, beating the performance of Hubble and SDSS in-domain trained models and nearly matching the others. This suggests potential for a more practical telescope-agnostic neural codec for astronomy data in the future.

---

References

[Pence, 2009] Pence, W. D., Robert Seaman, and R. L. White. "Lossless astronomical image compression and the effects of noise." Publications of the Astronomical Society of the Pacific 121.878 (2009): 414.

We sincerely appreciate the reviewer’s positive feedback on our dataset’s ease of accessibility, thorough evaluation of compression methods, and constructive criticism. We are happy to address these below, point by point.

Since many of the datasets are from publicly accessible resources, therefore I suggest that the authors should contribute more over the analysis and deeper evaluation of the re-organized AstroCompress dataset.

While the raw data is publicly available, our work follows in the tradition of landmark datasets like ImageNet by carefully identifying, curating, and transforming complex astronomical images into a cohesive, accessible format for the machine learning community. Data collected by scientific instruments in its most raw form, as is the case for our datasets, is often highly complex and requires domain knowledge to standardize. We outline some of these considerations below:

1. Astronomical image collection is full of complex parameters and data types, with each survey employing unique imaging patterns over the sky and across time. Writing AstroSQL and the relevant APIs to compile data is nontrivial, partially due to a lack of documentation. Some examples: The SDSS telescope has collected dozens of images of the same patches of sky at different points in time, and it requires extensive processing to find these overlapping images, align them according to star positions, and compile neatly packaged multispectral and temporal data cubes. The Keck survey has changed its data format over time [Keck, 2009], and we have written a script in HuggingFace to standardize it into a single format. JWST contains several kinds of “detector readout patterns,” and we have sub-selected the most relevant ones for science [JWST NIRCam Detector Readout Patterns, 2016]

2. Telescope optics brim with a variety of detector technologies, wavelength filters, pupils, and other choices that capture different celestial structures. Domain knowledge was critical in selecting these parameters with the correct scope, such that the ML models would be sufficiently general for practical use but not so general that model performance suffers due to edge cases. For example, we carefully selected a specific range of exposure times on the Hubble and JWST datasets to avoid undesirable artifacts of “saturated pixels,” where the CCD detector gets maxed out. An example illustrating optical parameters that had to be selected for our data can be readily seen on this JWST documentation page: [JWST NIRCam Pupil and Filter Wheels, 2016] More details can be found in our “utils/*downloading” files, found in each HuggingFace dataset.

3. By design, telescopes often capture overlapping images of nearby regions of the sky. It is useful to treat these as different training data points because variation still exists between such images. However, for our 2D single-frame datasets, this presents the risk of an image in the train set overlapping on the sky with an image in the test set. To avoid this train-test contamination, we developed a robust preprocessing approach to ensure disjoint sky footprints across the train/test splits. This was done by clustering data points by their sky coordinates and assigning each cluster to either train or test, minimizing the amount of data we were forced to discard. This algorithm is outlined in Appendix A.4 and in the HuggingFace “utils/*filtering.ipynb” files of each dataset.

All of our code that processes the raw, messy data from public sources into accessible code is on HuggingFace. To get a better impression of the complexity, we encourage the reviewers to view this code in the utils folders and <dataset_name>.py files of each dataset; we believe the scale of domain knowledge and preprocessing will be readily apparent.

---

References:

[JWST NIRCam Detector Readout Patterns, 2016] “NIRCam Detector Readout Patterns.” JWST User Documentation, 2016, jwst-docs.stsci.edu/jwst-near-infrared-camera/nircam-instrumentation/nircam-detector-overview/nircam-detector-readout-patterns#gsc.tab=0.

[JWST NIRCam Pupil and Filter Wheels, 2016] “NIRCam Pupil and Filter Wheels.” JWST User Documentation, Space Telescope Science Institute, 2016, jwst-docs.stsci.edu/jwst-near-infrared-camera/nircam-instrumentation/nircam-pupil-and-filter-wheels#gsc.tab=0.

[Keck, 2009] “LRIS Data Format.” W.M. Keck Observatory, University of Hawaii, 2009, www2.keck.hawaii.edu/koa/public/lris/lris_data_form.html.