Alberta Wells Dataset: Pinpointing Oil and Gas Wells from Satellite Imagery

Thank you for your careful reading of our paper and the very helpful suggestions. We have attempted to address your comments, including adding significant new experiments regarding the effect of well type and the near-infrared band, as well as the ConvNeXt architecture. Detailed responses are below.

1. There are minor suggestions for improvement in the phrasing. For example, line 431 states, "For the binary segmentation task framing, we train both CNN-based and Transformer-based backbones, considering the prevalent imbalance in the image data due to the small size of wells." This description is somewhat vague. If it implies that the backbones were pre-trained, it should be clearly stated which datasets were used for this pre-training. In Section 4, BENCHMARK EXPERIMENTS, it might not be clear to the reader whether the experiments used only the RGB three channels or included the near-infrared to make four channels. Although Section 3 states that the dataset provides four channels, repeating this in the experimental setup could be beneficial.

Thank you for this feedback. Regarding the phrasing in line 431, we appreciate your observation and would like to clarify that the backbones (both CNN-based and Transformer-based) were trained from scratch on our Alberta Wells Dataset, and no pre-training from external datasets was used. Although we did use 3-dimensional, ImageNet initialized weights of the backbone for encoder but modified the initial layers afterwards to support 4-dimensional multispectral images. We have updated the manuscript to explicitly state this.

Regarding the use of RGB vs. RGB+NIR in the benchmark experiments, we confirm that all models were trained using both RGB and Near-Infrared (NIR) channels, not just RGB. We have revised Section 4 (Benchmark Experiments) to make this clear, ensuring that the experimental setup explicitly mentions the use of RGB+NIR data.

2.Tables 4 and 5 display the benchmark performances of various classical semantic segmentation and object detection architectures on this dataset. While this work does not provide experiments with different model sizes (such as small, base, large versions), this might be sufficient for this dataset. However, including the parameter count and computational load of each benchmark model in the tables would add value to the discussion of segmentation and detection results.

Thank you for this excellent suggestion. We agree that including the parameter count and computational load for each benchmark model would add value to the discussion of the segmentation and detection results in Tables 4 and 5, and we have added this in the revision to compare the various different models we used, e.g. different sizes of ResNet architectures and Transformer-based models.

3. The state-of-the-art backbone, ConvNeXt, was omitted in the experiments. Providing its performance in remote sensing image analysis would make the article's benchmark presentation more comprehensive.

This too is an excellent suggestion. We have included initial results for ConvNeXt in the Appendix of the revision, and are currently running additional experiments.

4. In Figure 4, the satellite images reveal that wells may not always be perfectly circular, yet the semantic segmentation annotations are uniformly circular, and the object detection bounding boxes are uniformly square. This may not accurately reflect the real shape of wells, but the article lacks an explanation for this, except for a mention on line 335 about the "teardrop shape typical for well sites." The accuracy of these annotations might be the primary limitation of this dataset.

Thank you for this feedback. We agree that the annotations in Figure 4 (circular segmentation masks and square bounding boxes) may not capture the exact shape of some wells. However, the goal of both the object detection and segmentation tasks is ultimately to pinpoint the locations of wells, not their exact shape - these are weak labels that enable training. While this may lead to a slightly increased reported loss, we do not expect this to significantly affect the overall performance of the models. Deep learning algorithms are known to be somewhat robust to moderate amounts of label noise, as shown in previous work (e.g., Rolnick et al., 2017). We will include a clearer explanation of this issue in the revised manuscript.

Thank you for the detailed and thoughtful responses during the rebuttal process, as well as the additional experiments conducted. Your clarifications and revisions have effectively addressed my concerns and significantly enhanced the quality of the manuscript. Based on your replies and updates, I am pleased to maintain my rating.

Thank you for your careful reading of our paper and the very helpful suggestions. We have attempted to address your comments, including adding significant new experiments regarding the effect of well type and the near-infrared band. Detailed responses are below.

1. Despite the high data volume, the study area is limited to Alberta, Canada. In comparison, an existing dataset (NEPU-OWOD V3) features sites across China and California, US. In general, I argue that adding more data for a specific region (although I also acknowledge that Alberta is a large region) is less important than including oil and gas wells from a diverse set of geographic locations to test the generalization ability of models. The importance of this (specifically the robustness of models to geographic distribution shifts) is also one of the key points in Rolf et al. (2024) which the paper cites.

Thank you for your feedback. We agree completely that the global generalization capability of remote sensing models is an important issue - however, we believe this need not be the only priority for a benchmark dataset. Our results show that well detection in Alberta already presents a meaningful challenge for remote sensing models. Alberta, within itself, encompasses a wide range of landscapes, including prairies, forests, lakes, and mountains. Furthermore, Alberta has the third largest oil reserves in the world, so by itself it represents a key use case for well detection, even if algorithms were to be applied nowhere else.

While NEPU-OWOD V3, as mentioned, does encompass such large-scale geographic diversity, it contains only 722 well images with 3.7k wells. By contrast, our dataset includes over 213k wells. This makes it the first benchmark dataset suitable for training state-of-the-art deep learning models from scratch on the problem of well detection.

2. Although the well type data is a key contribution of the paper, the authors did not include any experiments to demonstrate its usefulness. Furthermore, the dataset splitting does not seem to take well types into account. I suggest adding well type distributions across the sets to Figure 1.

Thank you for your feedback. We completely agree that demonstrating the impact of well type data is important Regarding dataset splitting, we have added plots showing the well type distributions across the dataset splits in Section D4 (Dataset Size & Distribution of Samples) of the appendix. We did not explicitly take well types into account in our splitting algorithm because the distributions of abandoned and suspended wells are very similar to those of active wells. Indeed, in the new figure one can see that the frequencies of active, suspended, and abandoned wells are similar across splits. Regarding the effect of training on different well types, we have conducted initial experiments comparing the impact of training models on data that includes all well types versus only active wells. The preliminary results show that training on a mix of well types is indeed beneficial in performance when evaluated across the whole dataset. We are currently running more detailed experiments on this topic for the camera-ready.

3. The authors state that existing datasets predominantly contain active wells, which limits their applicability in the context of identifying abandoned or suspended wells (l86), while also admitting that the physical appearance of different well types is very similar (l133). The latter would mean that models trained on only active wells generalize to all well types, which contradicts the former. Please clarify this (ideally with experimental results).

Thank you for this question. Our explanation wasn’t clear, and we have attempted to improve it and added additional experiments. The key point here is that while the physical structure of active, suspended, and abandoned wells can be quite similar, the environmental context surrounding abandoned and suspended wells can differ significantly from active ones, depending on the time at which the well was abandoned. Abandoned wells undergo degradation over time (e.g., vegetation growth, land subsidence, and surface changes), which can make them more challenging to detect, even though their basic infrastructure might appear similar to active wells. Thus, training on suspended and abandoned wells is important if models are to be used in detecting them.

In the revision, we have added preliminary experiments to compare the performance of algorithms in predicting abandoned wells when trained using only active wells versus a mix of active and abandoned wells, which show that the latter is indeed beneficial. We hope this explanation clarifies the apparent contradiction, and we plan to include further experimental results in the camera-ready version. Thank you again for your thoughtful feedback.

Thank you for your thoughtful follow-up. We very much appreciate your recognition of our additional experiments and revisions.

In response to your comments, we would highlight that our dataset is more than 100 times larger in image count than the nearest comparable datasets such as NEPU-OWOD V3, and contains over 50 times the number of wells. For this reason, we believe that our benchmark makes it uniquely possible to test algorithms that require large datasets. Additionally, our dataset includes a well-distributed set of non-well images, which is lacking from many existing datasets.

Indeed, we do not claim that a model trained purely on Alberta data would necessarily be able to generalize zero-shot to other regions or landscapes. Rather, since this is a benchmark dataset, the goals are threefold: (i) to provide a testbed for algorithms on the important task of well-detection - Alberta makes it possible to identify algorithms that will be able to train on and perform well across diverse landscapes, but as in other tasks these algorithms would likely still need to be at least fine-tuned within any region of application, (ii) to represent an immediately impactful use case, since detecting wells within Alberta is itself useful, (iii) to provide value to the field of ML with a challenging task.

We obtained per-class performance metrics for specific well type in the Binary Segmentation Task (Model Architecture : UNet; Backbone : ResNet50) . As observed, the model's performance aligns closely with the full test set results presented in Appendix A.3. These results highlight the importance of including all well types (active, suspended, and abandoned) in the training dataset to ensure comprehensive detection across diverse scenarios. Training solely on active wells results in significantly poorer performance when applied to individual well types and mixed well-type test sets (IoU ~0.5), indicating limited generalization capability. In contrast, training on all well types enhances the model's ability to handle real-world variability, as evidenced by improved IoU, F1 score, and recall for each well type and the entire test set. The improved recall reflects the model's capability to detect a broader range of wells, which is critical for environmental monitoring, where overlooking even a single abandoned well could lead to unaddressed methane emissions or groundwater contamination. While precision is higher for the model trained exclusively on active wells, this is offset by its low recall, which limits its effectiveness in detecting all wells in an image. Therefore, incorporating all well types into the training process ensures a balanced, reliable performance, enabling accurate detection and classification across a wide range of realistic scenarios.

Thanks for pointing out the minor stylistic inconsistencies regarding decimal places, we have now corrected this point.

Thank you for your thorough and thoughtful review. We appreciate your recognition of the strengths of our work, including the dataset construction, rigorous benchmarking, and the explanation of limitations. Below, we respond to the concerns and questions you raised to provide additional clarity. We have also added additional experiments in the revision to support our conclusions on the benefit of near-infrared imagery and the advantage of training on data from multiple well types.

1. The area is limited to Alberta, Canada, which makes the dataset lose universality.

We agree that it is a limitation that our dataset is not even larger. However, previous datasets on this topic are limited to much narrower locations, typically with an average of around 5,000 wells, while our dataset covers the entire province of Alberta (an area larger than the UK and Germany combined), encompassing a diverse range of geographical zones with over 217,000 oil wells, by far the largest of any benchmark dataset released on this problem anywhere in the world.. By training machine learning models on this large-scale dataset, we can enhance their robustness and improve their ability to generalize to similar, less-studied regions with fewer labeled data points. Furthermore, Alberta has the third largest oil reserves in the world, so by itself it represents a very impactful use case, even if the algorithms are applied nowhere else.

2. The resolution scale is limited to 3 meters due to the use of PlanetScope-4-Band imagery.

While the 3-meter resolution of PlanetScope imagery is a limitation, we believe this resolution strikes a balance between coverage and detail at a relatively low cost, representing a practical choice for large-scale monitoring.

Additionally, PlanetScope data offers some unique advantages: its satellite constellation is updated daily, allowing for consistent imagery across the dataset. Moreover, PlanetScope provides multispectral imagery (RGB + Near-Infrared), which is valuable for identifying features such as well pads and depressions, which may be difficult to detect using standard RGB images. (We have added experiments in the revision that show the benefits of using the near-infrared channel.)

3. The baseline algorithms do not utilize some physical information about wells.

We would like to clarify what is meant by "physical information" in this context. In the current baseline evaluation, we focused on standard deep learning approaches for object detection and segmentation, which primarily rely on image data and do not incorporate additional physical features of the wells (such as well pads, tanks, or surrounding infrastructure). These features, while valuable, are often challenging to extract automatically from imagery without expert input or additional data sources, and can vary considerably between individual wells.

4. What’s the relation of segmentation and detection for wells analysis?

We offer both object detection and segmentation framings of the task of pinpointing wells, since it is not clear which approach may be most effective. Ultimately, the goal of both tasks is to understand where wells are, a goal for which object detection is the standard approach. However, the generally stereotyped shape of wells makes it possible also to train a segmentation algorithm on well locations; therefore, we have presented this task framing as well.

5. Why is the splitting principle using the geographical location?

We split the training, validation, and evaluation data geographically to reduce the risk of spatial autocorrelation, which could cause performance to be overestimated, as well as to ensure the model can generalize effectively to new, unseen regions, reflecting real-world scenarios where well detection models are applied to areas beyond the training data.

Our approach to dataset splitting ensures that the training and test sets are balanced, geographically diverse, and yet non-overlapping. We developed a custom splitting algorithm that clusters data into non-overlapping groups (k1), which are subsets of larger groups (k2). In this setup, one member from each smaller group (k1) is included in both the test and validation sets, while the rest are part of the training set. This approach helps maintain diversity and reduces the risk of data leakage.

We would like to express our sincere gratitude for your thoughtful review of our submission. We appreciate your feedback and would like to address the concerns you raised. Below is a detailed response to each of the issues raised in your review. We have also added additional experiments in the revision to support our conclusions on the benefit of near-infrared imagery and the advantage of training on data from multiple well types.

1. The actual contribution of the work is not clearly established. The ground truth dataset is taken from AER ST37, where all geolocations are available. The dataset contains only that data; no new data is contributed here.

Indeed, as with many benchmark datasets, we did not gather new field data ourselves - instead the value of data-gathering comes from two factors: It is highly unlikely that someone from the ICLR community would be able to understand, much less, use raw data from AER ST37. Besides the errors in the database which must be corrected, it is simply not in a form that a machine learning researcher could parse - it is not formatted for code and is encoded using jargon specific to oil well operators. AER also of course does not include satellite imagery - our release of PlanetScope imagery is owing to a specific agreement with Planet Labs - this imagery would not be made publicly available at all were it not for our work. While our dataset is based on the AER data, we have performed extensive quality control, curation and enhancement of this data, in collaboration with domain experts.

Previous datasets have been introduced for this problem which were orders of magnitude smaller than ours. This goes to show the effort involved in creating such a benchmark dataset - if it were not hard, these prior works would likely have been significantly more in-depth.

2. The model evaluation lacks in-region and out-of-region performance testing, which is crucial for remote sensing tasks.

Thank you for this question. We believe that the geography-based splitting approach we adopted is an effective method for evaluating geographical generalization. Our approach ensures that both the training and test sets cover a broad spectrum of Alberta's landscape, balancing the dataset to capture diverse geographical features, including prairies in the east, mountains and forests in the west. This strategy maintains geographical diversity and mimics real-world conditions.

Our results show that well detection in Alberta already presents a meaningful challenge for remote sensing models, irrespective of generalization outside the region. While future work may wish to generalize our dataset to other regions, there is clearly room for improvement on state-of-the-art within Alberta. Furthermore, with the third largest oil reserves in the world, Alberta by itself it represents a key use case for well detection, even if algorithms were to be applied nowhere else.

3. The figure quality is very poor, especially Figures 3 and 4.

Thank you for your valuable feedback regarding the quality of the figures. We have replaced Figure 3 with an updated version and have enhanced the quality of Figure 4, which was due to a compression artifact in conversion from high-resolution remote-sensing images.

4. For object detection, only axis-aligned models are used. The authors have not implemented recent models or an oriented bounding box (OBB) approach, such as YOLO-OBB or Faster R-CNN with Rotated Bounding Boxes, which could potentially enhance performance.

Thank you for your feedback. We agree these approaches could be promising. Since experiments will require significant additional dataset engineering, we have not been able to complete these during the rebuttal period but can look at adding them for the camera-ready version.

5. The dataset documentation follows the NeurIPS 2024 format and includes a footnote stating: "Submitted to the 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks," which violates the rules of the conference.

We apologize for the oversight and any confusion it may have caused. The footnote in the supplementary material on Dropbox referencing the code and data was included by mistake, and we have removed it to ensure full compliance with ICLR submission guidelines. The dataset documentation in the supplementary materials on Dropbox will be updated accordingly.

Although the necessary changes, including plans for hosting the dataset and other updates, were made and included in the appendix of the main manuscript at the time of submission, we have ensured that these updates are clearly reflected in the revised version available on Dropbox as well.

Thank you sincerely for providing comprehensive responses. I appreciate that some of my concerns have been addressed. As articulated in the authors' rebuttal, their focus lies within the dataset and benchmarking subdomain of ICLR, which aligns well with the scope of an ICLR subject area dataset and benchmark. I upscale my rating from 3 to 5.