AirPhyNet: Harnessing Physics-Guided Neural Networks for Air Quality Prediction

4. Cross city Knowledge Transfer: We appreciate your insightful question, which could be a valuable extension to our model. Training our model in one city and applying it to another city which can be referred to as cross-city generalization. It is possible to implement taking into account certain considerations such as:

   • Spatial Differences: Different cities have different spatial characteristics, such as road layouts, building distributions, and overall urban structures.
   • Different Sensor Networks: Sensor networks in different cities may use different types of sensors, have varying sensor densities.

   With these considerations we can learn a similarity function among the sensor networks of each city considering spatial features. Then the model trained on a source city can be finetuned based on this similarity function and data from the target city. Since the physical processes are invariant across cities, no specific adjustments are needed inside the existing model architecture. Similar approaches have been proposed in the traffic prediction domain: Wang et al.,2018. We included this idea as a plausible future work in our Conclusion and Future work section of the revised manusctipt. Thank you for pointing this out.

5. Exploratory Analysis of Data: Both datasets used in this study are periodic in nature and has around 6.5% and 1% missing values in Beijing and Shenzhen data, respectively. These data also include sudden changes in air quality concentrations. Strong spatial correlations exist in both the datasets. Corresponding timeseries plots and correlation plots can be found here.

6. Effect of Wind Data: As stated under Appendix 5 in our manuscript, all the baseline models were trained both with and without wind-related attributes. However, with the exception of PM2.5-GNN, the rest of the baseline models exhibited degraded performance when wind-related attributes were incorporated in the respective models. Therefore, the reported baseline performances do not include wind-related attributes, except for PM2.5-GNN.

   The proposed model without wind data is similar to the DiffOnly Model which do not consider the advection component as illustrated in the Ablation study. DiffOnly model exhibits a slightly degraded performance to our overall model. The results are included in the Ablation study in Section 3.4.

   If more covariates are considered, they can be used as auxiliary features in the model. Additional covariates were not considered in our experiments, since they are not essential to model the physical processes. We tried to achieve optimal performance with less data since most of the weather data are not often available at the monitoring station level. This can be considered one of the advantages of our model.

7. Implementation Details of HA and VAR: Further implementation details of these models are included in Appendix (A.5) of the revised manuscript.

8. AirPhyNet Comparison with CTM: We appreciate your observation regarding the absence of a direct comparison with Chemical Transport Models (CTMs). Widely used CTMs such as CMAQ (Community Multiscale Air Quality) and WRF-Chem (Weather Research and Forecasting with Chemistry) simulate wide range of atmospheric dynamics in addition to diffusion and advection which demands for multiple input variable including land use data, boundary conditions, meteorological parameters like boundary layer height. Furthermore, these models require sophisticated parameter calibrations and higher computational resources. Due to these concerns CTMs were not used in this study for analysis. Instead, we introduce AirPhyNet as a potential solution for the aforementioned challenges.

Thank you for the insightful review. We appreciate the acknowledgment of our novel architecture and strong results. In response to your feedback, the noted concerns are addressed below.

Addressing Weaknesses:

1. Model Scalability: We acknowledge your concern about the relatively smaller number of stations considered in our experiments. The primary reason for conducting experiments in a limited number of stations is the unavailability of comprehensive public datasets at the monitoring station level. We explored multiple available datasets (such as the KnowAir Dataset) where data is available at a coarser granularity. However, given our emphasis on fine-grained predictions at the monitoring station level, we opted the current datasets which have also been used in majority of the similar work below.
   • Du, S., Li, T., Yang, Y., & Horng, S. J. (2019). Deep air quality forecasting using hybrid deep learning framework. IEEE Transactions on Knowledge and Data Engineering, 33(6), 2412-2424.
   • Wang, C., Zhu, Y., Zang, T., Liu, H., & Yu, J. (2021, March). Modeling inter-station relationships with attentive temporal graph convolutional network for air quality prediction. In Proceedings of the 14th ACM international conference on web search and data mining (pp. 616-634).

We also conducted some experiments on a sample dataset provided by Airformer Model upto 150 nodes. We vary the number of nodes and measure the training time and inference time. The results shown here exhibit the training and inference time of our model as the number of nodes increase. However, we assumed airformer model's adjacency matrix weights include distance information since the explicit location details are not available for this dataset. We hope that the generated trend line could be insightful to reflect the scalability of our model despite of the limitations and assumption. We acknowledge the importance of extending our experiments to a broader geographical scope and intend to explore collaborations and avenues to acquire additional datasets for future research.

Weaknesses 2 and 3 are addressed below.

Addressing Questions:

1. Decoder: We acknowledge the concern with regard to the decoder architecture. Accordingly, we clarify the decoder module under the last paragraph of Section 3.4 in the revised manuscript as follows.

   Finally, the decoder module uses the output from the GNN-based DE Network representing the latent space, to generate predictions. It employs reshaping and aggregation steps to produce the desired output format with the correct dimensionality at each time step. AirPhyNet is trained through backpropagation by minimizing the Mean Absolute Error (MAE) (see A.4) between predicted values and ground truth values.

2. Significance of the encoded state $z$ : The significance of this encoded state lies in its ability to capture and represent the temporal dependencies in the input data. Direct visualization of z might be challenging as it is in a higher dimensional space. However, it can be transformed into a lower dimension for visualization and used to gain insights on the learned temporal representations.

   It is possible to apply Neural ODEs directly to the original data without an intermediate encoding step. However it might result in a more complex and computationally demanding model, as the network has to learn to capture critical temporal dynamics without an encoder.

3. Impact of the Lookback window: As shown in the LookbackWindow visualization, the error reduces as the lookback window increases, since a wider temporal range is captured to make the predictions. As stated in Section 4.1, similar to the prior studies (Wang et al., 2020b; Liang et al., 2023) we use past 72-hours as the lookback window.

Please find the clarifications for the remaining questions in the next response window due to the character limitation per response.

Dear Reviewer tKfZ,

We are happy that our responses have effectively addressed your concerns. We sincerely appreciate the time and effort you have dedicated to reviewing our work. Especially your suggestion extending our methodology for cross city transfer learning is indeed a promising avenue for future research. It helped us to broaden our future research directions.

As we approach the end of the author/reviewer discussions within the next few hours, we kindly request your reconsideration of the score and the associated confidence level in light of the responses and revisions we have provided.

Thank you so much for devoting time to improving our work!

Thank you for the insightful and constructive feedback. We appreciate your positive remarks on AirPhyNet's innovation and experimental demonstration.

Addressing Weaknesses:

1. Clarification on Diffusion and Advection: We understand your concern about the clarity of the distinction between diffusion and advection. As you’ve stated, both these processes describe the transport of air pollutants over space and time. The distinction lies in the transport mechanism. Diffusion explains transport of particles among locations due to the difference in concentration gradient while the advection explains the transport of particles due to the effect of an external flow field like wind. In the revised manuscript, we added a clearer distinction in section 2.2 as follows:

In contrast to the diffusion process which explains the transport of particles due to the difference in concentration gradients, advection explains the transport of particles due to the effect of an external flow field.

2. Reparameterization Trick: We appreciate your insightful question regarding the adoption of the reparameterization trick. The main motivation to use this trick arises from the training and generative capabilities of Variational Autoencoder (VAE) framework. The use of the reparameterization trick in our model architecture allows for the introduction of stochasticity into the encoding process. Directly using the final hidden state of GRU as ${z_{t_0}}$ , would result in in a deterministic representation, limiting model's ability to capture the inherent uncertainty in the latent space. By introducing the reparameterization trick, we enable the model to sample from a distribution defined by mean and standard deviation, providing a stochastic element to the latent representation.

   In summary, this trick is useful in enhancing model’s robustness and flexibility as it enables to account for variability in the temporal patterns of PM2.5 concentration, acknowledging that a single deterministic hidden state might not fully capture the complexity and uncertainties.

3. Decoder and Loss Function: Based on your concern, we clarify the decoder module and loss function under section 3.4 of the revised manuscript as follows. The formulation of the loss functions is included in the appendix due to the constraints of the page limit.

Finally, the decoder module uses the output from the GNN-based DE Network representing the latent space, to generate predictions.It employs reshaping and aggregation steps to produce the desired output format with the correct dimensionality at each time step. AirPhyNet is trained through backpropagation by minimizing the Mean Absolute Error (MAE) (see A.4) between predicted values and ground truth values

4. Advantages of Integrating Physical Principles: We acknowledge the importance of addressing the unique contributions of our approach in comparison to existing forecasting methods, such as DCRNN and Airformer. As you’ve stated principles like Diffusion is reflected in the DCRNN model even though it is not explicitly utilized as a physics principle, but Airformer model does not incorporate these processes as it uses an attention-based transformer architecture which does not utilize a Graph Neural Network structure. Furthermore, we would like to highlight the following major advantages of injecting physical principles into machine learning models, as opposed to data driven methods like DCRNN:

   • Enhanced Interpretability: By incorporating a physics-informed framework, AirPhyNet provides enhanced interpretability in its predictions. The physical principles guide the learning process, allowing for a more transparent understanding of the underlying mechanisms influencing air quality dynamics.
   • Improved Generalizability: Physics-informed models are designed to capture underlying physical processes, which can improve the generalization capability of the model to unforeseen conditions or locations. While data-driven models like DCRNN and AirFormer rely heavily on available training data, our approach leverages physics-based priors to enhance the model's ability to make accurate predictions in diverse scenarios.
   • Domain Knowledge Integration: Injecting physical principles allows us to integrate domain knowledge into the learning process. This incorporation of expert knowledge ensures that the model adheres to the fundamental laws governing air quality dynamics.
   • Consistency with Physical Laws: The physics-informed approach ensures that the generated forecasts are consistent with known physical laws governing the movement and dispersion of air pollutants. This consistency enhances the reliability of predictions, especially in situations where traditional machine learning models may struggle due to their lack of adherence to these fundamental principles.

5. Typos: Thank you for pointing this out. We reviewed and fixed these typos in the revised manuscript.

Thank you for your thoughtful review. We appreciate your feedback and have addressed your suggestions and concerns below.

Addressing Weaknesses

1. Research Question specification: We would like to draw your attention to Section 2.1 where we have specified our research question. This section and specifically Equation 1, elaborates on the air quality prediction problem with the corresponding notations.We also specify the rationale behind the air quality prediction problem in section 1, highlighting the limitations of existing techniques. We further discuss the potential of integration of physics dynamics into deep learning networks to enhance the interpretability and predictive accuracy of the problem.

2. Related Work Discussion: We would like to clarify that in our current manuscript structure, a brief overview of related work is presented in the Introduction section. We provide a comprehensive discussion of related work in a dedicated section later in the manuscript. To address your concern, we have moved the Related Work section after the Introduction and before the Methodology in our revised manuscript.

3. Citations: Regarding your concern on the citations, we accept the fact that some of the citations, specially under “Air Quality Prediction” subsection are outdated. That is because we have included the progressive development of techniques within this domain from initial physics-based methods to the current state-of-the art data driven methods. All these methods are within our technical scope.

   We appreciate your suggestions on some up-to-date related work, and we have included two of them (Su et al., 2023, Wang et al., 2022) together with some new citations under related work of the revised manuscript. However, we would like to emphasize that the other suggested works are out of the scope of this work. The first two works (Nilesh et al, 2022; Mondal et al 2023), involve image processing methods for Air Quality Prediction which is distinct from our study which focuses on leveraging physics-guided deep learning for forecasting tasks, specifically tailored to monitoring station data. The last work (Lin et al, 2023) involves anomaly detection, which differs from our primary focus on air quality forecasting task.

Weaknesses 4 and 5 are addressed below.

Addressing Questions

1. Spacing Issues: We carefully reviewed the manuscript and adjusted the minor spacing issues present. Thank you for pointing this out.

2. Abbreviation: The full form of Differential Equation (DE) is introduced in section 2.3 under the Diffusion-Advection Differential Equation Function, when it is first mentioned in the manuscript to enhance clarity for readers.

3. Section Order: We have moved the Related Work section after the Introduction and before the Methodology in our revised manuscript.

4. Discussion Section: We appreciate your suggestion on the discussion section, and it has been included in the revised manuscript in the Appendix (A.7) due to the constraints on the page limit. The discussion section includes summary of results, implications of the findings in general, limitations and few recommendations for further enhancement.

5. Limitations: Limitations are included within the discussion section in the revised manuscript as given below. We also highlighted these limitations within section 5 (Conclusion and Future Work). A seperate subsection was not included due to the constraints on the page limit.

While the results are promising, it is vital to acknowledge certain limitations in this study. Our focus solely on PM2.5 concentration, while significant, may not account for the complete spectrum of pollutants that contribute to overall air pollution. There exist pollutants with diverse chemical properties and reactivity patterns which might exhibit behaviors that are not captured by our model. Therefore, the applicability of our method to highly chemical-reacting pollutants remains uncertain. Additionally, our deep learning architecture emphasizes the incorporation of physical processes, limiting its suitability for pollutants heavily influenced by chemical reactions.This could potentially hinder the model’s accuracy in scenarios dominated by chemical reactions rather than physical transport.

Please note that the revised manuscript is uploaded and all the revisions are indicated in red font color.

Dear Reviewer 6p5Q,

We sincerely appreciate the time and effort you have dedicated to reviewing our work. We have addressed the your questions and concerns above. Given that the end of author/reviewer discussions is just in few hours, this is to kindly ask for your reconsideration of the score. We would really appreciate it if our next round of communication could leave time for us to resolve any of your remaining or new questions.

Thank you so much for devoting time to improving our work!

Dear Reviewer 6p5Q,

Thank you for your feedback and engagement with our paper. We're pleased to hear that our detailed responses to your questions and comments have contributed to a positive impression. Your updated rating of 6 is much appreciated, and we are grateful for your constructive input.

Best Regards, Authors

Addressing concerns on Contributions:

We appreciate your concern about the perceived lack of contribution to hybrid AI or AI in general. Allow us to clarify the technical gaps and the specific contributions our work brings to the field to bridge these gaps. As highlighted in Section 1, Paragraph 2, the existing deep learning models including hybrid AI models impose technical challenges on interpretability, generalizability and require ample amount of data for precise predictions. Thus, as opposed to the existing hybrid data driven approaches, we introduce synthesis of physics dynamics and deep learning methods for air quality prediction which is a non-trivial task. Thereby some of our unique contributions are as follows:

• Integration of Physics Principles with Deep Learning to develop a comprehensive architecture:

  • The incorporation of physics priors within a deep learning architecture for air quality predictions, as presented in our paper, involves more than just introducing new equations or fusing exisitng architectures. It involves tailoring and leveraging existing deep learning methodologies to represent physics dynamics which we believe is a significant contribution.
  • We draw inspiration from existing literature on physic-based (Section 3.2), define these physics dynamics of Air Pollutant transport on a sensor network (Section 3.3) and seamlessly integrate them into a deep learning architecture with custom designed adjacency matrices and neural network layers(Section 3.4).
  • Even though, there exist methods which use these physical priors for air quality prediction (such as CTM), they have not been explicitly combined with deep learning architectures. Our application and evaluation of these concepts in a real-world context represents a novel and challenging endeavour.

• Leveraging the model with Interpretability: The interpretability of deep learning models in diverse urban problems such as air quality prediction is a significant technical challenge that has not been adequately addressed. Our model, AirPhyNet not only enhances the predictive performance but also provides forecasts which are interpretable based on physics dynamics, thus contributing to a better understanding of the model’s decision-making process.

With the aforementioned contributions and technical challenges addressed, we believe our work aligns well with the goals and interests of the ICLR research track.

Addressing concerns on Experimentation:

The primary reason for conducting experiments in a limited number of cities is the unavailability of comprehensive public datasets. As you pointed out Liang et al. (2023), whose work we cited in our experimental setup utilized their own dataset covering mainland China. Only 20 instances of each training, validation and test sets of this dataset are made accessible which would not yield comparable and meaningful results.

We would also like to draw your attention to the KnowAir Dataset which is another publicly available dataset we explored but did not use in our analysis. This dataset provides air quality and meteorological data at the city level. Given our emphasis on fine-grained predictions at the monitoring station level, we opted the Beijing and Shenzhen Datasets which have also been used in majority of the past work in the literature*. However, we acknowledge the importance of extending our experiments to a broader geographical scope and intend to explore collaborations and avenues to acquire additional datasets for future research.

*Given below are some of the latest works in literature which have used the Beijing and/or Shenzhen Datasets:

• Du, S., Li, T., Yang, Y., & Horng, S. J. (2019). Deep air quality forecasting using hybrid deep learning framework. IEEE Transactions on Knowledge and Data Engineering, 33(6), 2412-2424.
  • Conducted experiments on two datasets from Beijing covering 1 station and 36 stations respectively.
• Han, J., Liu, H., Zhu, H., Xiong, H., & Dou, D. (2021, May). Joint air quality and weather prediction based on multi-adversarial spatiotemporal networks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 5, pp. 4081-4089).
• Wang, C., Zhu, Y., Zang, T., Liu, H., & Yu, J. (2021, March). Modeling inter-station relationships with attentive temporal graph convolutional network for air quality prediction. In Proceedings of the 14th ACM international conference on web search and data mining (pp. 616-634).
  • Conducted experiments on two datasets from Beijing and Tianjin covering 35 stations and 26 stations respectively.
• Han, J., Liu, H., Zhu, H., & Xiong, H. (2023). Kill Two Birds with One Stone: A Multi-View Multi-Adversarial Learning Approach for Joint Air Quality and Weather Prediction. IEEE Transactions on Knowledge and Data Engineering.

Thanks for the thorough response. I haven't gone through it in detail yet, and will do so later. Then I will definitely reconsider my score.

If I may ask one last clarification question: Can you please point me to where in the case study has been validated your claim "our method enhances interpretability by physics-informed feature representations"?

I agree with the general intuition that physics-informed representations will help with more interpretable predictions. However, I do not see this potential being used in your experiments and/or case study. Maybe I missed it. Thanks!

Dear Reviewer pK4K,

We sincerely appreciate the time and effort you have dedicated to reviewing our work. We have addressed the your questions and concerns above. Given that the end of author/reviewer discussions is just in few hours, this is to kindly ask for your reconsideration of the score. We would really appreciate it if our next round of communication could leave time for us to resolve any of your remaining or new questions.

Thank you so much for devoting time to improving our work!

Technical Contributions to integrate air pollutant dynamics into a deep learning architecture

As stated in our earlier response, the integration of physics priors within a deep learning architecture for air quality predictions, involves more than just introducing new equations or fusing exisitng architectures. In order to address the aforementioned distinct challenges, our contributions are as follows.

• Association of Physical Priors with Graph Geometry: We have innovatively associated physical priors with the graph geometry to derive the diffusion-advection differential equations. This unique alignment allows for a more accurate representation of pollutant transport mechanisms across the sensor network. The derivation and integration of these equations within the graph structure are non-trivial, necessitating a deep understanding of both atmospheric physics and graph theory.

• Custom designed Adjacency Matrices and NN layers: Central to our methodology is the development of custom NN layers and adjacency matrices that accurately model the derived diffusion-advection differential equation. These components were designed taking into account vital factors such as the capturing of spatiotemporal dependencies within the network and influence of wind patterns on air particle movement.

  A critical aspect of this design is accommodating the influence of wind patterns on air particle movement. We introduced a novel Graph Neural Network (advGNN), specifically to model the advection process, drawing inspiration from molecular dynamics. This aspect of our model design is particularly innovative in its approach to representing complex environmental interactions.

• Fusion of multiple physical processes: Recognizing the need to concurrently consider various physical processes, we undertook extensive experimentation with different fusion approaches. The implementation of a gated fusion mechanism emerged as the most effective strategy, demonstrating superior performance in integrating diverse elements of atmospheric dynamics.

All in all, this is not a straightforward approach and we paid much attention and consideration to these critical factors. The successful integration of these components, within a physics guided deep learning architecture itself is a noteworthy advancement in addressing the intricate challenges posed by atmospheric processes. As to our knowledge, no existing methods have explored a similar fusion of physics priors with graph geometry, especially within the realm of air quality modeling.

Experiments:

We understand the importance of a diverse dataset for a thorough validation of our proposed method. As noted in our previous response, the primary reason for conducting experiments in a limited number of cities is the unavailability of comprehensive public datasets.We intend to explore collaborations and avenues to acquire additional datasets for future research.

We acknowledge that the improvements over existing methods, may appear modest. However, it's important to note that our improvements are still significant and promising specially in diverse scenarios like sparse data and sudden change prediction in the field of environmental monitoring, where even small enhancements in predictive accuracy can have substantial practical implications for public health and policy-making.

Interpretability:

Your point about the diffusion magnitude being potentially replicable with other GNNs is well noted. Please find the clarification below on the unique aspects of our proposed method in terms of interpretability.

• Beyond Black Box Interpretability: While it's true that the diffusion magnitude could be represented using standard GNN architectures, the key distinction of our model lies in its physics-guided framework. Unlike traditional GNNs, which often operate as 'black boxes', our model integrates specific physical laws and processes governing air pollutant dynamics. In other words, the GNN based network within our model is defined based on the diffusion and advection within the sensor network. This integration ensures that the model's learning and predictions are not just data-driven but also constrained and guided by established physical principles.

• Physics-Informed Feature Representations: Our method enhances interpretability by generating physics-informed feature representations. This means that each node and edge in our network is not only a data point but also a representation of physical phenomena (like pollutant dispersion patterns). This approach provides a deeper understanding of how and why certain predictions are made, linking back to the underlying physical processes.

Dear Reviewer pK4K,

This is to kindly remind you to reconsider our score as we are approaching the end of the rebuttal period in a few hours. We have provided clarification on your concerns above. We sincerely appreciate the time and effort you have dedicated to reviewing our work.

Thanks again for your valuable suggestions and comments for improving our work !

Best Regards, Authors