Physics-Constrained Comprehensive Optical Neural Networks

Thank you for your positive comments and valuable suggestions for improving the quality of our manuscript. As you rightly pointed out, our work is indeed motivated by the challenges presented by ONN models, specifically the differences between the ideal, simulated systems and their real, physical implementations. Furthermore, as you noted, our ultimate goal is to achieve more precise and robust optical computing applications. We have carefully considered your comments, and below are our responses to your feedback. 1：The generalization of the method to broader applications, not only image classification, and its scalability remains unclear.

Thanks for your insightful question. As we mentioned in our conclusion. Although our loss design is tailored for classification tasks, the focus of our work is on compensating for various known and unknown errors in the physical system to achieve consistency between experimental and simulation results. Therefore, the framework of our method can still be applied to other tasks. We will explore the scalability of our approach to different tasks in future research.

2：The performance depends on the accuracy of the physical models used to represent the system. This could limit the applicability of the framework in environments/situations where the physical modeling is challenging or impractical.

Your question is very practical. Our work has been validated for accuracy in the relatively simple 4f physical system. To achieve consistent results with more complex and challenging physical models, we would need to optimize and adjust the DNN architecture to adapt to these more complex systems, as well as integrate more quantitative physical information extracted from the physical system into the network training.

3：The authors discuss measurable physical errors — however, less insight is provided into the impact of unpredictable errors, which could be of significant importance when applying the framework to dynamic environments.

Thank you for your valuable feedback. In our next steps, we will consider testing in dynamic environments. In addition to static and fixed unpredictable errors in experiments, some other errors can vary with changes in system parameters and often interact with each other. Modeling and analyzing their formation process is quite challenging. However, there are certain characteristic disturbances these errors cause in the experimental output images. Therefore, we chose to use DNNs to learn the impact of unpredictable errors by comparing ideal simulation images with actual experimental outputs, ultimately reducing their effect on experimental accuracy. For dynamic environments, we may need to further explore new DNN architectures and training methods based on the findings of this work.

4：the integration of physical constraints might introduce additional complexity in the training — which could potentially limit its use in more resource-constrained situations. How to the authors address that?

The method proposed does add training burdens. However, it provides computational efficiency advantages compared to other error correction methods (as detailed in the "Convergence Rate and Model Architecture" section of the supplementary materials. By incorporating quantitative physical information, our DNN remains simple with fewer parameters, enhancing efficiency. Our focus is on model deployment and application; once trained, the model improves experimental results without altering the physical system. The inference process does not require additional computational resources due to the physical constraints introduced during training.

5：How does the model adapt to previously unseen physical environments? I am thinking, for example, as an implementation in autonomous vehicles or robotic systems.

You’ve raised an excellent point that aligns with our ultimate goal. In automotive and robotics applications, issues such as slight changes in the relative positions of optical components, vibrations, and significant variations in laser intensity may arise. Since our current optical experiments are conducted in a controlled laboratory environment, further research is needed to address unpredictable errors in dynamic systems. We greatly appreciate your practical insight, as it will guide our future research and provide direction for the practical implementation of ONNs. We will continue to work towards this goal in future research, aiming to enhance ONN performance in previously unseen physical environments.

6. Limitations: The authors briefly mention the limitations of this study, focusing specifically on the demonstration only for classification tasks. However, questions such as the application of this method on dynamic, unpredictable environments, remains unclear.

We will add the following paragraph to the paper to discuss the limitations mentioned by the reviewer: “Our method was developed and tested in a laboratory environment, where most errors are static and some are predictable. By incorporating predictable errors as physical priors into network training along with a relatively lightweight DNN architecture, we achieved consistency between simulation and experimental results. However, in dynamic and unpredictable environments, our proposed method would require more complex DNN architectures and new training approaches. We will work towards this direction in future research to develop ONNs that can operate stably and accurately under dynamic and unpredictable conditions.”

We sincerely appreciate the valuable feedback and insightful comments you have provided. Your suggestions have been instrumental in refining our work and enhancing the quality of our manuscript. Thank you for your thoughtful review and for contributing to the improvement of our research.

Thank you for your valuable feedback and for acknowledging the motivation and improvements in our study. Our work addresses errors in practical optical neural networks, enhances experimental performance, and aims to advance reliable optical computing. We will reply your questions and suggestions below.

1: Limited Experimentation: The experiments were conducted only on MNIST or similar, similar challenging tasks, which can be effectively addressed even with basic approaches. The author might need to validate their approach on more complex datasets. A simple single-layer fully connected (FC) network could handle MNIST, so the robustness of the proposed method on more challenging tasks should be explored.

Thank you for your question about the robustness of our method. Our approach can handle more challenging datasets like CIFAR-100. To do this, we simply replace the dataset loaded onto the DMD with CIFAR-100 images. The initial classification accuracy with CIFAR-100 is 30%, which improves to 57% after optimization using our method. However, since current research ("Dual Adaptive Training of Photonic Neural Networks" published in Nature Machine Intelligence, and "Deep Physical Neural Networks Trained with Backpropagation" published in Nature.) on spatial 2D light-based ONNs predominantly uses the datasets mentioned in the paper, we did not include the CIFAR-100 classification results in the main text due to the lack of comparative benchmarks. Moreover, the core contribution of this paper is addressing the inconsistency between simulation and actual optical neural networks, rather than just developing a high-performing ONN. For instance, experimental accuracy for MNIST was 61.6% without our method, but improved to 96.5% with it. Similarly, accuracies for CIFAR-100 and Quickdraw16 increased from 30% and 63% to 57% and 85.7%, respectively. These results validate the effectiveness and robustness of our method in correcting experimental errors.

2：One drawback of the 4f approach is that it requires three elements (two lenses and a spatial filter), resulting in a bulky optical system with a greater propensity for misalignments compared to single-element optical systems. Therefore, the author might need to explain why use the 4f system, and how does it compare with meta surface approaches?

Thank you for pointing out the drawback of the 4f approach. We agree that it is more prone to misalignments than single-element systems, and its bulkiness complicates error correction. However, since the errors analyzed are common to optical systems, our method can also be effectively applied to single-element systems, where misalignment errors are easier to manage. We agree that the meta surface approach is better for spatially structured light computing due to its industrial potential and benefits for miniaturizing ONNs. Our other team is already working on meta surface-based ONNs. We used the 4F system here because it's a classic setup that allows us to validate our method in the laboratory, but our approach is also applicable to meta surface-based ONNs.

3：It is also unclear how the author handles the nonlinearity issue. The author only mentioned that they use a linear ONN, but dropping nonlinearity from NNs is non-trivial. It might be better to explain this gap.

Your comments on nonlinearity are crucial. Our ONN includes a nonlinear activation function at the camera's receiving surface, performing a squared transformation on the light field matrix to output its squared magnitude. This operation is recognized as nonlinear in the literature, such as "Optronic Convolutional Neural Networks of Multi-Layers with Different Functions Executed in Optics for Image Classification" in Optics Express. In both our simulations and optical experiments, we incorporated this nonlinear operation. However, due to noise, this nonlinearity did not improve experimental accuracy. Nonetheless, our method can also ensure consistency between simulation and experimental results, ultimately enhancing accuracy.

4. Broader Impact: The author should emphasize the benefits of optical neural networks (ONNs) for a broader audience, such as comparing power consumption more comprehensively. Previous Work: Some important previous work, such as from the "meta surface ONN" field, might be missing. Notable references include or discussed.

Thank you for your suggestion. The advantages of optical neural networks regarding energy consumption and processing speed are mentioned in the Introduction and discussed in the referenced articles. We will add further discussion to the manuscript to comprehensively cover these advantages. “ONNs offer significant advantages over traditional electronic neural networks, including high-speed computations and low latency due to massive parallel processing with light. This efficiency reduces power consumption, handles complex tasks effectively, and is scalable for compact systems. Additionally, light transmission minimizes electrical interference and heat dissipation, enhancing performance and reliability.”

5. Limitations: The experiments were conducted only on the MNIST level dataset. The 4f system might not perform as well on more challenging datasets, such as CIFAR or ImageNet. While I don't think the author needs to conduct these additional experiments, some discussions would be great.

Thank you for your concern. This issue is mentioned in our reply to Question 1, Our work can also handle more challenging datasets as well, such as the CIFAR-100 dataset, which indicates that the method proposed in the paper exhibits a degree of generalization capability.

Finally, we would like to express our gratitude for your valuable feedback. In our future work, we aim to contribute to the practical implementation of metasurfaces for optical neural networks.

I am very grateful to your comments for the manuscript. Your questions were answered below.

1: Novelty is quite limited. (1) Using physical or analytical error modeling in training for error compensation is common practice. (2) Using a neural network to fit unknown errors to emulate the response of real hardware is also widely used in analog ONNs.

The purpose of this paper is not to emulate the response of actual hardware for building simulation-based ONN models. Instead, we aim to use deep neural networks to ensure that the actual optical path accurately replicates the results of the simulation model, ultimately achieving consistency between simulation and experimental results.

Regarding point (1), while error compensation training based on physical error model analysis is widely used in Physics-Informed Neural Networks (PINNs), this work is the first work to conduct large-scale physical experimental analysis of light source instability and exposure time mismatches in optical neural networks (ONNs). These issues are common in optical setups and significantly impact the final experimental accuracy. We have proposed corresponding solutions to address the issue. As Reviewer 8n42 mentioned, our proposed strategies have significantly narrowed the gap between simulated and physical ONNs. Our research thus provides a pathway to more accurate and reliable optical computing applications. Moreover, the physical experimental results we have documented can serve as a valuable reference for other researchers engaged in innovations in optical networks.

Regarding point (2), unmeasurable errors in optical systems are categorized into fixed and coupling errors (analyzed in Section 3.3 of the main text). While fixed errors can be addressed using DNNs, our approach updates optical system parameters each epoch, causing changes in coupling errors. We synchronize DNN updates with these parameters, allowing the DNN to correct both fixed and coupling errors more effectively. This synchronization has significantly improved experimental ONN performance. As noted by Reviewer W9o4, our method achieves state-of-the-art results on the MNIST dataset. Our method represents a significant innovation, offering unique advantages over existing approaches and demonstrating its potential for laying the foundation for the practical development of ONNs.

2: The parallel residual DNN compensation network on digital hardware significantly reduces the system throughput and efficiency. What are the main advantages of such a system compared to analog AI hardware or digital accelerators?

As noted, our approach uses residual connections between the ideal ONN model and the DNN to ensure consistency between the simulation model and the actual optical path. The DNN's ~5,000 parameters, compared to the ONN's 360,000(as detailed in the supplementary materials under Model Architecture), minimally impact system efficiency while improving consistency between experimental and simulation results.

Additionally, the DNN affects only the training complexity, not the inference process. In practical engineering applications, we are more concerned with the inference process rather than the training process. During inference, ONNs offer advantages such as low power consumption, high parallelism, and strong resistance to interference compared to analog AI hardware or digital accelerators. These advantages are highlighted in the papers “Deep Physical Neural Networks Trained with Backpropagation” published in Nature and so on.

3: The experiments are performed in very small examples. For ONNs submitted to NeurIPS, CIFAR100 or TinyImageNet are required.

For the CIFAR-100 dataset, our system was adapted by replacing the DMD images with CIFAR-100 images, maintaining the training process as described. For CIFAR-100, we used only one channel of grayscale images for training and forward experiments, achieving an improvement in classification accuracy from an initial 30% to 57%, further demonstrating the effectiveness of our method. Due to a lack of comparative results, this was not included in the main text. Current ONNs using 2D spatial structured light primarily utilize the datasets mentioned in our paper, as noted in the aforementioned articles and "Hybrid Training of Optical Neural Networks" in Optica.

4: It does not compare the efficiency and robustness with extensive literature on robustness-driven hardware/software co-design methods or on-chip training, hybrid training methods. It only compares to direct deployment.

Table 1 compares our classification performance with methods from "Dual Adaptive Training of Photonic Neural Networks" in Nature Machine Intelligence and "Deep Physical Neural Networks Trained with Backpropagation" in Nature. Section B of the supplementary materials details the convergence rate, showing that our method achieves convergence within 5 epochs despite the lower precision of initial experimental deployment. These results underscore our advantages in both accuracy and training efficiency.

In Table 1, PAT and DAT are hybrid training methods. Our results show that our method outperforms these in both training efficiency and final experimental accuracy. On-chip learning, which involves backpropagating optical signals through hardware, introduces significant experimental errors and is thus highly challenging. Most researchers use optical forward inference combined with loss function computation and backpropagation in computer systems, as seen in the papers "Deep Physical Neural Networks Trained with Backpropagation" in Nature and "Backpropagation-Free Training of Deep Physical Neural Networks" in Science. Therefore, we did not compare our method with on-chip learning approaches.

Finally, we sincerely thank you for your valuable feedback on our paper. We will address the issues you raised through further in-depth research.

We acknowledge that using neural networks for error compensation is a common technique in analog NN calibration. However, the errors present in physical systems are often too complex to be fully addressed by simple neural networks. As discussed in our experiments, even with the aid of a compensation DNN, the classification accuracy on the MNIST dataset reached only 93.5%. By innovatively integrating quantitative physical information into the model design and training, guide the design of the neural network and narrow the optimization scope of key parameters using the physical model's transmission process and known critical parameters. we were able to improve the accuracy from 93.5% to 96.5%, demonstrating the effectiveness of our approach.

Therefore, we argue that our contributions extend beyond the solid engineering efforts; they include original observations and innovative ideas specifically designed to bridge the gap between simulated and physical ONNs, thus laying the foundation for the practical development of ONNs.

What’s more, for the research of error compensation and backpropagation algorithms for optical neural networks, which is a multidisciplinary topic covering optic、computer science、artificial intelligence, the related papers are accepted/published at different venues. Some are published at top-tier journals, e.g., Nature, Science etc., which are deemed as comprehensive journals, and some are published at more specific journals, e.g., Nature Machine Intelligence, Nature Computational Science etc. Therefore, we believe our manuscript is well suited to and adequately match the coverage of NeurIPS, which is a broad and inclusive platform that welcomes contributions across all areas of artificial intelligence, including the application and integration of engineering efforts that advance the field. We believe that our original research, which addresses the gap between simulation and experimental results in ONNs, holds significant value for the AI community. The impact of NeurIPS is comparable to that of high-impact journals, and we are confident that our work can spark interest and inspire further research across the fields of AI and machine learning.

We greatly appreciate your thorough review and the positive feedback on our work. As you mentioned, the core aim of our study is to narrow the gap between simulated and physical ONNs and to provide a pathway for more accurate and reliable optical computing applications. Your description is a very precise summary of our paper.

1: (1) Providing an analysis of the computational costs associated with the proposed framework, including comparisons of training and inference times. (2) Including a more in-depth analysis of how the proposed methods differ fundamentally from other approaches would clarify their unique advantages.

Regarding (1), during the training of the ONN, we train based on the transmission equations of light in a 4f system, achieving end-to-end mapping through the adjustment of the system's tunable parameters. The DNN used is a lightweight fully connected structure, which converges quickly. The only limitation on training efficiency is the acquisition of experimental images. To address this, we developed a synchronized trigger control unit to ensure that image refresh rates in the DMD (Digital Micromirror Device) are synchronized with camera image capture. The DMD refresh rate is typically very high, capable of refreshing an image every 10 microseconds, but industrial cameras often struggle to match this speed. Consequently, training and forward inference efficiency depend on the camera frame rate; using a higher frame rate camera can improve training efficiency and accelerate inference times.

Regarding (2), unlike other methods, we are the first to incorporate quantitative physical information into network training. By integrating this quantitative physical data into the model training, we are able to reduce model complexity of error compensation networks and enhance convergence efficiency as discussed in Convergence Rate section of supplemental material.

2：（1）How well does your framework generalize to datasets with different characteristics (e.g., noise levels, resolutions, and complexity)?（2）How did you determine the parameters for the loss function?

Our experiments were conducted on three datasets with different characteristics: MNIST, Fashion-MNIST, and QuickDraw16. Additionally, we also processed the CIFAR-100 dataset. The features extracted for classification from these datasets and the number of categories differ significantly, and there is substantial variation in the brightness and contrast among the datasets. Additionally, the optical paths used in practice are highly sensitive to variations in brightness and contrast. For the MNIST and CIFAR-100 datasets, our method improved accuracy from 61.6% and 30% to 96.5% and 57%, respectively. Ablation studies across these datasets demonstrate that our method consistently aligns simulation and experimental results and significantly enhances experimental accuracy for datasets with varying features. Regarding the noise levels in the datasets, due to time constraints, we plan to incorporate noise into the data in upcoming experiments to further assess the generalization capability of our method.

The parameters for the loss function are determined based on physical prior information. For example, due to factors such as laser jitter, camera dark current, and other irregular noises, the grayscale values of the same region in an image are not constant but fluctuate over time. We record the range of grayscale value fluctuations and use this information to design the relevant parameters in the loss function.

3：Limitations：The authors have identified key limitations, particularly their focus on image classification tasks and the specific applicability of their intensity gap loss function. I think authors should also discuss how the framework might perform under real-world conditions in this work.

The issues you raised are highly practical. As shown in the Experimental Setup section of the supplementary materials, the actual optical layout of the 4f system indeed presents challenges such as large volume, numerous optical components, and difficulty in optical alignment. However, our proposed method addresses common errors in optical systems and ensures consistency between experimental and simulation results, significantly improving the final experimental accuracy even in error-prone 4f systems. Therefore, we believe that our method can be effectively applied to single-element systems, such as metasurface-based ONNs, which are more suited for real-world conditions.

We sincerely thank you for your valuable feedback and thoughtful comments. Your insights have greatly contributed to improving the quality of our manuscript. We appreciate the time and effort you have dedicated to reviewing our work.