One Forward is Enough for Neural Network Training via Likelihood Ratio Method

Responses to Weaknesses

AW1: Thank you for your feedback. We will modify the formulas and definitions for readability in the revision. We provide a step-by-step expansion of the loss expectation, explicitly annotating the random variables, to further illustrate details in the derivation of ULR as follows: $\mathbb E_{z^0,\cdots, z^{L-1}}[\mathcal L (x^L)] = \mathbb E_{x^l}[\mathbb E_{z^l,\cdots, z^{L-1}}[\mathcal L (x^L)|x^l]] = \mathbb E_{x^l}[\mathbb E_{x^{l+1}}[\mathbb E_{z^{l+1},\cdots, z^{L-1}}[\mathcal L (x^L)|x^{l+1},x^l]|x^l]] = \mathbb E_{x^l}[\int_{\mathbb R ^{d_{l+1}}} \mathbb E_{z^{l+1},\cdots, z^{L-1}}[\mathcal L (x^L)|\xi,x^l ]g^l(\xi)d\xi],$ where the first two equalities are justified by the law of iterated expectation, and the third equality comes from the definition. Moreover, the notation $J_{\theta^l} \varphi^l(x^l;\theta^l)$ represents the Jacobian matrix of the original output $\varphi^l(x^l;\theta^l)$ with respect to $\theta^l$, but not that of the perturbed result $x^{l+1}$.

AW2: Although ULR has better scalability than previous methods and can support training larger models, the application of pure ULR on large neural networks might not be the most economical, as stochastic optimization requires multiple evaluations to improve performance. Thus, in our paper, we conduct experiments with pure ULR as well as other baselines on relatively shallow neural networks to verify the effectiveness of ULR training. However, as presented in Section 3.4, ULR can rearrange the gradient computation graph in deep learning and work with BP, where ULR is used to estimate the gradient of the last part in each module, and BP passes the information throughout the module. This approach has much lower pressure on memory than pure ULR, as well as improves the parallelism of gradient computation in large and deep models, thereby enhancing efficiency, as verified by two demos in Section 4.2.

Responses to Questions

AQ1: Firstly, though only one forward is necessary for ULR, we agree that ULR needs multiple forward evaluations with respect to $z^l$ to enhance the gradient estimation. However, it should be noticed that forward evaluations with different $z^l$ are independent and thus can be easily parallelized. We can stack multiple copies of the original data batch into a large batch for just one forward computation. The computation with a large batch size can fully utilize the capacity of GPUs. Moreover, we apply multiple tricks to improve estimation efficiency, as shown in Alg. 1, which largely alleviates the issue. There is still significant potential to further reduce consumption, such as by reusing evaluation results through the importance sampling technique. Secondly, ULR can work in conjunction with BP/BPTT, balancing training duration and memory occupation to improve training efficiency. As presented in Section 3.4 and the second demo in Section 4.2, we can use ULR to cut the original long gradient computation chain into multiple independent short graphs, which do not require a lot of forward computation but can greatly improve parallelism. Thirdly, ULR saves backward computations on those parts that do not need optimization, thus reducing computational effort compared to BP. When the network embeds pre-trained models or other, even black-box components, ULR can bypass these parts for gradient calculation, as shown in the first demo in Section 4.2.

AQ2: The training process with perturbation-based methods essentially aims at optimizing all parameters in the perturbed model, including those of noise distributions, which is a reasonable proxy of the training without any noise. Besides, noise injection plays a role in increasing the model capacity and regularization, and appropriate noise parameters can contribute to the performance and robustness. Thus, we use ULR to calculate the gradient of noise distribution parameters with respect to classification loss and optimize them with gradient descent algorithms. By viewing the training logs, we surprisingly find that the gradient estimation accuracy is also improved compared to that with the non-optimized noise, which indicates a good byproduct of the improved gradient estimation accuracy by noise optimization in ULR training. We appreciate the idea that noise parameters can be optimized for more accurate gradient estimation and hope this study can motivate the following works to derive new theoretical tools for this purpose.

AQ3: Yes, ULR-WL represents the hybrid perturbation manner in which we introduce noise to the weights of the first several layers and to the logits in the subsequent layers for gradient computation. In the case of ULR-WL training on VGG-8, we specifically perturb the weights in the first three layers and the logits in the remaining layers.

AQ4: In convolutional neural layers, each element within the same output channel shares a single bias scalar. Therefore, the whole bias term, denoted as $b$, consists of $c_\text{out}$ elements.

A: Thank you for your suggestion. We have modified the explanation of the cosine similarity in the manuscript. We also seek the reviewer's advice on whether "Only Forward is Necessary for Neural Network Training via the Likelihood Ratio Method" would be a more appropriate claim/title for this work.

The current title has two intentions. Firstly, it aims to underscore the distinction between ULR and other algorithms like FF and SPSA, which require at least two forward passes per iteration. Although utilizing repeated evaluations is desirable for enhanced training quality, a single forward pass is theoretically acceptable for ULR when the numerical overflow issue is addressed with care, e.g., by parameter clipping. Secondly, the title emphasizes that ULR eliminates the need for recursive backward calculations, in contrast to forward computations.

AQ1: This is largely because ULR does not suffer from the gradient vanishing/exploding issues that can occur with BP in deep forward propagations. Since gradients are calculated iteratively and multiplied by the derivative of the activation function at each layer/step in BP/BPTT, the gradient signal can decrease exponentially throughout the propagation when the derivatives of the activation function are close to zero. This can result in a stagnation of model performance growth [1,2]. However, in the BPTT rearrangement, ULR divides the gradient computation recursion into several shallower ones, mitigating this impact of the activation function. And in pure ULR, the gradient computation does not involve any differentiation of the activation function nor recursive calculations throughout the computation graph, thereby avoiding gradient-related issues.

Moreover, the curves plotted in Fig. 7 represent the empirical means of cosine similarity, primarily reflecting the variance of our gradient estimation. ULR still yields an unbiased estimation of the true gradient. We observe that the depth of layers is not the primary factor affecting the variance. Rather, the results of pure ULR in Fig. 7 are due to both perturbation strategies using high dimensional noises in the middle layers. But the noise dimensionality can be reduced in the hybrid application with BP or further parallel of ULR. Additionally, the estimation variance does not necessarily suggest a failure of gradient descent, as the noise from an unbiased estimator, denoted by $\delta_k = g(\theta_k)- \nabla_{\theta}\mathbb E[\mathcal L(\theta_k)]$, where $\mathbb E[\delta_k] = 0$, can be averaged out throughout the gradient descent recursion [3]. Appropriate noise injection can even play a role in regularization, thus increasing the performance on the test dataset [4,5].

[1] Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. "On the difficulty of training recurrent neural networks." International conference on machine learning. PMLR, 2013.
[2] Vorontsov, Eugene, et al. "On orthogonality and learning recurrent networks with long term dependencies." International Conference on Machine Learning. PMLR, 2017.
[3] Harold, J., G. Kushner, and George Yin. "Stochastic approximation and recursive algorithm and applications." Application of Mathematics 35.10 (1997).
[4] Zhou, Mo, et al. "Toward understanding the importance of noise in training neural networks." International Conference on Machine Learning. PMLR, 2019.
[5] Smith, Samuel, Erich Elsen, and Soham De. "On the generalization benefit of noise in stochastic gradient descent." International Conference on Machine Learning. PMLR, 2020.

AQ2: Thank you for your suggestion. We have specified the definition as "$b = (b^o)\in\mathbb R^{c_\text{out}}$" to clarify that $b^o$ is the $o$-th elememt of $b$ in the revision.

Responses to Questions 1-3

AQ1: In ULR experiments, we optimize parameters in noise distributions during training for enhanced performance. ULR supports the estimation of gradients for all parameters within the neural network. We calculate the gradients of loss values with respect to the noise generation parameters and apply gradient descent algorithms, such as SGD or Adam, to update these parameters. The estimated gradient formulas for noise distribution parameters are presented in Section 3.1 and Appendices A.1-5. With the adaptive perturbation technique, the performance of neural networks can be further improved. We conduct ablation studies on adaptive perturbation. Following the same setting in our paper, we respectively use ULR with and without adaptive perturbation to train ResNet-5 and VGG-8 on the CIFAR-10 dataset, and RNN, GRU, and LSTM on the Ag-News dataset. The results are reported as follows.

Tab. R1 Classification accuracies (%) of ResNet-5 and VGG-8 on the CIFAR-10 dataset. We respectively use ULR with ($\checkmark$) and without ($\times$) adaptive perturbation in neural network training.

Models	Adaptive Perturbation	Accuracy
ResNet-5	$\times$	61.3
ResNet-5	$\checkmark$	64.8
VGG-8	$\times$	75.9
VGG-8	$\checkmark$	77.3

Tab. R2 Classification accuracies (%) of RNN, GRU, LSTM on the Ag-News dataset. We respectively use ULR with ($\checkmark$) and without ($\times$) adaptive perturbation in neural network training.

Models	Adaptive Perturbation	Accuracy
RNN	$\times$	69.2
RNN	$\checkmark$	70.9
GRU	$\times$	86.6
GRU	$\checkmark$	88.9
LSTM	$\times$	87.8
LSTM	$\checkmark$	89.4

As reported in Tab. R1 and R2, we observe an improvement in neural network training with ULR by adopting adaptive perturbation. This demonstrates the effectiveness of optimizing the noise distribution in boosting the accuracy of gradient estimation for ULR.

AQ2: Thank you for your insightful suggestion. ULR supports various noise injecting strategies, which hinge on how we define the module $\varphi^l(\cdot;\theta^l)$ in Theorem 1. Injecting noise into the weights can be interpreted as perturbing the logit output of an identity mapping, with the weights as its input. And we agree that there may be better strategies to apply our ULR framework. As mentioned in the CNN part, the variance of the gradient estimation grows as the total dimensionality of injected noises increases. Therefore, the choice between the step-wise and weight perturbation largely depends on $T$ and $d_x$. Furthermore, if the impact of each step on the final loss is additive, then we can obtain a more refined result for RNNs with step-wise perturbation as shown in Appendix A.7. For scenarios where the effect of a single forward step is transient, truncation tricks can be employed to further reduce the impact of $T$, akin to the introduction of a discount factor in reinforcement learning. Consequently, when $T$ is manageable, and the loss is decomposable, the noise injection strategy outlined in our paper is preferred. In contrast, we may consider the weight perturbation mentioned above.

AQ3: We apply our proposed methods for variance reduction, including layer-wise noise injection, antithetic variables, and Quasi-Monte Carlo, to the ES algorithm. We use ES to train ResNet-5 and VGG-8 on the CIFAR-10 dataset. For comparison, we also present the results of ULR in our paper. The results are as follows.

Tab. R3 Classification accuracies (%) of ResNet-5 and VGG-8 on the CIFAR-10 dataset using the ES and ULR, respectively.

Model	ResNet-5	VGG-8
ES without tricks	35.1	65.8
ES with tricks	46.9	67.2
ULR	64.8	77.3

As shown in Tab. R3, with the proposed tricks, there is a significant improvement in ES for training neural networks, indicating the effectiveness of our tricks in variance reduction. However, even with the integration of these tricks, ES still struggles to optimize neural networks effectively, demonstrating a lower capacity in gradient estimation compared to ULR. This argument is also supported by our ablation study in Fig. 7(b) of Section 4.3. When perturbing only the weights for gradient estimation, we achieve considerable accuracy in only the first one or two layers but fail to calculate the rest accurately. This further highlights the superiority of ULR over ES in gradient estimation, as shown in Fig. 7$c$.

Responses to Questions 4-7

AQ4: In Fig. 6, we present a performance evaluation of BPTT rearrangement. The evaluation metrics include the classification accuracy of trained RNNs on the Ag-News dataset and the training duration per epoch, where the classification accuracy and training duration are represented by the curves and bars, respectively. The x-axis represents the depth of the gradient computation graph used in BPTT boosted by ULR, while 'BPTT' on the right side indicates no cuts on the graph. The number '5' indicates that we cut the original graph into multiple subgraphs with a maximum depth of 5 and use ULR to launch the gradient computation at the leaf node in each subgraph, then apply BPTT to pass information throughout the rest. It can be observed that with BPTT boosted by ULR, the training efficiency of RNNs is improved, where the classification accuracy remains high, but the training duration is reduced.

AQ5: In Fig.7, the reference BP gradient is compared with the estimated gradient by ULR. In ULR, the estimated gradient is computed by injecting noise into the loss. At the same time, the BP gradient is computed on the vanilla network, where no noise is injected into neural activities, thus excluding the effect of perturbation. We use the cosine similarities to evaluate the gradient estimation accuracy.

AQ6: We agree that the application of ULR cannot avoid the forward pass on non-trainable/frozen layers. The advantage of ULR is to skip the gradient computation of these layers during the backward pass, thus bringing an efficiency improvement as presented in Tab. 1. As pointed out by previous studies [1,2], one backward pass in a neural network theoretically takes about twice as long as the forward pass. Computation hardware, such as GPUs and TPUs, supports the forward pass far better than the backward pass [3]. Thus, by disabling part of the backward pass in gradient computation, ULR can achieve efficiency improvement in practice in our experiments as well as in theoretical analysis. Moreover, ULR can take advantage of the importance sampling technique in the future to update parameters several times with only one forward pass, which further reduces unnecessary calculations.

[1] Lin, Zhouhan, et al. "Neural networks with few multiplications." arXiv preprint arXiv:1510.03009 (2015). [2] Narayanan, Deepak, et al. "PipeDream: Generalized pipeline parallelism for DNN training." Proceedings of the 27th ACM Symposium on Operating Systems Principles. 2019. [3] Liu, Xiao-Yang, et al. "High-Performance Tensor Learning Primitives Using GPU Tensor Cores." IEEE Transactions on Computers (2022).

AQ7: We only use one augmentation strategy, namely the random horizontal flip, in our model training on the CIFAR-10 dataset. All the parameters have been fine-tuned for the best performance, and the models are trained sufficiently to converge. However, due to the limited size of the neural networks, i.e., ResNet-5 and VGG-8, they do not achieve high accuracy on the CIFAR-10 dataset, even with BP for gradient computation. Similar results can be found in a previous study [1], which used BP to train a shallow ResNet and achieved around 60% accuracy on the CIFAR-10 dataset. We also compare this work in our appendix of Section B.6.

[1] Ma, Wan-Duo Kurt, J. P. Lewis, and W. Bastiaan Kleijn. "The HSIC bottleneck: Deep learning without back-propagation." AAAI, 2020.

Responses to Weaknesses

AW1: Thank you for your suggestion. We wonder whether the reviewer thinks "Only Forward is Necessary for Neural Network Training via the Likelihood Ratio Method" is an appropriate claim/title for this paper. The initial intent of the current title is to highlight the distinction between our ULR algorithm and others like FF and SPSA, which require at least two forward passes, as well as the absence of a need for recursive backward calculations in contrast to forward ones. Although using multiple evaluations is preferable for enhanced training quality, one forward pass is theoretically acceptable. Furthermore, multiple evaluations can be packed into a single batch to be processed in one forward pass.

AW2: The purpose of these extensions is to provide readers with specific examples, opening up ideas for the use of ULR. We offer corresponding solutions for issues related to spatio-temporal parameter sharing, unstructuredness, and discontinuity, which cover almost all possible network structures in mainstream deep learning, demonstrating the flexibility of ULR training. This part also inspires our subsequent discussions on computation graph rearrangement and the relationships between different works/paradigms.

AW3: Please see the response in AQ3.

AW4: ULR offers better scalability than previous methods for training larger models, but using it exclusively on large neural networks may not be that cost-effective. Replacing deterministic calculations with stochastic ones may require repeated loss evaluations. Thus, in our paper, we conduct experiments with pure ULR as well as other baselines on relatively shallow neural networks to verify the effectiveness of ULR training. Since ULR is compatible with BP, a viable application prospect in large neural networks is to use ULR to skip the backward computation in non-updating parts of the network or to transform iterative computations into parallel ones, as demonstrated in Section 4.2.

Dear Reviewer s8CG,
We have submitted our response to your questions. We sincerely appreciate your valuable feedback on improving the quality of our paper.
Are there any additional questions or concerns we can answer? Thanks for your reply!

Yours,
Authors

Responses to Weaknesses

A: We agree with the reviewer that consumption will increase for ULR to produce accurate gradient estimations as the scale of neural networks grows. There remain three promising directions to alleviate this issue. Firstly, since ULR is compatible with BP, using BP in modules with good continuity and ULR in black boxes, non-updating, or discontinuous parts can be a more economical option than using either BP or ULR solely, as we demonstrated in Sections 3.4 and 4.2. The perturbation occurs only in the parts where ULR is applied, thus making the consumption acceptable. Secondly, the importance sampling technique has the potential to be integrated into ULR with care, which allows multiple updates with one forward evaluation and further reduces the computational effort. Thirdly, we might consider relaxing the requirements for extremely accurate gradient estimations and exploring robust optimizers that match ULR, where the impact of the estimation variance can be averaged out during iterations.

Responses to Questions

A: We intend to study the role of the proposed tricks presented in this paper through ablation experiments. The cosine similarity between gradients computed by ULR and BP serves as a proxy indicator for network training quality. We observe that the accuracy of gradient estimation varies across different neural layers and exhibits a 'buckets effect' during training. Hence, it is necessary to plot each layer separately to analyze the effect of these proposed tricks. Firstly, as shown in the first row of Fig. 7 and 11, perturbing the logits results in improved figures when the input dimensionality is small and the weight dimensionality is large, aligning with the discussion in Section 3.2. Secondly, the middle row of Fig. 11 indicates that the optimal choice of initial noise magnitude depends on the structure and order of the layers. Thirdly, we visualize the effect of the proposed variance reduction methods in the last row of Fig. 7 and 11. Their absence consistently negatively impacts all layers.

Dear Reviewer hjbZ,
We have submitted our response to your questions. We sincerely appreciate your valuable feedback on improving the quality of our paper.
Are there any additional questions or concerns we can answer? Thanks for your reply!

Yours,
Authors

Responses to Weaknesses

AW1: Thank you for your suggestion. We will modify the assumptions of Theorem 1 in our revision. In our derivation, the expectation of the loss is expanded as follows: $\mathbb E_{z^0,\cdots, z^{L-1}}[\mathcal L(x^L)] = \mathbb E_{x^l}[\mathbb E_{x^{l+1}}[\mathbb E_{z^{l+1},\cdots, z^{L-1}}[\mathcal L(x^L)|x^{l+1},x^l]|x^l]],$ where we have reformulated the second expectation as an integral in the paper. Thus, the variable $\xi$ represents $z^l$, and $g^l(\cdot)$ denotes the conditional density of $x^{l+1}$ given $x^l$. By applying the law of iterated expectation, ULR pushes the parameter $\theta^l$ from the loss evaluation $\mathcal{L}(x^L)$ into the conditional density $g^l(\cdot)$, restricting the differentiation within $g^l(\cdot)$ as shown Fig. 2. The conditions involve not only $f^l(\cdot)$ but also $\varphi^l(\cdot;\cdot)$, and we intend to encapsulate them using $g^l(\cdot)$ for readability. Moreover, since $\varphi^l(\cdot;\cdot)$ can be defined flexibly, excluding those non-differentiable but non-parametric items, whether the assumptions of Theorem 1 are met primarily hinges on the smoothness of $f^l(\cdot)$.

AW2: Yes. The star in Eq. (3) represents the convolution of $x^i$ with kernel $\varepsilon^o$. That is the reason why ULR for CNNs can be implemented with existing high-performance convolution algorithms in standard machine learning toolkits, e.g., torch.nn.functional.conv2d in PyTorch. More details can be found in lines 58--71 of cnn/module.py, which is provided in the supplementary.

AW3: Please see the response in AQ2.

AW4: Thanks. We fix this typo in the revision as follows: "Since these methods change the classical training paradigm or even give up the gradient information, many algorithmic or hardware technologies developed based on BP by predecessors cannot be fully leveraged, making such approaches computationally unfriendly."

Responses to Questions

AQ1: There are two extreme perturbing strategies for gradient estimation. One strategy is to perturb different neurons independently and precisely compute the gradients of parameters in each neuron. However, it is computationally intensive and leads to unbearable training efficiency. The other perturbs all neurons of the whole neural network. While it requires less computation for each estimation, it results in lower gradient estimation accuracy and, consequently, diminished neural network performance. In our paper, we further propose a layer-wise perturbation strategy, perturbing the neurons in one layer at a time. This design strikes a balance between gradient estimation accuracy and computation efficiency. ULR allows flexible perturbation across different parts of the neural network, depending on the consideration of the location of optimizable parameters, the training efficiency, and the gradient estimation accuracy.

AQ2: The current practical limitation of ULR is the large number of data copies required for accurate gradient estimation when training large models on large datasets, which imposes an additional burden on device memory. Training ResNet-50 on ImageNet using pure ULR may be uneconomical due to the expensive memory footprint.

However, as presented in Sections 3.4 and 4.2, ULR is compatible with BP and can be used to segment the gradient computation graph into multiple subgraphs. In each subgraph, gradient computations from leaf nodes are initiated independently and in parallel using ULR, while BP transmits information throughout the rest. This design doesn’t require as many copies for accurate gradient computation and can make a trade-off between parallelism and repeated forward evaluation, which may enable us to train large models on large datasets with promoted efficiency. Thus, we hope our study can motivate future work on 1) proposing novel techniques to reduce the gradient estimation variance in ULR in a computation- and memory-friendly manner; 2) designing an efficient training framework by rearranging the gradient computational graph with ULR.

AQ3: ULR offers the flexibility to wrap any structure as a unit for independent gradient computation, which only requires the unit to be differentiable. This feature enables the selection of various structures for the untrainable or non-parametric components within neural networks. The direct impact is that we do not have assumptions of activation functions. In Section 4.1, we effectively train SNNs with ULR, where the non-differentiable spiking activation function hinders the application of vanilla BP. From a broader perspective, ULR enables the integration of black box elements like simulators, pre-trained large models, and commercial optimizers into neural networks as static modules. This significantly facilitates the fusion of domain-specific knowledge with neural network architectures. In Section 4.2, the first demo for domain adaptation provides an example of this purpose.

Dear Reviewer HFF5,
We have submitted our response to your questions and revised our paper as you suggested. We sincerely appreciate your valuable feedback on improving the quality of our paper.
Are there any additional questions or concerns we can answer? Thanks for your reply!

Yours,
Authors