Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression

Thank you for your feedback. We would like to clarify what we mean by GVA. The “GV”, or gradient variance, stands for stochasticity introduced by the stochastic gradient updates. The presence of gradient variance is a known phenomenon. By amplification, we do not mean that the training procedure increases variance of gradients. Rather, we mean that the evaluation of a policy in a closed-loop setting, namely, imitation learning, is responsible for said amplification. More specifically, as is highlighted in bold on page 5 of the submission, as well as emphasized in italics several lines below, GVA is the phenomenon by which small perturbations of parameters, caused by variance in the stochastic gradients used during training, is amplified by the closed feedback loop created when past model predictions are fed back into the model during the rollout.

Thus, it is the disparity between the reward function of interest and the loss being optimized that leads to these oscillations. Nowhere do we claim that this variance is amplified during training, which is why no such plot is exhibited in our paper. To reiterate, the main claim is not that gradient variance amplifies through training but rather that closed feedback loops lead to parameter sensitivity, which leads to poor performance when coupled with gradient stochasticity.

Our experiments were carefully chosen to justify our diagnosis of the GVA phenomenon: (a) More data, and various forms of data noising / regularization, do not address the issue (b) Increasing batch size does improve, but not entirely ameliorate GVA (c) EMA, which reduces iterate variance, nearly entirely fixes GVA (d) Through our analysis of local perturbations around imitator policies at different stages of training (Fig 1 right and Fig 13), we observe that while the validation loss landscape is relatively insensitive to parameter perturbations, the policy evaluation loss landscape is very sensitive to these.

In summary, we conclude that (i) evaluation loss is highly sensitive to changes in parameters and (ii) that methods which reduce variance in iterates/gradients seem to ameliorate this sensitivity. The role of our theory is to show that the above hypothesis is mathematically plausible.

We thank the reviewer for their feedback. We address their concerns below:

We would like to clarify our exposition of GVA. First, note that our mechanistic understanding of GVA arises from both careful empirical findings and theoretical explanations. To reiterate, the empirical findings for GVA are: (a) More data and/or regularization do not mitigate GVA. (b) Increasing batch size improves, but does not entirely ameliorate, GVA. (c) EMA, which reduces iterate variance, nearly entirely fixes GVA

Through our analysis of local perturbations around imitator policies at different stages of training (Fig 1 right and Fig 13), we observe that while the validation loss landscape is relatively insensitive to parameter perturbations, the policy evaluation loss landscape is very sensitive to these.

In summary, we conclude that (i) evaluation loss is highly sensitive to changes in parameters and (ii) that methods which reduce variance in iterates/gradients seem to ameliorate this sensitivity.

The role of our theory is to show that the above hypothesis is mathematically plausible, and revealed even without considerations of non-convex representation learning.

Proposition 3.1: We wish to clarify the precise statement of Proposition 3.1, which is intended as an example of exponential error amplification but not intended to be perfectly representative of what we observe in experiments. Proposition 3.1 does not state that for every delta, the BC error is exponential-in-H larger than validation. Instead, it shows that For all delta > O(1/H), there exists a perturbed policy of error delta which witnesses exponential blow up For all delta < O(1/H), there is no exponential blow up.

The point of the proposition is that, to ensure no exponential blowup in the worst case, for long rollout horizons, the BC loss must be made sufficiently small. It may be possible, with long training times and sufficient learning rate decay, that one could make the BC loss sufficiently small as to ensure the rollout reward does not suffer from exponential blowup. Indeed, our experiments corroborate this qualitative finding (point to Fig): that if we train for a longer time with sufficient learning rate decay, we do get consistently large rollout reward. This, however, is wasteful in terms of training time, as EMA does just as well without excessive numbers of training epochs.

Cliff Loss:

Surprising, we find that GVA is not a major obstacle when learning to control linear systems (which includes the example in Prop 3.1). If we examine how the cloned policy behave during rollout time, we found (and will include figures in the final revision) that the deviation between cloned policy and expert, under the rollout of the cloned policy, starts to deviate wildly after a specific time step in the RL environment. We interpret this as the presence of dynamical stiffness in the task, e.g. the contact that the locomoting agent makes with the floor surface. Importantly, MuJoCo environments are well approximated by Piecewise Affine Systems, a classical model for contact dynamics.

In the revision, we will clarify that the cliff loss is a simple and tractable mathematical abstraction of the above phenomenon. The cliff models a “two-region” contact system, where ‘falling off the cliff’ corresponds to a change in region.

Questions

This is a good question that we will address in the revision. To be precise, in our particular example in the proof of Proposition 3.1, there are good subsets only around parameters that are `marginally stable' and thus many policies do not have these good neighborhoods. We believe this phenomenon to persist in the general case, with Figure 1 (right) and Figure 13 providing empirical evidence in favor of this claim in the MuJoCo Walker environment.

We thank the reviewer for their detailed and positive feedback. We also would like to thank the reviewer for pointing out typos; we will fix them for the revision. We respond to each of the points individually:

1. Anecdotally, we find that the effect of EMA on GVA is relatively insensitive to the specific decay schedule of the EMA. We have not experimented with more complicated, non-monotonic decays, but will do this for one of the MuJoCo experiments and include the result in the final paper.

2. Thank you, this is a good point. We will add the labeling in the final paper. Note that for evaluation of the current paper, the approximate scaling is implied by the location of the highlighted boxes on the left of Figure 4; for example, the leftmost zoomed in box ranges from validation loss around 1.01 to loss around 1.14.

3. We agree that changes in validation perplexity are orthogonal to GVA (and we have pointed this out in the paper, which we will attempt to further clarify in the revision). As a more direct parallel to the imitation learning experiments, we also studied full-sequence switching costs in argmax decoding. That GVA appears in perplexity is interesting in its own right, in that perplexity is not intrinsically autoregressive (evaluating the likelihood of entire text generation); note that this also occurs in the MuJoCo setup (see Figure 6b). We are excited to understand how GVA is so widespread as to affect even the 1-step loss, as well as developing more refined metrics, to future work. We emphasize that our LM results are preliminary and primarily illustrative of the phenomenon of error amplification when a 1-step objective is used as a proxy in closed-loop settings.

Questions

1. Yes, in each inequality in the proposition, the Delta is allowed to depend on the precise parameters and thus may be different. Note that this is a stronger statement than if they must be the same.