An Information Theoretic Approach to Interaction Grounded Learning

We thank Reviewer yTyJ for his/her time and feedback on our work. The following is our response to the reviewer's comments and questions.

1. Regularizer in Objective (4) and "over-fitting" problem.

The reviewer asks for stronger evidence for including the regularizer $I(X,A;R_\psi)$ in Objective (4) and explanations for the "over-fitting" problem. As this has been empirically shown in our numerical results in Table 4, we would like to discuss it in greater details. Indeed, we can show that when minimizing only $I(Y;X,A|R_\psi)$: (A1) any feedback-dependent reward decoder $\psi:\mathcal{Y}\to[0,1]$ attains a small value, and (A2) the minimum is attained by a subset of feedback-dependent reward decoders $\psi:\mathcal{Y}\to\{0,1\}$ that assigns value 1 to "an arbitrary half of" the feedback in the data distribution, if the feedback variable $Y$ is (nearly) deterministic to the context-action $(X,A)$. Both cases would lead to the "over-fitting" problem. We totally agree with the reviewer that presenting these results before defining the regularized objective helps motivate our approach. We have included this discussion in the revised Section 4.1.

Detailed analysis. To showcase these arguments, first note that minimizing $I(Y;X,A|R_\psi)$ is equivalent to $\min_\psi\mathcal{L}'(\psi):=(H(R_\psi|X,A)-H(R_\psi|X,A,Y))-(H(R_\psi)-H(R_\psi|Y))$ where $H$ is the Shannon entropy. This holds by the facts that $I(Y;X,A|R_\psi)=I(Y;R_\psi|X,A)-I(Y;R_\psi)+I(Y;X,A)$ and the last term $I(Y;X,A)$ is independent of the reward decoder. Hence, the value is non-positive for any reward decoder $\psi:\mathcal{Y}\to[0,1]$ by noting that $H(R_\psi|X,A,Y)=H(R_\psi|Y)$ and $H(R_\psi|X,A)\le H(R_\psi)$. In contrast, $\mathcal{L}'$ can be positive for reward decoders that also depend on the context(-action), which justifies (A1). Further, if the feedback $Y$ is (nearly) deterministic to the context-action $(X,A)$, we have that $H(R_\psi|X,A)\approx H(R_\psi|Y(X,A))\approx0$, and hence the minimum $-\log2$ is attained by any reward decoder $\psi:\mathcal{Y}\to\{0,1\}$ that assigns value 1 to "an arbitrary half of" the feedback in the data distribution (i.e., $\mathbb{E}\_{x\sim d_0,a\sim\pi_\textup{b}(\cdot|x)}[\psi(Y(x,a))]=0.5$), which justifies (A2).

2. Large variance for experiments with $\beta=0$.

We believe that the observation that the experiments with $\beta=0$ displayed the highest variance is closely related to the minimization of $I(Y;X,A|R_\psi)$, which we have discussed in response to the reviewer's Comment 1 above. Since any feedback-dependent reward decoder can attain a small value of $I(Y;X,A|R_\psi)$, the training process will be highly unstable and the optimized reward decoder will be constantly changing at every iteration that could lead to high variance.

3. "Does the context distribution have to be fixed throughout training?"

Yes, while the results are reported for fixed context distribution throughout training, the algorithm can be also applied to non-stationary context distribution.

4. Notation of $V(\pi)$.

We thank the reviewer for pointing out the possible ambiguity in the notation. We have revised it to $V(\pi):=\mathbb{E}\_{x\sim d_0}\mathbb{E}_{a\sim\pi(\cdot|x)}[\mu(x,a)]$ in the paper.

5. "Ensure: Policy $\pi$" in Algorithm 1.

We thank the reviewer for pointing out the typo in Algorithm 1. We have corrected this typo in the revision.

We thank Reviewer Jvuq for his/her feedback on our work. The following is our response to the reviewer's comments.

1. Finite-Sample Guarantees.

Please refer to our general response.

2. Novelty compared to the existing work.

We note that the papers mentioned by the reviewer focus on the variational estimation of mutual information. On the other hand, our work focuses on the specific RL problem in the IGL setting. Please note that our main contribution is to propose an information theoretic approach to this RL problem, while the reviewer seems to evaluate the novelty of our estimation method for conditional mutual information. We sincerely hope that the reviewer would also consider our contributions on addressing the IGL-based RL setting in his/her evaluation of our work.

3. Appendix

We wish the reviewer could be more specific on what is missing in the main text. Since all our theoretical derivations and numerical results have been already included in the main text, we did not include an Appendix in the submission. According to the conference rules, submitting an Appendix is not mandatory for the submissions. We will be happy to answer any follow-up questions the reviewer might have on what he/she thinks is missing in the paper. Also, we have provided our code in the revision as a supplementary file, which can be used to verify the numerical results if necessary.

We thank Reviewer 8FsD for his/her time and feedback on our work. The following is our response to the reviewer's comments and questions.

1. Quantity after Equation (4).

We thank the reviewer for pointing out the typo in the quantity after Equation (4). In the revision, we corrected the typo and replace $I(X,A|R_\psi)$ with $I(X,A;R_\psi)$.

2. Derivation of Theorem 3.

We note that the derivation of Theorem 3 has been already explained in Section 4.2. Based on the reviewer's question, we have further discussed the derivation of Theorem 3 in Section 4.2 of the revised text.

3. Selection of $f$-divergences.

Different $f$-divergences have been shown to lead to different convergence and generalization behaviors [1, 2]. As a result, a proper choice of $f$-divergence could heavily depend on the target RL setting. This point is confirmed by the results in Table 2 showing that the optimal performance in different RL problems can ne attained by the algorithms based on different $f$-divergences.

[1] Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems 29 (2016).

[2] Rubenstein, Paul, et al. "Practical and consistent estimation of f-divergences." Advances in Neural Information Processing Systems 32 (2019).

We thank Reviewer kBCY for his/her time and feedback on our work. Here is our response to the reviewers' comments and questions:

1. "How much the KL approximation optimum can be far from the original optimal solution?"

We follow a standard approach to apply variational analysis to estimating the mutual information. In Proposition 2, if the function class $\mathcal{T}$ includes all functions, then the KL approximation is tight. As our proposed algorithm leverages neural networks and gradient-based method, we adopt the parameterized function class. We note that lower bounding the mutual information by the variational optimization is standard [1, 2]. For a comprehensive review on this topic, please refer to the work of [3].

[1] Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems 29 (2016).

[2] Belghazi, Mohamed Ishmael, et al. "Mutual information neural estimation." International conference on machine learning. PMLR, 2018.

[3] Nguyen, XuanLong, Martin J. Wainwright, and Michael I. Jordan. "Estimating divergence functionals and the likelihood ratio by convex risk minimization." IEEE Transactions on Information Theory 56.11 (2010): 5847-5861.

2. Finite-Sample Guarantees.

Please refer to our general response.

3. Numerical results in the noiseless setting.

We note that the algorithm proposed by Xie et al. (2021) is designed for the noiseless IGL setting, while our proposed algorithm $f$-VI-IGL is designed to address the RL task under a noisy feedback, which is often the case in real-world scenarios. As we observed in our numerical results, while Xie et al.'s algorithm attains better numerical results in the noiseless setting, its performance can degrade significantly under a noisy observation of the feedback variable. In contrast, our proposed method managed to achieve better results in the noisy case.

4. Computational costs of the proposed RL method.

While our proposed $f$-VI-IGL algorithm requires training of deep neural nets to estimate the mutual information terms in the objective, we observed that the training often enjoys a fast convergence. In our experiments, the training times of $f$-VI-IGL were in worst case $2\times$ more than the training time of Xie et al.'s algorithm, while achieving satisfactory numerical results in the noisy IGL environment.

We thank Reviewer 1D19 for his/her time and feedback on our work. The following is our response to the reviewer's comments and questions.

1. Finite-sample guarantees

Please refer to our general response.