Novelty Detection in Reinforcement Learning with World Models

We thank the reviewer for their suggestions on improving our work. In response, we outline provisional revisions below:

First, we note that we have fixed each of the typos and syntax errors that were identified. Specifically, we have:

• Moved the period to after the citation.
• Rephrased Ln 92 to 96 - col 1 to: To develop the bound, we observe that, under nominal conditions, any divergence should be smaller than that of the predicted latent world state computed with the initial hidden state, as the latter prediction becomes increasingly inaccurate.
• Rephrased Ln 215 - clo 2 to: (blue differences collapse to orange).

Regarding the following statement:

Referencing Srivastava et al. 2014 for Proposition 3.2 is not enough, a little bit more explanation is needed to make the paper self-sufficient.

We first reword Proposition 3.2 to align with terminology used by other works [2]: If the cross entropy score comparison becomes nonnegative when introducing the vector $x_t$, then 4 defines a decision boundary in the latent space over the measure of robustness to partial destruction of the input (Vincent et al., 2008; Srivastava et al., 2014), i.e.:

$KL[p_{\phi}(z_t|h_t,x_t)||p_{\phi}(z_t|h_t)] + KL[p_{\phi}(z_t|h_t,x_t)||p_{\phi}(z_t|h_0,x_t)] \leq\\ KL[p_{\phi}(z_t|h_t,x_t)||p_{\phi}(z_t|h_0)]$.

and have extended the derivations section in the appendix to include a more explicit tone:

Let the full posterior $p_{\phi}(z_t|x_t,h_t)$ be the most informed distribution and the desired distribution with support over the latent space $\mathrm{Z}$ at any given $t$, and let $f_\theta$ map to the representation of a distribution with the support of $\mathrm{Z}$ as well.

Utilizing notation similar to that in Vincent et al. (2008), denote $(h_t,x_t)$ as the clean input $x$ (not to be confused with $x_t$), and define three variants of $x'$: $\left\lbrace x_{(-h_t)}', x_{(-x_t)}', x_{(-h_t,-x_t)}' \right\rbrace$, which denote the removal of features $(h_t)$, $(x_t)$, and $(h_t,x_t)$ from the clean input $x$, respectively.

Since the loss is at the latent level, we measure the robustness to partial destruction criterion of a single input $x$ for a desired distribution as: $L(p_{\phi}(z_t|h_t,x_t), f_{\theta}(x'))$

Therefore, if $L$ is chosen to be the KL divergence, and $x'$ becomes explicit, then we can derive Proposition 3.2 as:

$L(p_{\phi}(z_t|h_t,x_t), p_{\phi}(z_t|h_0)) \geq L(p_{\phi}(z_t|h_t,x_t), p_{\phi}(z_t|h_t)) + L(p_{\phi}(z_t|h_t,x_t), p_{\phi}(z_t|h_0,x_t)) =$ $KL(p_{\phi}(z_t|h_t,x_t) \parallel p_{\phi}(z_t|h_0)) \geq KL(p_{\phi}(z_t|h_t,x_t) \parallel p_{\phi}(z_t|h_t)) + KL(p_{\phi}(z_t|h_t,x_t) \parallel p_{\phi}(z_t|h_0,x_t))$

where $p_{\phi}$ corresponds to the distribution modeled by $f_{\theta}$ given some $x^{'}$, and $h_0$ represents the state of $h$ when no conditioning information is available.

The quality of the paper can be improved further if authors come up with some guarantee on the bound.

Finally as suggested, we agree that we should attempt to give the reader intuition surrounding our method by extending the appendix with the following theoretical guarantee which shows that for a theoretical perfectly trained world model (i.e: $z_t \perp x_t \mid h_t$ and there exists a set of parameters $\phi^*$ such that: $p_\phi(\cdot|\cdot) = p^*(\cdot|\cdot)$ where $p^*( \cdot | \cdot)$ denotes the well defined true conditional distribution) then EQ 4 holds if: $E_{p_\phi(z_t \mid h_t,x_t)}\left[\log \frac{p_\phi(z_t \mid h_0,x_t)}{p_\phi(z_t \mid h_0)}\right] \ge 0.$ Which can be interpreted as holding if the Expected Information Gain (EIG) of $x_t$ is nonnegative under the distribution of $p_\phi(z_t \mid h_t,x_t)$.

Ideally, although as an informal side note, we would like to intuitively interpret $KL[p(z_t |h_t x_t )|| p(z)]$ as proportional to $I(z_t;h_t,x_t)$ (which is often a substitution in practice [1]). Where

• $KL[p(z_t |h_t, x_t )|| p(z_t)]$ is substituted by $I(z_t;h_t,x_t) = H(z_t) - H(z_t|h_t,x_t)$
• $KL[p(z_t |h_t, x_t )|| p(z_t | h_t)]$ is substituted by $I(z_t; x_t | h_t) = H(z_t|h_t) - H(z_t|h_t,x_t)$
• $KL[p(z_t |h_t, x_t) || p(z_t | x_t)]$ is substituted by $I(z_t;h_t | x_t) = H(z_t|x_t) - H(z_t|h_t,x_t)$

Therefore, if showing (EQ 4) was somehow related to showing: $H(z_t) - H(z_t|h_t,x_t) \geq H(z_t|h_t)- H(z_t|h_t,x_t) + H(z_t|x_t)- H(z_t|h_t,x_t)$

Then simplification would show: $H(z_t) - H(z_t|h_t) \geq H(z_t|x_t) - H(z_t|h_t,x_t) \implies$

$I(z_t;h_t) \geq I(z_t;h_t|x_t) (*)$ Where a potential key result would lie: EQ 4 holds if knowing X reduces the mutual information between Z and H.

[1] Zhao, S., Song, J., & Ermon, S. (2019). InfoVAE: Balancing Learning and Inference in Variational Autoencoders. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 5885-5892.
[2] Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, Extracting and composing robust features with denoising autoencoders. ICML 2008

Thank you for your review, hopefully we can interpret your questions and clarify as well as possible:

• Why is the ground truth for Table 1 so noisy?
  • The ground truth component of the table shows the true observation given to the agent, specifically the noisy observation is a sample from the noisy environment that occurs when we inject the novelty ‘noise high’ in the humanoid setting.
• I don't understand this point "in nominal conditions, any divergence should always be smaller than the divergence of the predicted latent world state computed with the initial hidden state instead of the current initial hidden state; the latter prediction should be increasingly inaccurate." Can the authors elaborate on this?
  • Thanks for pointing out that this statement might be confusing to the reader, with that statement we are attempting to describe our observation in Figure 1. In response we have rephrased the statement to be: “to develop the bound, we observe that, under nominal conditions, any divergence should be smaller than that of the predicted latent world state computed with the initial hidden state, as the latter prediction becomes increasingly inaccurate.”
• The authors state "We introduce a novel technique for novelty detection without the need to manually specify thresholds, and without the need for additional augmented data." but then they later state "Our technique calculates a novelty threshold bound" so there seems to still be a reliance on the novelty threshold bound? I'm unsure clarification would be helpful.
  • No problem. To clarify, we are stating that the user does not have to be mindful of explicitly/manually setting a measurement of uncertainty $\lambda$ beforehand to trigger the detection of novelty. Our technique will simultaneously handle the bound (as well as the detection), while minimizing false positives.
• The authors used an older version of dreamer (V2 instead of V3) which begs the question--does the proposed approach work for newer MBRL based approaches? … Why did they use DreamerV2 instead of the newer DreamerV3?
  • We consider our work on the V2 algorithm to be mainly based on the RSSM component of the world model, and although the major contributions of going from the Tensorflow V2 to the JAX V3 introduced faster optimizations, this move would have made the experimental design more complex despite both V2 and V3 being centered around the core RSSM framework. Therefore, for newer MBRL approaches, we believed it was more impactful to take time to test the newer Diamond (2024) [1], and IRIS (2023) [2] world models, which unlike DreamerV3 or V2, do not use the RSSM framework.
• The proposed approach is not compatible with raw inputs (i.e. when using diffusion) or non probabilistic latents. I also see this as a small weakness.
  • For diffusion based approaches, we found that the PP-MARE method (see Table 7, Section 4 PP-MARE, and Appendix B paragraph Diamond) could successfully be used directly on raw inputs. As for non-probabilistic latents, (possibly used if it is expected that the environment is deterministic and fully observable), then our setting of being in a stochastic partially observable setting no longer holds. However, we hypothesize that a subtle extension could be that the KL loss could be substituted with a binary operator (or suitable deterministic metric) over the posterior and prior representations, when conditioned on different information.
• While the approach is simple, that also means the contribution is rather limited. I also see this as a small weakness
  • Thanks for the feedback. We purposely crafted the main method to be simple, but to the best of our knowledge, we are the first to test novelty detection in both Atari and 3D-DMC environments entirely in the image space (see footnote 2). We also introduce PP-MARE for those who are interested in working outside of the latent space, and are okay with specifically defining their threshold.

[1] Alonso, E., Jelley, A., Micheli, V., Kanervisto, A., Storkey, A., Pearce, T., and Fleuret, F. Diffusion for world modeling: Visual details matter in atari. 2024.
[2] Micheli, V., Alonso, E., and Fleuret, F. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations, 2023.

We thank the reviewer for their time in working to improve our presentation and their encouraging feedback. We are delighted that you took the time to assist the quality of our work. In response, we outline provisional revisions below:

• We agree that the discussion of C.2 may be more informative to the reader than the section Detection Score Trend Over Time, since the world model's final performance is directly tied to the exploration of the environment. Therefore, we have moved section C.2, Figure 7, and Table 8 into the main manuscript, and have moved Detection Score Trend Over Time, and Figure 2 into the appendix for those interested in visualizing the anomaly score trend. Section C.2 now appears after the section alternative world models.
• We agree that it is important to make a note to the reader about the downsides of not using a proper metric such as the KL divergence, and the method's dependence on variation approximations. In response, we've extended the derivations section to discuss explicit details of the dependence on variational approximations as well as assumptions on the properties of the model itself. We state: “We briefly note that KL divergence is not a true metric. The bound in Proposition 3.2 compares the posterior to three different priors and provides a useful approximation for detection, rather than a strict mathematical guarantee. Formally, for distributions P, Q, and R, the KL divergence does not generally satisfy $D_{KL}(P \parallel R) \leq D_{KL}(P \parallel Q) + D_{KL}(Q \parallel R)$. This means Eq 4 might be violated not only due to novelty but potentially due to the inherent non-metric behavior of KL divergence, particularly in high-dimensional or complex/untrained latent spaces (see exploration effects)”. In addition, we also redirect the reader to this discussion as we note the reliance of functional approximators in the limitations section, which reads: While our proposition using KL divergence has proven effective empirically, future work might explore alternative divergence measures with metric properties, such as the Jensen-Shannon divergence or Wasserstein distance, which could provide stronger theoretical guarantees. Additionally, exploring the relationship between latent space dimensionality and the reliability of variational approximations (of which our bound is based on see Appendix D) could yield insights into optimizing model architecture for novelty detection.
• We have summarized the contents of Appendix H in Appendix B to improve the self sufficiency of the paper, (which ties into C.2 Exploration effects) of which we discuss the exploration trick of weighing a single action heavily that IRIS incorporated to successfully earn a higher score (which could affect the performance of the world model). Specifically, we add to Appendix B: Although our initial interpretation had moderate success, we expect that future work should be able to improve by adjusting the bound as well as addressing the exploration concerns given the exploration trick used in IRIS, of which the "up" action was stated to be heavily oversampled in favor of improving the agent performance, rather than improving the world model performance. Ultimately, we leave the task of enhancing novelty detection in the IRIS world model for future work.