What can we learn from Harry Potter? An Exploratory Study of Visual Representation Learning from Atypical Videos

Although the paper is readable, the writing quality is low (grammatical mistakes, convoluted writing). Overall, the presentation quality is low (organization of the manuscript, completeness of the captions, notation, clarity, etc.).

The contribution is overclaimed in the abstract/introduction. The paper only show results on OOD (and even that in an extremely narrow setting) but claim a contribution to "visual representation learning in the open world".

The proposed "atypical" video dataset is mostly a combination of existing, public datasets. The authors collect some videos from Sci-Fi movies and animation, but seems like they will not be able to share this data (it is collected from YouTube and Hollywood movies neither of which allow re-distribution by 3rd parties). As such, there is no dataset contribution in the paper.

The dataset collection protocol is not described in sufficient detail (How the "super-categories" were selected? How the individual samples were selected for each "super-category"?) Also, the dataset is too tiny to support any representation learning claims (fewer than 6k videos).

Overview of existing video datasets doesn’t include the most recent, large-scale efforts (e.g. WebVid).

Lots of important details are missing or aren't clearly described. For example, the notation is incomplete/inconsistent: L_OE is not defined (which is the key objective in the approach), the original loss is denoted inconsistently in the equations and in the text. The notation in Table 3 and Figures 4, 5 is not defined. Gaussian noise dataset is not described in sufficient detail, which dataset is used as an original to add noise to? How exactly the amount of noise to add is determined? For some reason a new outlier dataset (diving48) is introduced inside the experiments section. It is unclear how the outlier samples are introduced during fine-tuning (e.g. is there some sample balancing between outlier and in-distribution samples?).

Outlier exposure datasets are either much larger (Kinetics400) than the in-distribution UCF-101 dataset or comparable in scale (proposed Atypical), which is not a realistic scenario. Nota that these datasets need to be labeled with action categories, because they cannot include samples from the training distribution. In practice, in a representation learning scenario, one would want to use the vast majority of the labeling effort for in-distribution data.

It is unclear why the evaluation of the effect of each datasource in Table 3 only considers pairs of data-source, and never reports the effect of each individual data source separately. On the same note, to fairly compare individual data sources, their size has to be made uniform first. Otherwise it is impossible to claim that the largest source (e.g. Oops) leads to better results because of its content, not simply because of its larger scale.

The biggest issue with this work is that the contribution seems to be minimal, if it exists at all. Is it in the observation that more diverse OOD data during training helps to better detect OOD samples at test time? This is hardly surprising/novel. Moreover, the experimental setting is too narrow to make even this unoriginal conclusion. Strictly speaking, this paper shows that using Opps + data which is very different in appearance from standard action recognition datasets (e.g. animation) is (slightly) better than using Kinetics400 when trying to learn OOD detection on UCF. And even this narrow conclusion is not clearly established because the experimental setup is somewhat flawed (see comments above). No recipy for automatically collecting/selecting useful OOD training data is provided so it is unclear how to generalize this approach to other scenarios.

Major: The paper is lacking in novelty and is applying known methods on known datasets. This would fit better in an applications track at a conference rather than a general research track since there isn’t much novel about the method or the datasets. This rise to the level of novelty required to be published at ICRL or similar conferences.

Authors need to cite Terry Boult’s work where “atypical” are called “known unknowns” and aid in detection and have been around even before this works cited here: Abhijit Bendale, Terrance E. Boult; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1563-1572

Equation 2 has many undefined elements that are crucial to understanding the work. What is LOE? This equation is taken from the Hendrycks paper but you didn’t include any of the accompanying references to Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. International Conference on Learning Representations, 2018 who came up with the loss you are using here. These need to be included to make this understandable.

Why did you stop at OOD rather than do OSR? What is the benefit of not classifying the known data? It would be interesting to explore if using this “atypical data” would hurt the known class classification to explore the tradeoffs with this kind of data.

The noise to create OOD is very close to many adversarial work to show robustness or to attack networks. For example: Jiang, Linxi, Xingjun Ma, Shaoxiang Chen, James Bailey, and Yu-Gang Jiang. "Black-box adversarial attacks on video recognition models." In Proceedings of the 27th ACM International Conference on Multimedia, pp. 864-872. 2019. This is related to this approach since you are using this type of noise to determine OOD.

Minor: Line 126ish: OSR has an OOD problem within it. OSR is a two step process where the first step is to do OOD and then, if from a known class, classify it. OOD could be considered an anomaly detection task as well though your definition above (Line 143) says that you are more focused on class labels.

Figure 4, please add horizontal lines.

Line 269, you are saying it is difficult but that means it is possible. Are you actually stating this is possible for real-world applications?

1. Very limited experiments - fine-tuning only vanilla ResNet, with one in-distribution dataset and showing improvement on that is not enough at all. There are a lot of strong models in existing literature that do OOD detection with high robustness to outliers. To show effectiveness of the proposed atypical dataset, need a much more extensive experiments on stronger models and more in-distribution datasets.

   2. Missing quantitative evaluations - Randomly combining some of the 2,3 categories of atypical dataset does not give any meaningful result. To get a more meaningful performance, need to show all combinations of categories. Moreover, the mean result across datasets is not a meaningful quantitative performance because of the difference in data distribution, performance across different such datasets cannot be averaged.

   3. The generation of the dataset is not well motivated enough. Sci-fi and animation data is non-existent in real-world scenario, so having these as OOD samples and claiming it will generalize open-world OOD detection better is too far-fetched and not supported by quantitative evaluation. Fine-tuning the model on only these categories has worse performance than baseline (Figure 4, Table 4), which again proves that introduction of these samples are not helping the model in any way.

   4. The dataset statistics is incomprehensive - important explanation about how videos were selected for unintentional and abnormal category from existing datasets, how frames were sampled, why the number and video length of unintentional category is much higher than others etc is missing. These important details about the skew in data distribution might drive a better analysis of performance for this category.

   5. The effect of fine-tuning with Gaussian noise, diving48 and K400 is not well explained. No extensive analysis provided on those datasets about how they are not enough and why atypical is a more effective OOD dataset than these for outlier exposure? Moreover, fine-tuning with Diving48 already gives much better performance than fine-tuning with atypical dataset. This invalidates the effectiveness of the proposed atypical dataset.

   6. Formatting and readability issues - what most of the symbols denote is not mentioned in table captions. Redundant figures (figure 4 and 5) that provide no new information. Extremely small font on figures and placement issues hamper readability. Moreover, baseline performance not being present in Table 3, 4, 5 causes severe readability issues.

• The experimental results are insufficient.
• There is a lack of insight regarding the core atypical data.