Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli

Thank you very much for your review. We’re happy you find our paper to have “good contribution and novelty” and to provide a “good experimental analysis”. In the following we would like to address your concerns.

Comparison to SOE-Net. Thank you for making us aware of the work by Hadji and Wildes. We agree that this work should be discussed, and are happy to include a comparison to SOE-Net in our paper. Regarding the architecture, we see the following similarities and differences compared to the Simoncelli & Heeger model:

• Both models use a separable Gaussian derivative filter, divisive normalization, a variant of quadratic nonlinearity, and Gaussian spatial pooling. Implementation details for all components differ, nevertheless we expect these components to behave very similarly.
• Larger differences exist in the overall architecture: The Simoncelli & Heeger model is applied in parallel at different spatial scales, while spatio-temporal processing stages are applied sequentially in SOE-Net. Moreover, SOE-Net does not include a stage corresponding to MT, which is used by the Simoncelli & Heeger model to resolve ambiguities in the local motion signals.
• The processing stages of the Simoncelli & Heeger model have been experimentally compared to neural activity. This allows us to establish a compelling link between neural mechanisms and high-level behavior. To our knowledge, it has not been established how well SOE-Net matches computations in the brain.
• The memory and computational footprint is much higher for SOE-Net. We see two main reasons for these differences: (1) Each spatial scale in the Simoncelli & Heeger model reduces the input size by a factor of two, while the SOE-Net with default parameters uses larger feature maps (only a single reduction by a factor of 2 in the 3rd layer). (2) The layers in the Simoncelli & Heeger model use at most 28 feature maps, while SOE-Net uses 20, 400, 800 and 1600 feature maps, respectively. Taken together, these differences make it challenging to use the model for dense prediction tasks from an engineering perspective.

We reimplemented SOE-Net using PyTorch, but needed to make small adjustments to reduce the memory footprint for our dense prediction setting:

• We used a stride of 2 in the spatial pooling for each layer. The original model only decreased the resolution after the third application of the spatio-temporal model. This change also means that the model predicts the segmentation mask in only half of the input resolution. We use bilinear interpolation to predict the full resolution segmentation.
• We replaced the global pooling in the last layer with spatial and cross-channel pooling as used in the previous layers.

We use the representations of SOE-Net after each layer as multi-scale input to our coarse-to-fine segmentation network. We trained segmentation models using two variants of SOE-Net, using a square nonlinearity and the two-path square nonlinearity, respectively. As the results below show, the model performs similarly as the Simoncelli & Heeger model on the original videos. The performance gap might be explained by the reduced resolution of SOE-Net. However, SOE-Net does not generalize to random dot stimuli. We didn’t have enough time to perform further analyses, so we can only speculate on what causes this difference.

Standard computer vision benchmarks. We agree that scaling our approach to standard computer vision datasets would have been interesting and continue to explore this direction. Optical flow estimators have been heavily improved over the recent years while motion energy approaches have received hardly any attention in comparison. While we think that scaling motion energy to standard computer vision benchmarks is possible, we think the engineering effort required for that goal is beyond the scope of this work. As discussed in our general response, we see the main focus of our work in showing that a classical motion energy approach has more similar inductive biases compared to humans than state-of-the-art optical flow estimators. We believe that our dataset is adequate to support this claim.

Influence of training data. To rule out a strong influence of the training data we additionally evaluated the OCLR model (Xie et al. 2022) on the random dot stimuli. To our knowledge, this is the state-of-the-art motion appearance free (i.e. purely optical flow based) motion segmentation model. This model uses optical flow predicted by RAFT using several displacements and was trained for strong performance on standard computer vision benchmarks. As the results below show, the model transfers well to our clean stimuli but fails to generalize to the random dot stimuli. This result suggests that the choice of training data is not critical.

Thank you very much for your positive review. We are happy that you think that our work “can open a new line of investigation to model human motion perception”.

Human-machine comparison. We see our study as an ideal observer analysis and argue that humans are not expected to reach the model performance. Several participants of our study reported accidental mistakes and missed stimuli due to slipping attention during the experiment. Both are no concerns for the model. Moreover, the model computation is not affected by noise in the same way as biological systems. Finally, while the motion energy model is motivated from neuroscience the segmentation model is not and might use information more efficiently than humans. Our analysis therefore is an ideal observer analysis (Burge, Annu. Rev. Vis. Sci. 2020). This analysis shows that optical flow methods do not extract sufficient information to explain human performance while the motion energy model does. In our view this is clear evidence in favor of the motion energy approach. We agree that this point deserves more attention and will include it in the discussion of our paper.

Performance of online study participants. Several factors probably influenced the low performance of the participants in the online study. First of all, the viewing conditions (contrast, stimulus size, brightness of the surrounding, …) were not controlled so that participants might have taken the study in a difficult setting. Moreover, we did pay subjects independently of their performance. Since the performance depends on concentration and motivation, the participants might not have entirely fulfilled their potential. As promised in the paper, we repeated the study with more people in a controlled lab setting, where participants reached consistent performances (results in the PDF). These results provide evidence against the possibility of large subjective differences.

Additional ablations. Following your suggestions, we performed several additional ablations on the Motion Energy model: We removed the entire MT block, the MT linear layer, the pooling layers, and replaced the types of normalization and nonlinearities used in the model. The results below suggest that among the ablated components, pooling and normalization are particularly important for generalization to random dots. As discussed in our response to reviewer 8R4V, we think that the role of the pooling layers is particularly interesting regarding future improvements of the model. We will include the additional ablations in our paper.

Inconsistency between tables 1 and 2. Thank you for making us aware of this inconsistency. We used a slightly different architecture of the segmentation network for the ablation study in Table 2. We agree that this is suboptimal, and repeated the ablation study using the very same setup as for Table 1 for which we obtain consistent results.

Learning of motion perception in the brain. We agree that this is a highly interesting question and are happy to extend the discussion of this result in the paper. While we cannot offer definitive truth, we have hypotheses that might inform future work in this direction:

• We only use motion information for segmentation, while the motion based tasks performed by the brain are much more diverse (e.g., Nishida et al., Annu. Rev. Vis. Sci. 2018). Training on a more exhaustive set of tasks might be important for obtaining the parameters in the first place.
• There might be additional inductive biases, e.g. due to the biology of the brain. In our ablation study we did not restrict the trained parameters in any way. So there is the possibility to substantially deviate from the mechanisms of the original model, e.g. beyond the spatio-temporal energy approach. Inductive biases are assumed to be central for learning in the human brain (Goyal and Bengio, 2022).

Thank you for your detailed review. We are happy that you see “many strengths” in our work.

Comparison to Yang et al. (2023). Indeed, this work is very related since it also compares various motion estimation models to human perception. Thank you for making us aware of this paper. The focus of Yang et al. is however on naturalistic stimuli and does not consider segmentation, which limits direct comparisons. Interestingly however, Yang et al. find that FlowNet 2.0 agrees better with human perception than the more recent RAFT model which parallels the results in our work. We will extend the related work and discussion sections in our paper to include a comparison to the results of Yang et al. We’re not entirely certain, however, whether we correctly understood the suggested comparison to the pooling methods. The pooling baselines from Yang et al. use ground truth optical flow as input, so in our setting the performance is equal for the original and random dot videos by design—which is much closer to human perception than RAFT. Please correct us if we misunderstood your suggestion, we’re happy to include additional experiments in this direction.

Additional ablations. Following your suggestion, we performed additional ablation studies by removing or replacing components of the motion energy. We provide detailed results in our response to reviewer DKrM. The results suggest that among the ablated components, pooling and normalization are particularly important for generalization to random dots—which confirms your suggestion that the pooling is an important component which deserves further focus. Interestingly, removing the pooling slightly improved the results on the original videos, at the expense of much worse generalization. Gaussian pooling is a relatively simple method which does not take into account object boundaries. It has been suggested that the brain uses a more sophisticated approach (e.g., Braddick et al. 1993), so we think that improving the pooling is promising for further improving the model in the future. We will include the additional results and discussion in our paper.

More recent optical flow models. We included FlowFormer++ and GMFlow in our analysis and provide results in the additional PDF page. Both models support the claim of our paper: FlowFormer++ performs extremely well on clean stimuli but does not generalize to the random dot stimuli at all. GMFlow is among the best optical flow models that we tested, but like FlowNet 2.0 still falls substantially short of the motion energy model on random dot stimuli. Nevertheless, we believe this is a very interesting result since it is an exception to the trend that more recent models generalize worse which was observed so far. We are thankful for the suggestion and will include all results in our paper.

Finetuning on random dots. We expect that training optical flow estimators on random dot stimuli would most likely improve their performance. However, as argued in the general response, the inductive biases of human motion perception allows for zero-shot generalization to random dot stimuli which recent optical flow models do not exhibit—and which is not resolved by finetuning. At the same time, we think it is unlikely that the differences between the motion energy model and the optical flow methods can be fully explained by manual overfitting to random dots. In particular, the model by Simoncelli & Heeger has not been optimized on random dot stimuli according to the original paper. As discussed in our response to reviewer DKrM, we think that the result that training the Simoncelli & Heeger model does not lead to a generalizable model stems from missing inductive biases or too narrow tasks. For example, we expect that including an explicit loss term for the motion predicted by the model will improve the results.

Scaling the motion energy model. We see practical steps for improvement in combining the strengths of both approaches. Optical flow estimators exploit the rich information of deep feature representations, which are lacking in the motion energy model. At the same time, the spatio-temporal approach of the motion energy model which goes beyond the dominant two-frame paradigm turned out to be highly effective. We believe that combining these approaches is possible and will allow for both high-performing and human-aligned motion perception models. We agree that a discussion of this aspect is helpful, and will update our paper in this regard.

Differences between human performances and the motion energy model. As argued in our response to reviewer DKrM, we see our experiment as an ideal observer analysis. Machine algorithms do not suffer from accidental mistakes, lack of concentration or noise which is inherent to biological systems. These effects could be modeled by introducing corresponding noise in the model. Moreover, as suggested in your review, we think that the lack of appearance cues in our model also could also explain some of the differences: While appearance cues are not informative for the random dot stimuli, the brain has to explicitly ignore these cues which might make this task more complex for humans.

Alignment of computer vision and perceptual models. We agree that the goals of modeling human perception and building computer vision models of motion estimation are not completely aligned. Motion illusions, like the one you mentioned, are an important feature for models of human perception but might be considered errors in a computer vision setting. However, we think that both goals are sufficiently close to be mutually beneficial. Moreover, motion estimation is usually only a means to an end for downstream tasks. The types of illusions that humans are sensitive to don’t seem to be harmful for overall perceptual performance. So research on motion illusions might reveal “acceptable errors” for real-time and energy efficient systems that behave successfully in the real world.

We thank you for your thoughtful review. As argued in our general response, we see the main contribution of our work not to computer vision but to cognitive neuroscience; by establishing a compelling link between a biologically motivated model and high-level inductive biases of human perception.

Defining motion energy and optical flow. Thank you for the suggestion. We will update our paper to review the concepts of motion energy and feature-matching based optical flow. In a nutshell, we’re comparing two approaches for motion estimation: Matching features between two frames and using spatio-temporally oriented filters (3d convolution). This is also the difference we ment in our statement in L.64, which we will clarify accordingly.

Comparison to optical flow estimators. Deep learning based optical flow estimators have achieved remarkable accuracy during recent years. Due to this success, we find it natural to consider this class of models for our goal of building a model for motion perception in humans. While the goals of computer vision and modeling human vision differ in some aspects (for example regarding motion illusions) we believe that both lines of work are sufficiently close to be mutually beneficial.

Number of input frames. The different numbers of frames used by motion energy and optical flow methods is a valid concern in our view. We therefore conducted an additional ablation study in which we applied optical flow using the same number of frames as motion energy. As is common for motion segmentation, we are using multiple gaps of [-4, -3, …, +3, +4] to the central frame and using the concatenated optical flow as input to the segmentation network. The results in the additional PDF show that optical flow models are unable to achieve the same generalization to random dot stimuli as motion energy even when using the same number of frames.

Relevance of random dot stimuli. We agree that the “true” optical flow for random dots is ambiguous and believe that this makes random dot stimuli particularly interesting. The consistent perception of humans demonstrates that the sparse information of these stimuli is enough to perceive shapes, which reflects inductive biases of the underlying motion perception system. While the physically correct prediction might not be well defined, a good model for human perception clearly should reflect these biases. Moreover, random dot stimuli are well established in the literature on human motion perception. If models handle these stimuli similar to humans, this opens many opportunities for comparison with previous data.

Relevance for computer vision. The main goal of our work is to improve models for human motion perception and interpretation. While clean videos are more relevant for practical applications of computer vision, random dot stimuli are a highly important stimulus class in human vision. Nevertheless, we think that this study also provides some directions for building better computer vision models. Given that the motion energy model has several orders of magnitude fewer parameters compared to the optical flow models (<2.5K), the model performs remarkably effectively. We hypothesize that the spatiotemporal filtering efficiently aggregates information from multiple frames, which is an operation that could well be included with computer vision models and help to identify more efficient architectures.

Contribution. We argue that our study makes several valuable contributions for modeling human perception and fits the “neuroscience and cognitive science” subject area of NeurIPS very well.

• The motion energy model was not developed as an image-computable model for end-to-end machine learning, but to explain the behavior of individual neurons. We show that this model can nevertheless be used as a DNN component for the computer vision task of motion segmentation.
• Our knowledge of how humans organize motion patterns into a higher-level scene representation is still limited (e.g., Nishida et al. 2018). Our work makes a valuable contribution to this end by showing how a well understood, low-level mechanistic model of neural activity leads to inductive biases that agree with human scene perception.
• Random dot stimuli have a long tradition in vision science. Nevertheless we are the first to measure segmentation of random dot stimuli in humans. We believe that this experimental data is a valuable contribution to the psychophysics of human perception. The other reviewers seem to agree with us on these points and write that “this paper provides a valuable set of experiments that can guide future work in this interdisciplinary area” (834V), denote our work as a “Good contribution and novelty” (crYq) and posit that it “can open a new line of investigation to model human motion perception” (DKrM).

Additional Remarks. Thank you for your feedback, we’re happy to improve the clarity of our paper in these regards.

• Our new dataset consist of 901 training and 100 test videos, each having 90 frames at 30Hz. We use distinct backgrounds and foreground objects for both subsets. Models predict a binary segmentation with distinct foreground and background channels. So the IoU and F-Score metrics are computed as in for semantic segmentation, but using 2 classes.
• Motion energy and optical flow are not normalized or transformed.
• We argue that moving object segmentation is exactly the task which is targeted in our work, but are happy to clarify that we’re focussing on the generalization to random dot stimuli.
• Compared to humans, our model does not suffer from accidental mistakes and slipping attention. Please also see our response to reviewer DKrM where we discussed this aspect in greater detail.