L-WISE: Boosting Human Visual Category Learning Through Model-Based Image Selection and Enhancement

We thank the reviewer for their time and effort in reviewing our submission, and for their insightful feedback, which has led to improvements in our manuscript.

Weaknesses

1. The establishment of the empirical observations is somewhat unconvincing. Do these observations hold in more complex classification tasks or when applied to medical imaging?

We assume that the reviewer is referring to panels A1 and B1 of Figure 1, which show that robust ResNet-50 models can predict image difficulty and generate perturbations that increase image categorization accuracy on a 16-way animal classification task derived from ImageNet. To address the reviewer’s concerns, we have added Figure S11 to the Appendix, extending image difficulty prediction and image enhancement observations to the three fine-grained image category learning experiments. Paragraphs 2 and 3 of Appendix Section S8, which are newly added, read as follows:

“

In addition to the agreement of expert MHIST annotators, the ground truth logit successfully predicts the proportion of human participants who select the correct ground truth label across all tasks we tested. Difficulty prediction results from the 16-way ImageNet task are shown in Fig. 1 in the main text, and from the moth photograph, dermoscopy, and histology tasks in Fig. S11 (Panels A1, B1, and C). For the non-ImageNet tasks, we rely on test-phase data from control group participants who had just learned the tasks in question.

We can also attempt to measure the extent to which images with higher levels of enhancement are easier for novice participants to recognize during the learning tasks (Fig. S11A2,B2). This analysis is limited to the first training trial blocks in the “ET (shuffled)” participant group in the ablation study (main-text Table 1), the only group that viewed enhanced images without monotonically decreasing ϵ. Note that there were 6 discrete ϵ values (1 per block in the non-shuffled ET condition), and the analysis is complicated by the fact that participants were still learning the task when they made the responses underlying these plots. We are also unable to compare with ϵ = 0 unmodified images because participants did not view new unmodified images in the corresponding training blocks. There are no results here for the histology task because the ablation study was conducted only for moth photos and dermoscopy images. In the dermoscopy task, participants respond with the original, correct category label statistically significantly more often when viewing images enhanced with greater ϵ (Fig. S11B2). A similar trend was not statistically significant for the moth photograph task (Fig. S11B1), perhaps due to the limitations of this particular analysis outlined above. We can observe strong effects of image enhancement in both the moth photograph and dermoscopy tasks by examining the enhancement only (“Enhancement Taper/ET”) arm of the ablation study (Table 1 in the main text): for both tasks, participants in the ET condition had higher test-phase accuracy than participants in the control conditions, and these differences were statistically significant.

“

Furthermore, we have updated the analysis of image difficulty prediction in the appendix (please see Section S5 and Figure S4) to show more comprehensively that our human image difficulty prediction approach outperforms the prior state-of-the-art established by Mayo et al. 2023 [1].

Finally, regarding image enhancement results in ImageNet, we have independently replicated the finding that human participants are significantly more accurate at classifying images enhanced by robust ANNs than the original, unmodified images, in 3 separate experiments with 3 sets of participants. The first is Figure 1, the second is the newly added Figure S6 in the Appendix (which compares enhancements from various ANN guide models), and the third is in Appendix Figure S8 (logit-maximization enhancement works on both validation set and training set images, and is superior to cross-entropy-minimization enhancement).

References:

[1] Mayo, David, Jesse Cummings, Xinyu Lin, Dan Gutfreund, Boris Katz, and Andrei Barbu. "How hard are computer vision datasets? Calibrating dataset difficulty to viewing time." Advances in Neural Information Processing Systems 36 (2023): 11008-11036.

2. Beyond the empirical observations, is there any physiological insight or analysis on why the proposed model-based selection and enhancement method could improve novice learners’ performance in categorization tasks?

We thank the reviewer for raising this question. It would be very interesting to (for example) conduct neurophysiological recordings while participants complete the learning tasks. Ultimately, however, the goal of this study was not to investigate the physiological mechanisms underlying visual perception or learning.

The guiding principles behind designing L-WISE were not based on neurophysiology or neuroscience per se, but were rather established by behavioral observations from many past studies of human learning. Many studies have found that beginning a learning task at a low level of difficulty and then gradually increasing it while keeping the difficulty at a manageable level increases the speed of learning [1]. In particular, this type of ``curriculum effect’’ has been observed in perceptual learning of simple visual perceptual tasks, such as line orientation discrimination [2]: this led us to hypothesize that a similar effect could be observed with more complex and highly challenging visual perceptual learning tasks such as those involving medical images. L-WISE is designed to begin the learning process at a manageable difficulty level by selecting relatively easy images (based on image difficulty predictions) and enhancing them using models to reduce the difficulty further and highlight important features - it then gradually increases the difficulty by including more and more difficult images and enhancing them by perturbations limited to smaller and smaller $\epsilon$ values (see Figure 2, Panels E-F). We believe that this easy-to-hard curriculum effect at least partly explains L-WISE’s demonstrated efficacy at augmenting visual learning.

References:

[1] Green, C. Shawn, and Daphne Bavelier. "Exercising your brain: a review of human brain plasticity and training-induced learning." Psychology and aging 23, no. 4 (2008): 692.

[2] Lu, Zhong-Lin, and Barbara Anne Dosher. "Current directions in visual perceptual learning." Nature reviews psychology 1, no. 11 (2022): 654-668.

3. The authors do not discuss the related machine teaching literature. An in-depth comparison with machine teaching methods, particularly with "Teaching Categories to Human Learners with Visual Explanations" (CVPR 2018), would be valuable. This work similarly considers image difficulty; an introduction and comparison with it are beneficial.

We thank the reviewer for these suggestions - we were not aware of the suggested CVPR 2018 paper, which is indeed highly relevant. We have added a dedicated Related Work section to more comprehensively discuss prior works in machine teaching as well as difficulty prediction, image enhancement, and aligning ANN models with humans.

4. The authors should also discuss the connection to human-machine vision alignment methods, such as "Harmonizing the Object Recognition Strategies of Deep Neural Networks with Humans" (NeurIPS 2022)

We thank the reviewer for this suggestion. We have included this paper and other papers that discuss human-machine vision alignment in our expanded Related Work section.

5. The sample size of participants is relatively small, and expanding the participant pool is recommended to enhance the reliability of the results; also, recruiting participants from diverse backgrounds would improve the generalizability of the findings. If expanding the participant pool is impractical due to time or budget constraints, performing a power analysis or discussing effect sizes could help strengthen the reliability of the analysis.

Our experiments involved more than 500 human participants with a minimum of 27 in each group for the learning experiments, and approximately 168 trials (learning experiments) or 352 trials (ImageNet recognition experiments) per participant. This is a high number of participants compared to most psychophysics papers. Our decisions regarding the number of participants was guided by a power analysis, which we have now included in Appendix Section S11. We report the results of statistical tests for all of the findings.

We recruited participants who are diverse in age, gender, and race/ethnicity (see new Table S2 in the Appendix).

3. Did you collect qualitative feedbacks from participants? Did the new curriculum and enhanced images increase mental stress of human learners? Additional learning costs beyond training time should be considered, such as cognitive load and emotional well-being.

We agree with the reviewer that factors such as mental/emotional stress and cognitive load are crucially important, particularly if techniques similar to L-WISE are used in educational settings in the future. The present study was focused on proof-of-concept rather than practical applications, and as such we did not formally collect qualitative feedback. In future work that is more oriented towards classroom/clinical learning, we will systematically collect qualitative feedback - this is an excellent suggestion.

We predict that the effects of L-WISE on stress and cognitive load in an educational setting would actually be positive, not negative. We were motivated to create L-WISE by observations that certain visual tasks, particularly in the clinical realm, are very difficult for beginners and thus frustrating to learn at the early stages. L-WISE ameliorates this by selecting easier images for the beginning of the learning process, and also enhancing them to make it easier to recognize important features. Many studies on human learning have found that beginning a learning task at a low level of difficulty and then gradually increasing it while keeping the difficulty at a manageable level, increases both the speed of learning and the motivation of learners [1]. One thing we can measure in our experiments is how often participants withdraw from the experiment before completing the whole learning task, a common occurrence in online studies where humans attempt to learn difficult tasks [2]. We present a new analysis of task withdrawal rates by condition in Appendix Section S9. Participants who were assisted by L-WISE were significantly more likely (p < 0.05) to complete the entire learning task, suggesting that they found the task more interesting and/or less frustrating than participants in the control group without L-WISE assistance.

References:

[1] Green, C. Shawn, and Daphne Bavelier. "Exercising your brain: a review of human brain plasticity and training-induced learning." Psychology and aging 23, no. 4 (2008): 692.

[2] Wang, Pei, Kabir Nagrecha, and Nuno Vasconcelos. "Gradient-based algorithms for machine teaching." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1387-1396. 2021.

Ethics

1. The use of ANN-guided models in human learning may introduce unintended biases, potentially affecting participants' learning outcomes in ways that favor certain demographic groups over others. For instance, if the training of ANNs inadvertently enhances accuracy for only a subset of the population (such as specific genders, ages, or races), it could lead to biased learning outcomes. Such biases could potentially skew related job opportunities, ultimately reinforcing inequities. Ensuring that ANNs benefit all demographic groups equitably requires further investigation and ongoing evaluation to mitigate these risks.

We thank the reviewer for raising this concern. Our recruited sample of participants was diverse in age, gender, and race/ethnicity - we have added a table of participant demographics to the Appendix (Table S2). In general, it is indeed a concerning possibility that AI-assisted learning could disproportionately benefit certain population groups due to the models capturing group-specific characteristics of visual processing. We argue that this type of bias in learning assistance is highly unlikely for the types of visual tasks we focus on. There is no evidence of strong or fundamental differences in basic visual perception or learning among different genders or ethnic groups [1], with the possible exception of face recognition (which is not the focus of this study) [2]. L-WISE works by making challenging tasks easier at the beginning and then gradually increasing the difficulty - it would be quite surprising if this was fundamentally helpful for some demographic groups but not others. This study demonstrates that boosting human learning via model-generated image perturbations, previously unknown to be possible at all, is achievable and in a diverse sample of participants. Future studies will be needed to evaluate potential biases such as those raised here.

References:

[1] Kreiman, Gabriel. Biological and computer vision. Cambridge University Press, 2021.

[2] Young, Steven G., Kurt Hugenberg, Michael J. Bernstein, and Donald F. Sacco. "Perception and motivation in face recognition: A critical review of theories of the cross-race effect." Personality and Social Psychology Review 16, no. 2 (2012): 116-142.

3. Uses only the ResNet-50 architecture, limiting generalization; further testing with models like vision transformers (ViT) is needed to support broader conclusions.

This work shows that a reasonable model of human vision, which is thought to be among the leading models of the ventral visual stream, can be used to enhance image category learning in humans; we do not claim that human learning can be augmented using any ANN architecture, nor do we claim that robustified ResNet-50 is optimal for this goal. Our work is rather a proof-of-concept showing that augmenting human learning with an ANN is possible, and across varied and potentially useful image domains. To the best of our knowledge, this is the first work demonstrating the possibility of enhancing visual learning in humans by using any computational model to enhance images.

The choice of ANN architecture is one of many hyperparameters in L-WISE that could potentially be further optimized - others include the strength of image perturbations, the temporal profile of enhancement tapering and difficulty selection, the overall length of the curricula, and the data used for pretraining the ANN guide models. Hyperparameter tuning in this setting is time and capital-intensive given that dozens to hundreds of human participants must be recruited for each experiment. In our present work, we are focused on establishing proof-of-concept and leave further engineering optimizations for future works.

However, we agree with the reviewer that it is interesting to test different ANN architectures, particularly vision transformers given their recent success and widespread use. In this spirit, we conducted an entirely new experiment involving 60 human participants to compare the image enhancement and difficulty prediction capabilities of 6 different ANNs. Among them is robustified XCiT [1], a vision transformer architecture that is better suited to adversarial pretraining than the original ViT [2]. The perturbations from robustified XCiT improved accuracy comparably to robustified ResNet-50 (See Appendix Figure S6), and difficulty prediction accuracy is also comparable between the two models (Appendix Figure S5).

References:

[1] Ali, Alaaeldin, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev et al. "Xcit: Cross-covariance image transformers." Advances in neural information processing systems 34 (2021): 20014-20027.

[2] Debenedetti, Edoardo, Vikash Sehwag, and Prateek Mittal. "A light recipe to train robust vision transformers." In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 225-253. IEEE, 2023.

4. Image enhancement may introduce biases, potentially improving accuracy only for certain major classes; additional metrics like precision and recall per class, rather than just mean accuracy, should be reported to provide a clearer assessment.

The possibility of L-WISE improving recognition of some classes and not others is an important concern that was not addressed in our initial submission. We have added Panel B to Figure 4 (and Table S1 to the Appendix) showing that, while precision and recall do vary among classes, both precision and recall are improved by L-WISE for each class across all three image domains we tested. The reviewer’s intuition was correct that improvements to precision and recall, while universally positive, do appear larger in magnitude for some classes than others.

We thank the reviewer for their time and effort in reviewing our submission, and for their insightful feedback, which has led to improvements in our manuscript.

Weaknesses

1. Lacks a dedicated related work section, which would help contextualize the research.

We agree with the reviewer, and as suggested, we have now added a substantial Related Work section (Section 2), including discussion of many additional works related to image enhancement, machine teaching, and different ways of aligning models with human perception.

2. Both low and high logits from ANNs show significant variation in human accuracy, making predictions less reliable in certain logit intervals.

As indicated by the reviewer, we agree that the 95% confidence intervals are wider at very low and very high logit values. This, however, is not necessarily due to less reliable predictions but rather reflects the limited statistical power of our measurement in those logit ranges. There are fewer images with extreme logit values as the images are roughly normally distributed along this axis. To increase the statistical power even in the extreme logit value ranges, we have updated Panel A of Figure 1 to incorporate newly collected data from additional human participants, resulting in narrower confidence intervals throughout the logit range.

We also wish to emphasize that our image difficulty prediction approach outperforms the prior state-of-the-art in this regime by a considerable margin. In addition to the change to Figure 1, we have updated the analysis in the appendix (please see updated Section S5 and new Figure S4) to more comprehensively compare our human image difficulty prediction method with the prior state-of-the-art established by Mayo et al. 2023 [1]. In the newly expanded analysis, we demonstrate that the ground truth logit of a robust model is a significantly better predictor than the c-score [2] (approximated by the epoch during training at which an image is first correctly predicted), image-level adversarial robustness [1] (perturbation size required to change the model’s prediction of the image), and prediction depth [3] (earliest layer upon which a linear probe makes the same prediction as the final output) calculated using both vanilla and robustified models.

References:

[1] Mayo, David, Jesse Cummings, Xinyu Lin, Dan Gutfreund, Boris Katz, and Andrei Barbu. "How hard are computer vision datasets? Calibrating dataset difficulty to viewing time." Advances in Neural Information Processing Systems 36 (2023): 11008-11036.

[2] Jiang, Ziheng, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. "Characterizing Structural Regularities of Labeled Data in Overparameterized Models." In International Conference on Machine Learning, pp. 5034-5044. PMLR, 2021.

[3] Baldock, Robert, Hartmut Maennel, and Behnam Neyshabur. "Deep learning through the lens of example difficulty." Advances in Neural Information Processing Systems 34 (2021): 10876-10889.

3. The perturbations - The perturbations if I am not mistaken are very subtle ones. For fine grained classifications, humans do follow curriculum learning but learning structures gradually from simpler to harder concepts. No experiments have shown this. It would be great if the authors can provide some empirical results/ explanation that can explain how will their method occur when you focus on structural cues rather than model perturbations that will benefit fine grained categorization.

We agree with the reviewer about the importance of the curriculum order by which images are presented. That is why we designed L-WISE to progress from easy to hard examples, both by selecting easy images for the early part of the task (by predicting their difficulty with a robust model) and by decreasing the magnitude of enhancements as the task progresses (please see Figure 2, Panels E-F). We also explicitly study the effects of curriculum ordering in our ablation study of L-WISE (Table 1): regarding perturbations specifically, one can compare the performance of “enhancement taper (ET)” with “ET (shuffled)” - ET has a progression of more to less enhancement as the task progresses, while ET (shuffled) randomizes the order to erase this easy-to-hard trend. The fact that ET (shuffled) has lower performance than ET confirms that the decreasing enhancement schedule is helpful for learning.

The image enhancement approach used in this study does not explicitly separate out specific visual concepts, but rather makes them more salient for the learner to notice - the degree to which each feature is enhanced by the model depends by construction on the model’s gradients, which are a reflection of which features are most important according to the model [1]. An interesting finding of our study is that allowing the model to decide which features to highlight has strong, measurable benefits for learners. Whether the L-WISE strategy overlaps with the notion of graded structural cues, or whether the latter is an effective strategy to confer a learning boost at all, is not a claimed contribution of the present study.

References:

[1] Selvaraju, Ramprasaath R., Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. "Grad-CAM: visual explanations from deep networks via gradient-based localization." International journal of computer vision 128 (2020): 336-359.

4. I am providing some citations related to Gaze and dual stream hypothesis that can help authors clarify my concerns

We thank the reviewer for suggesting this very interesting paper (which we have now cited in our related work section), and we think it is a great suggestion to incorporate aspects of human gaze in conjunction with deep learning models. We are currently planning a study of human gaze and models incorporating it in the context of medical images, although that work is not ready for inclusion in the present manuscript.

We wish to emphasize that our current work is a proof-of-concept showing that, with use of a model that incorporates (even incompletely) some aspects of human visual perception, we can strongly improve visual recognition accuracy and learning performance by enhancing images and predicting their difficulty. This is an entirely novel finding. It is quite likely that the results could be improved even further with a more comprehensive model of the human visual system, and the reviewer’s suggestion of incorporating dual-stream neural networks is insightful and interesting.

Questions

1. For model perturbations, can the authors please provide heat maps or any qualitative results that will help us track the ANN perturbations to human visual learning?

Inspired by this suggestion, we have added heat maps and ``difference’’ images to figures 3-4, and to the newly added Appendix Figure S1. We believe this improves the interpretability aspect of the work: it can be observed qualitatively that the ANN perturbations are localized in semantically important regions of the images.

We thank the reviewer for their time and effort in reviewing our submission, and for their insightful feedback, which has led to improvements in our manuscript.

Weaknesses

1. The paper focused on the performance of ventral stream. But we know that the human visual stream has a dorsal stream (where) that locates an object and the ventral stream (what) stream. And the interplay of these two streams forms the basis of human visual system. In this work the authors mainly focused on the ventral stream. From only quantified data, we can see the gains but it is very hard to trace this back to the nuanced perturbations the ANN produces. Hence, the suggestion is to use human gaze. The human gaze will precisely pin-point the "where" aspect and then will truly help us understand if at all the model perturbations are helping improve human performance. Can the authors please explain this?

We agree with the reviewer that a complete model of visual processing and object recognition behavior would have to account for dorsal stream processing and gaze behavior. We are unsure whether the reviewer is referring to the categorization or learning experiments. In the 16-way animal categorization task (Figure 1), images are presented for only 17 ms in the central portion of the participants’ vision, which does not leave enough time for the participant to make any eye movements. This is an established approach for studying ``core object recognition’’ (recognition per saccade stations) in isolation from gaze-related effects [1].

We agree with the reviewer that the “where” aspect of the image perturbations is important and interesting, and that it was not obvious to identify where exactly some of the images had been modified in the examples provided. To help clarify this and address the reviewer’s feedback, we have added “difference” images and heat maps showing which parts of the images are modified the most by the enhancement technique (please see Figures 3-4, and Appendix Figure S1). The changes are concentrated in semantically important regions of the images.

We note that we can readily evaluate how model perturbations improve human performance. Our results demonstrate with a high degree of statistical reliability that category recognition accuracy is higher when images are enhanced by model perturbations in a task participants are already familiar with (Figure 1), and also that learners achieve higher accuracy on new, unfamiliar tasks, and in a shorter period of time, when they are trained with a curriculum that incorporates image enhancements (Figures 3 and 4). These effects have never been observed before, and we introduce these results as a proof-of-concept without claiming that the models we employed are optimal choices. As the reviewer suggests, a better model of human vision, such as one incorporating aspects of gaze or the dorsal stream, could potentially lead to an even more effective enhancement of image recognition and learning.

References:

[1] DiCarlo, James J., and David D. Cox. "Untangling invariant object recognition." Trends in cognitive sciences 11, no. 8 (2007): 333-341.

2. A robust DNN actually has worse performance on nominal data points. Data points that have not been corrupted adversarially. What was the motivation of the authors to select such a model for their experiments?

The reviewer is correct that robust DNNs have worse performance on data points that have not been adversarially corrupted: this is referred to as the “robustness-accuracy tradeoff.” Our motivation for selecting robust models is based on existing evidence that robust models can produce perturbations that have strong disruptive effects on human category perception, while perturbations obtained in the same way with non-robust models have negligible or no impact [1].

To quantify the effect of robustness on image enhancement, we have added Figure S6 to the Appendix - this figure shows that enhancement perturbations from robust models produce large improvements in image recognition accuracy, but perturbations of the same magnitude from non-robust (“vanilla”) models have no effect. We have also added Figure S7 to the Appendix which shows examples of image enhancements produced by robust and non-robust models, providing intuition on why this is the case. To quantify the impact of robustness on predicting the difficulty of images, we have expanded Figure S4 in the appendix. This figure shows that using robust models to predict the difficulty of each image for humans outperforms state-of-the-art approaches to predicting image difficulty using non-robust models.

References:

[1] Gaziv, Guy, Michael Lee, and James J. DiCarlo. "Strong and precise modulation of human percepts via robustified ANNs." Advances in Neural Information Processing Systems 36 (2024).