To Learn or Not to Learn, That is the Question — A Feature-Task Dual Learning Model of Perceptual Learning

Thanks for the very detailed comments. We streamline reviewers' main concerns and address them one-by-one.

Q1: On the biological plausibility of location-specific plasticity. Based on a number of experimental evidence, we believe that location-specific plasticity is possible in certain conditions.

• First, the Double Training experiments (Xiao et al. 2008, Zhang et al. 2010, Wang et al. 2012, Zhang et al. 2014, Wang et al. 2016, and Xiong et al. 2016 ) and those involving attention to induce transfer (Donovan et al. 2015, 2018, 2020) have indicated that the location-specific plasticity is possible. For instance, Wang et al. (2012) showed that "Double Training" involving stimuli unrelated to orientation, such as distinguishing between a Gabor stimulus and the letter E, could achieve transfer to untrained locations. Xiong et al. (2016) used two experimental paradigms: one in which subjects were presented stimuli without awareness (bottom-up driven), and the other subjects were informed of the presence of stimuli but no such stimuli actually presented (top-down driven), and in both cases, the transfer effect to untrained locations was observed.

• Second, neuroimaging studies have suggested that the results of perceptual learning can transform the processing of location information mediated by top-down signals into those realized by bottom-up implementations. For instance, fMRI findings from Sigman et al. (2005) showed that during the initial stages of learning a visual search task, there was significant activity in the frontal-parietal network, suggesting a pronounced top-level regulatory role; however, as the learning progressed, the frontal-parietal network activity decreased, while the activity in the lower visual cortex increased,

Taken together, we argue that neurons with different orientation preferences can achieve location-specific plasticity through top-down signals (such as attention). The columnar structure can be seen as a way to support position-specific learning.

Q2: On the use of different read-out networks. In our simplified model, we only extract features related to angle and position, necessitating the use of different task-based networks to read out different orientation information. We agree that in real biological systems having much more complicated structures, a single network module can process different orientation information. Nevertheless, since the focus of this study is on elucidating the dual-learning framework, in particular, the effect of different speeds of the two learning processes, we could extend the current model to have a single task-based learning module to handle different orientations, which will not change our results qualitatively.

Q3: On the use of CNN. The CNN in our model is solely for simulating the task-based learning process with the capability of location generalization. Indeed, there may also be some degree of orientation transfer. Since our study focuses on location specificity, we employed different task-based networks for different orientation discrimination to avoid this interference. It ensures that the learning effect is specific to the location without confusion with orientation changes.

Q4: Indeed, the combination of Hebbian learning with normalization can induce distorted feature representations at untrained locations and subsequently cause degraded learning effects at untrained locations. However, this phenomenon is not biologically implausible. In real human subject experiments as cited in Fig 1C, it shows that repeated training at specific locations can indeed cause a decrease in subjects' performances at untrained locations.

Q5: Equation 2 presents a general learning rule. In practice, for simplicity, we simplified the operation. As shown in Equations A3 and A4, we considered

$$\lim_{a\to0} W_{t+1}(\mathbf{x}, \mathbf{x}^\prime) = \frac{A_{t+1}(\mathbf{x}^\prime)}{\sqrt{2 \pi} a} \exp \left[-\frac{(\mathbf{x} - \mathbf{x}^\prime)^2}{2 a^2}\right],$$

which shows that feature-based learning employs a Gaussian kernel to perform weighted convolution on representations. For simplicity, we considered the scenario where parameter $a$ approaches zero. In this case, significant weight updates occur only when the input and output positions are exactly the same, effectively creating the one-to-one connection between neurons at identical locations.

Q6: See reply to Q4 of the Reviewer UGym.

Q7: See reply to Q11 of the Reviewer Wypk.

Q8: Apologies for the typo. It should be "divide by 10".

Q9: Apologies for the confusion, indeed the labeling of "loc1_ori2" and "loc2_ori1" in Fig.7b were reversed. After correcting the labeling, it can be seen that our results are consistent with the experimental results in Xiao et al. 2008 (Fig.3).

After re-evaluating the improvement results shown in Fig.7b and Fig.7c, we find that the inconsistency arises due to that Fig.7b presented the average of 100 simulation runs, while the improvement in Fig.7c was calculated by averaging the rate of improvement from each simulation, where a few outliers affected the results. After excluding 4 outliers, the results are consistent, see Fig.3 in the attached PDF.

Q10: Thanks for the reminder.

Q11: Typos will be corrected.

Thanks for the careful and valuable comments. We streamline reviewers' main concerns and address them one-by-one.

Q1: On the goal of feature-based learning. In this study, we argue that the goal of feature-based learning is to capture the statistical characteristics of external features. Therefore, it only takes effect when there is a significant change in the distribution of external features. In our experiments, when the stimuli are presented repeatedly at the same location for many times, triggering the sense of statistical change of external stimuli by the brain, the weight changes associated with feature-based learning become pronounced, eventually dominating the model's performance and demonstrating specificity. This mechanism highlights how feature-based learning adapts to repetitive patterns, enhancing the model’s ability to specialize in recognizing consistent (salient) features in its environment.

Q2: Apologies for the confusion. Indeed, $\mathbf{x}$ should be a vector indicating the position. Specifically, for the task at hand, $loc(\mathbf{x})$represents the coordinates as $\mathbf{x}=(x,y)$, where $x$ and $y$ denote the horizontal and vertical coordinates, respectively.

Q3: Apologies for the confusion. Here, we actually consider the case of limit. Specifically:

$$\lim_{a \to 0} W_{t+1}(\mathbf{x}, \mathbf{x}^\prime) = \frac{A_{t+1}(\mathbf{x}^\prime)}{\sqrt{2 \pi} a} \exp \left[-\frac{(\mathbf{x} - \mathbf{x}^\prime)^2}{2 a^2}\right].$$

As the width $a$ approaches zero, the Gaussian kernel becomes very sharp, indicating that weight updates are concentrated at $\mathbf{x}=\mathbf{x}^\prime$. In this limit scenario, significant weight updates occur only when the input and output positions are exactly the same, corresponding to the one-to-one connection.

Q4 and Q8: In Fig.3B and Fig.3E, the gray line indicates the number of training epochs required for the model to achieve a 90% accuracy rate. This metric serves as the reference point for assessing the convergence speed of the model under different learning conditions.

Under the task-based learning only condition (as shown in Figure 3B), the model requires approximately 75 training epochs to reach 90% accuracy. However, under the condition of combining task-based and feature-based learning (as shown in Figure 3E), the model converges faster, needing only about 68 training epochs to achieve the same level of accuracy.

A statistical significance was found between these two conditions, with a p-value of approximately 0.00148 (< 0.005) in one hundred simulations.

Q5-7: Questions 5 to 7 are closely related. Feature-based learning is independent of task requirements, so it is possible that it may reduce the model's performance. For human subjects, feature-based learning and task-based learning occur simultaneously, thus they cannot be compared separately. This explains why we do not observe enhanced feature processing at the trained locations in Fig.3C. Rather, we should focus on Fig.3D, where we assess similarity by calculating the correlation between the representations before and after feature-based learning. The correlation is calculated as follows:

$$corr = \frac{n \left(\sum F^*_t F_t \right) - \left(\sum F^*_t \right)\left(\sum F_t \right)}{\sqrt{\left[n \sum (F^*_t)^2 - \left(\sum F^*_t \right)^2\right] \left[n \sum (F_t)^2 - \left(\sum F_t \right)^2\right]}}.$$

Here, $F^*_t$ and $F_t$ represent the representations before and after the feature-based learning module, respectively, and $n$ is the total number of representations.

Q9: In Section 4, all our model simulations utilize difficulty thresholds. Specifically, we adjusted the offset between two vertical Gabor filters in the task to create a series of difficulty levels, and we selected the difficulty level that corresponds to 80% accuracy of the model as the threshold (mentioned in line 193 and Appendix C2). The "discrimination threshold" introduced in line 191, however, refers to the threshold used in psychophysical experiments to assess learning performance. Although these two types of thresholds differ, their roles are similar: they are both used to measure the limit at which the model achieves a certain performance under specific conditions.

Q10: The results of the t-tests conducted on the data from Fig.6 are presented below, highlighting the comparison between T4 and T8. As observed, the statistical differences between these two conditions are not significant. This finding aligns with the results depicted in Fig.1C(ii), indicating the consistency across different parts of the study.

Table 1: Statistical comparison of Fig.6

Q11: We revised the description as follows: "The results are presented in Fig.6B, which show that the thresholds decrease gradually in the trained condition (left panel in Fig.6B). In conjunction with Fig.3E, it can be observed that in the untrained condition, the threshold should first decrease and then either increase or saturate. This indicates that with increased training, there is a transition from transfer to specificity at untrained locations." This describes how training intensity influences learning dynamics differently in trained versus untrained locations.

Thanks for the valuable comments of the reviewer. We streamline reviewers' main concerns and address them one-by-one.

Q1: On the removal of the feature-extraction module. The feature extraction module is akin to a vision representation extractor that has been trained through extensive experiences over time. If we remove or disrupt this module, other parts of the model will not be able to operate effectively, rendering the entire model ineffective.

Q2: On the choice of Hebbian learning or backpropagation. In the dual-learning framework, feature-based learning aims to capture the distribution variation of external stimuli, which is task-independent and requires a significant number of observations. Therefore, we used slow and unsupervised Hebbian learning to achieve this goal. In contrast, task-based learning is directly linked to task execution using the existing representations, and it can be quickly achieved by modifying the read-out weights. We therefore chose the fast and supervised BP to implement this part. Overall, the choice of the learning method is based on the biological plausibility.

Q3: The newly added ablation studies. We add two ablation studies to highlight the importance of the relative speeds of two learning processes.

1. Accelerating feature-based learning. As shown in Fig.1 of the attached PDF, we increase the learning rate of feature-based learning by tenfold and replicate the four experiments in the paper. The results are:

• In Exp1, the results do not differ significantly from the original model, but the accelerated feature learning significantly modifies feature representations, leading to significantly degraded performances at untrained locations (ori1_loc2).
• In Exp2, due to continuously changing experimental conditions and the accelerated feature learning rate, the representations are constantly altered, preventing the model from performing well in the learning task and failing to achieve human-like transfer.
• In Exp3, the transferability of learning in new conditions diminishes with increasing training numbers, but with significant changes in feature-based learning, the differences in learning curves under new conditions are no longer distinct and become entangled, showing a decreasing trend in learning gain (except for T12), with increased error bars. However, when these differences are statistically analyzed using a T-test compared to the results in the main text (as detailed in the reply to the Reviewer Wypk), the p-values have increased, suggesting reduced statistical significance between conditions.

  	T2	T4	T8	T12
  T2	1	0.049	0.00088	0.12
  T4	0.049	1	0.29	0.5
  T8	0.00088	0.29	1	0.053
  T12	0.12	0.5	0.053	1

Table 1: Statistical comparison of Exp3 with accelerating feature-based learning

• In Exp4, double training no longer achieves the transfer of learning, and performances in both transfer conditions are even worse after double training.

2. Slowing down task-based learning: As depicted in Fig.2 of the attached PDF, we decreased the learning rate of task-based learning by tenfold. As expected, this adjustment resulted in a degradation of the learning effect, rendering the model less capable of mastering the tasks efficiently.

• In Exp1, although the model cannot fully master the current task, a comparison with Exp1 in Fig.1 reveals that there is no significant difference in the effects of training on other conditions.

• In Exp2, especially under random conditions, it is observed that despite the model's inability to master the current task effectively, the improvements brought about by learning still transfer to new conditions.

• In Exp3, due to the model's inability to effectively master the current task, the learning curves for all four different conditions are almost entangled. Significance testing shows that p-values have increased, indicating smaller differences between different training conditions:

  	T2	T4	T8	T12
  T2	1	0.68	0.089	0.94
  T4	0.68	1	0.02	0.76
  T8	0.089	0.02	1	0.095
  T12	0.94	0.76	0.095	1

Table 2: Statistical comparison of Exp3 with slowing down task-based learning

• In Exp4, due to poor learning outcomes, double training did not facilitate the transfer of learning. However, when compared with Exp4 in Fig.1, it can be seen that there is minimal disruption to transfer conditions.

Due to time constraints, the above ablation study results are only statistical outcomes from 20 simulations. We plan to conduct 100 experimental simulations consistent with the main text and will perform a more detailed analysis. However, we do not expect significant deviations from the current ablation study results.

These two sets of ablation studies demonstrate that both slow feature-based learning and fast task-based learning are necessary for our model to reproduce perceptual learning phenomena.

Thanks for your encouraging and valuable comments. We streamline reviewers' main concerns and address them one-by-one.

Q1: In the current study, we have used a feature extraction module that remains unchanged during the learning process to reflect the pre-processing of visual inputs in the brain. This setup mimics the resulting capability of the human visual system that has undergone extensive exposure to diverse environments throughout life. In the typical psychophysical experiments which last at most hours, we regard this part of pre-processing as unchanged. To accomplish the Vernier discrimination task in this work, we have employed a simple basis function network for pre-processing. Certainly, if more complicated tasks are included, we can employ more sophisticated pre-trained models.

Q2: As for the choice of models and learning methods, please refer to the overall rebuttal.

Q3: Indeed, as pointed out by the reviewer, orientation tuning emerges in the visual cortex. Our description of feature extraction in the paper was not precise. The feature extraction module in our model should include the input layer of V1 (responsible for extracting visual features). We will revise the statement in the manuscript accordingly.

Q4: The CNN used in our model is a streamlined convolutional neural network comprising three convolutional layers.

• The initial layer, layer 1, inputs a single channel and uses a 3x3 convolutional kernel to output 6 channels.
• This is followed by layer 2, which processes these 6 channels through another 3x3 kernel to produce 10 channels.
• The concluding layer, layer 3, compresses these 10 channels into a single output channel using a 3x3 convolution.

The network employs ReLU activation after the first two convolutional layers to add non-linearity and includes a sequence of flattening and a 1D max pooling on the final output to structure the output appropriately.

Q5: As for the model choice, please refer to our overall rebuttal. Indeed, the most fundamental difference lies in the objective functions of the two learning processes. Utilizing Hebbian learning (Hebb) or backpropagation (BP) mainly simplifies the model implementation.

Q6: From the perspective of the dual-learning framework, variability enhances generalizability primarily because it prevents feature-based learning from confining learning effects to specific feature combinations, thus allowing task-based learning to dominate and exhibit transferability.

On the other hand, specificity also plays an important role in learning. For instance, according to Reicher's study in 1969, native English speakers, due to extensive reading in English, can recognize words significantly faster than individual letters or strings that do not conform to phonetic rules. This indicates that prolonged training on specific types of inputs can significantly enhance the efficiency and accuracy of processing these inputs.

Overall, our model aims to simulate these learning dynamics observed in real-world scenarios. It emphasizes the interaction between feature-based and task-based learning, highlighting their joint effects on perceptual recognition and task execution capabilities. Both learning are valuable for the brain to adapt to and interact with complex environments.

Q7: Yes, we set a much higher learning rate for task-based learning, resulting in the network parameters related to task-based learning being updated much more quickly, which allows the model to rapidly adapt to the task replying on the existing feature representations.