RTify: Aligning Deep Neural Networks with Human Behavioral Decisions

Methods of the current paper and the comparison between ACT

We will definitely strive to make the paper clearer and simplify our writing. We think our writing may have caused some confusion and here are some points we would like to clarify.

First, our main goal is to align the computational steps of an RNN with human RTs. Our main approach is not simply to place an RNN on top of a convolutional neural network (CNN) but to develop a differentiable framework that allows us to train an integrated model end-to-end on experimental data. Results from Figures 4-6 clearly show that we are able to derive such an RT measure better than previous models. After establishing that RNNs can be aligned with human decisions, we extended our work to align CNNs with human decisions. As this reviewer correctly pointed out, we, therefore, integrated an RNN with a CNN. Combined with our RTify method, we obtained an RT measure directly from the RNN steps required to complete the task. Figure 7 shows results similar to those in Figures 4-6.

Second, we used RTify for a dual purpose: (1) to fit human RTs, and (2) to flexibly minimize the number of computational steps needed to complete the task. This requires a computational method to directly optimize RTs. This cannot be done with ACT because it does not incorporate a way to directly train a loss function with respect to RTs. Instead, ACT can only indirectly be used for (2) by maximizing cumulative activities. Our gradient approach in Section 2.0 and Appendix D allows us to effectively achieve both goals (1) and (2) within the same framework.

We have provided a detailed comparison and clarification in the general rebuttal section to aid understanding. In brief, ACT is only compatible with image tasks using a self-supervision goal. Our results show that the model RT derived from the ACT method does not significantly correlate with human RT (r = 0.05, p = 0.24), see Fig R2.

Dataset issue

We understand that a large dataset will help strengthen our work. However, to our knowledge, none of the large datasets mentioned here (CIFAR10/100, ImageNet, COCO detection) come with human behavioral data and, hence, cannot be used to evaluate our model.

WW architecture

We added a detailed explanation (see general rebuttal.)

Formula of Phi

For the $\Phi$ function, it is a summation of $e(t)$, and $e(t)$ is $f_w (h_t)$. Where $f_w$ is achieved with an MLP.

There are two two different loss formulations for two purposes in the paper (i.e. fitting human RTs and minimizing time steps.)

For fitting RT, please directly refer to formula (3). Here $F$ is the mean square error(MSE) between the human RT distribution and model RT distribution.

For minimizing time steps, please refer to formula (4). And here we elaborate more on how to get the gradient. There are two terms in formula(4). One is the regular cross-entropy loss, which is totally differentiable and not worth discussion. The other is the penalty of RT, i.e. ($l_y \cdot \Tau_{\tau}$). When we are trying to get the gradient of the penalty term over $w$, since it is bivariate function, we first apply the chain rule $\partial( l_y \cdot \Tau_{\tau})/\partial w = \Tau_{\tau} \cdot \partial l_y / \partial w + l_y \cdot \partial \Tau_{\tau}/\partial w$. (formula *) For the first term of (formula *), since $l_y$ are just the logits of the network, that part is differentiable in the regular way. For the second term of (formula *), we need to apply formula (3) with a special case that $F$ is the identity function.

Runtime analysis of WW model

We are not entirely sure we understand the question because runtime is not really relevant here. The RT in the WW model is the number of time steps it takes for the circuit to make a decision. And we use that measure to model human RT.

In case we misunderstood, here is the actual runtime complexity of the WW model:

Input computation (2 matrix operations: one for each neuron).

Synaptic activity update (2 differential equations solved).

Noise update (2 noise generation and update operations).

Trajectory update (storing the values for later use).

Therefore, the WW circuit runtime is primarily governed by the number of time steps and the size of the batch to be processed, which results in a complexity of O(time_steps * batch_size). The reviewer is right that the total runtime of the RTified CNN is dominated by the forward step of the CNN since it contains most of the parameters, but we want to emphasize again that it has nothing to do with the RT.

Dear reviewer,

In summary your feedback can be listed in three main points. Let us respond to each one of them :

1. WW is still not clearly explained for practitioners, and providing the source code would help with this

We provided the source code in the original submission. See wong_wang.py in the supplementary.zip, for details on the implementation of WW. And we emphasize that our method is suitable for any reasonable decision-making circuit besides WW, in fact any RNN.

2. "I also don't see a significant breakthrough in the method itself, as it seems to be an RNN on top of features"

We respectfully disagree. Our main approach is not simply to place an RNN on top of features, but to have the RNN adaptively decide to stop based on the features of the RNN itself.Our technical breakthrough lies in the differentiability of time steps, which is neither solved in previous adaptive computing methods for RNNs (e.g. ACT), nor solved in human decision models (e.g. DDM). As we pointed out in our rebuttal. Unlike previous phenomenological models (e.g. DDM), our method allows us to develop image-computable, nonlinear adaptive models of decision making.

3. "I suggest that the authors focus on explaining the approach in more detail and ensure that the explanation benefits practitioners who are not experts in the field of reaction time."

We thank the reviewer for the feedback on explaining the method to a broader audience, which we will take into account for the camera-ready version. In general, however, we believe that this paper provides an important tool for neuroscience researchers to link the phenomenon of reaction times to neural mechanisms of decision making.

Psychophysics tasks

First, for the natural image dataset, we are using the dataset already peer-reviewed and published in Nature Neuroscience (Kar et al., 2019). As the reviewer pointed out, this is done via the Amazon Mechanical Turk (Mturk) platform. The reviewer is correct that these online experiments are not as easy to control compared to in-lab experiments but they come with many benefits (most prominently data collection is orders of magnitude faster to collect compared to in-lab experiments) and with they are now widely used by a large number of psychology labs (see Buhrmester et al., 2018 for a review).

We acknowledge that behavioral choices and RTs provide only a partial picture of human decision-making. However, we believe that these RTs and accuracy are by far the two most crucial metrics in the field of visual decision-making (Ratcliff, 2002; 2008; Chuan-Peng et al., 2002; Fengler et al., 2002), as they often reflect fundamental mechanisms of information processing. We hope our work will inspire experimentalists to conduct both well-controlled in-lab experiments and online experiments to further explore RTs and decision-making behavior.

WW architecture

First, we will add a detailed discussion of the details of the WW model in supplementary information (see general rebuttal) and we hope it can clarify the confusion.

Second, we also followed this reviewer’s suggestion and conducted an ablation study of the circuit for the random dot motion tasks. Our findings revealed an intriguing result that may prompt further research: ablating self-excitation causes the model to fail, resulting in flat RT distribution (MSE increases by 3369%), while ablating cross-inhibition deteriorates model performance (MSE increases by 38.5%). We thank the reviewer for bringing this up.

Predictions for other manipulations of reaction time

Because RTify tackles the task on a frame-by-frame basis, a practical future experiment could investigate how individual frames of the random moving dots stimulus affect RTs. Typically, in random dot motion tasks, researchers use a consistent coherence level and direction throughout each trial. However, our method enables frame-based interventions. For example, researchers could enhance the coherence of certain frames to determine the importance of early, mid, or late frames. Alternatively, specific frames could be reversed to an opposite direction or masked entirely. RTify opens up these possibilities, offering researchers new avenues for experimentation.

Lack of limitations and overstating broader impacts

We will soften our claims regarding broader impacts and include a limitation section so that readers can correctly understand the pros and cons of our current methods.

Reference:

Kar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019). Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nature neuroscience, 22(6), 974-983.

Fengler, A., Bera, K., Pedersen, M. L., & Frank, M. J. (2022). Beyond drift diffusion models: Fitting a broad class of decision and reinforcement learning models with HDDM. Journal of cognitive neuroscience, 34(10), 1780-1805.

Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Perspectives on psychological science, 13(2), 149-154.

Chuan-Peng, H., Geng, H., Zhang, L., Fengler, A., Frank, M., & ZHANG, R. Y. (2022). A Hitchhiker’s Guide to Bayesian Hierarchical Drift-diffusion Modeling with Dockerhddm.

Why is modeling RT important?

Modeling RT is crucial for two main reasons. First, because of the so-called speed-accuracy trade-off (fast decisions come with high error rates, and vice versa), a model that explicitly accounts for behavioral decisions and RTs is expected to reflect human brain processing better.

Second, RT itself is an important measure reflecting the processing speed and efficiency of cognitive functions (Heitz, 2014) . It captures the dynamic nature of the visual and decision-making systems (Kar et al., 2019; Wyatte et al., 2014; Ratcliff 2002; 2008). Numerous studies have used RT as a behavioral measure to understand how stimulus complexity, number of potential responses, or stimulus-response compatibility affect human decision making (Kaswan and Young, 1965; Fitts and Seeger, 1953; Fischman, 1984; Reddi and Carpenter, 2000). Integrating mechanisms responsible for behavioral choices and RTs will thus lead to a more complete model of human visual processing.

Organizing the model performance results

Thank you for the suggestion. We have organized the model results into a table (see rebuttal pdf).

Figure 7a

The distribution shown in Figure 7a is a distribution of signed RT, where the RT of a correct trial is positive, while the RT of an incorrect trial is negative. The distribution of signed RT is routinely used in decision-making studies to summarize experimental data into a single plot: (1) it shows the distribution for RT in all trials (2) the relative proportion of positive RTs over negative RTs reflects the accuracy. In this case, when our model shows a close match with such a distribution, we are successful in modeling both the accuracy and RT at the same time.

How does RT impact performance?

We are not entirely sure whether the reviewer is referring to the performance of human participants or the RTify model.

In terms of human performance: RT and performance are interdependent, as both behavioral choices and RTs are functions of the stimuli. Recent studies have found that if one is prone to error, one may deliberately take more time to decide (Adkins et al., 2024). This example shows how two factors may dynamically regulate each other. We believe this feature makes our RTify an important method since it now helps explain both the choices and the RTs.

In terms of the performance of how RTify predicts human decisions: this is a great question and we would love to look into this more closely. We expect a model trained on combined choice and RT data to better predict choice data. This could in theory be easily tested. Unfortunately, the datasets we use either provide choice data without the stimuli (Green et al., 2010) or they do provide the stimuli but do not provide choice data (Kar et al., 2019). We hope our paper can encourage researchers to collect more comprehensive datasets in order to better validate our method.

Runtime of the WW circuit

Thank you for raising this. We do see that more discussion over the WW circuit will help readers better understand the application of RTify over a specific canonical circuit. To this end, we put a detailed discussion of WW circuits in the general rebuttal section.

Limitations missing

We apologize for not being explicit about the limitations. We carefully took your suggestion into consideration and have written a separate limitation section (see general rebuttal section.)

Reference:

Kar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019). Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nature neuroscience, 22(6), 974-983.

Green, C. S., Pouget, A., & Bavelier, D. (2010). Improved probabilistic inference as a general learning mechanism with action video games. Current biology, 20(17), 1573-1579.

Heitz, R. P. (2014). The speed-accuracy tradeoff: history, physiology, methodology, and behavior. Frontiers in neuroscience, 8, 150.

Kar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019). Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nature neuroscience, 22(6), 974-983.

Wyatte, D., Jilk, D. J., & O'Reilly, R. C. (2014). Early recurrent feedback facilitates visual object recognition under challenging conditions. Frontiers in psychology, 5, 674.

Ratcliff, R., & Tuerlinckx, F. (2002). Estimating parameters of the diffusion model: Approaches to dealing with contaminant reaction times and parameter variability. Psychonomic bulletin & review, 9(3), 438-481.

Ratcliff, R., & McKoon, G. (2008). The diffusion decision model: theory and data for two-choice decision tasks. Neural computation, 20(4), 873-922.

Kaswan, J., & Young, S. (1965). Effect of stimulus variables on choice reaction times and thresholds. Journal of Experimental Psychology, 69(5), 511.

Fischman, M. G. (1984). Programming time as a function of number of movement parts and changes in movement direction. Journal of Motor Behavior, 16(4), 405-423.

Reddi, B. A. J., & Carpenter, R. H. (2000). The influence of urgency on decision time. Nature neuroscience, 3(8), 827-830.

Adkins, T. J., Zhang, H., & Lee, T. G. (2024). People are more error-prone after committing an error. Nature Communications, 15(1), 6422.

Comparison between ACT

We will add a comparison in the main text (see general rebuttal section).

Mechanistic account of RT

We will revise our wording to provide a more accurate claim about the scope of our work. Here, we emphasize how RTify helps our understanding of RTs. First, our results provide computational evidence that reaction times (RTs) are generated through an evidence accumulation process. Second, our findings provide further evidence that the computational principle underlying RTs is to achieve an optimal speed-accuracy trade-off. Third, RTify offers a tool to model other factors, such as learning and adaptation, that are known to affect RTs. Researchers can apply RTify to various experimental conditions and compare the resulting parameters.

Spelling errors

We will fix all these spelling mistakes in the camera-ready version.