Mouse Lockbox Dataset: Behavior Recognition of Mice Solving Mechanical Puzzles

Dear reviewer, we are thankful for your insightful feedback as it allowed us to improve our manuscript. Please allow us to address both the weaknesses and questions you stated in your review.

Any changes that we have introduced to the text of our revised manuscript have been marked with blue color.

As weaknesses 1 and 3, you argue that we ignored existing benchmark methods and did not provide a baseline. While there exist methods for behavior identification on individual mice (e.g., Keypoint-MoSeq [1], B-SOiD [2], and VAME [3]), these methods extract behavioral primitives of a mouse in isolation, rather than interactions with its environment as we do here. It is nontrivial to extend them such that they couple an action (e.g., touch) with an environmental element (e.g., a lockbox mechanism) and there is currently no established standard that could serve as a benchmark. Therefore, we decided against using these existing methods as a benchmark for our dataset. Instead, we applied an established keypoint tracking method (i.e., DeepLabCut [4]) to extract the 2D keypoints from the videos and implemented a generic processing pipeline, based on (Extended) Kalman filters, to detect proximity, touching, and biting in 3D space. We describe this method and evaluate its results in sections 2.1 and 3.4.

As weakness 2, you argue that the comparison between our and other datasets is unclear and suggest to give an overview in a table. We have added such a table (Tbl. 1).

In question 1, you ask for the definition of the keypoints used for tracking mice and mechanisms. We have added a figure (Fig. 5) and a table (Tbl. 5) to our appendix A.2 showcasing the keypoints we used.

In question 2, you asked if the videos were recorded under normal light conditions. The mice were kept in a 12/12-hour light/dark cycle with artificial (infrared) lighting. We have added this and some additional information in the first paragraph of section 3.1.

In question 3, you asked if the hunger state of the mice affected their ability to solve the lockboxes. We cannot answer this question because the hunger state was not varied systematically. The mice had ad libitum access to both food pellets and tap water in the home cages. Therefore it can be assumed that they were not hungry when entering the arena. The food reward was exclusively provided in the arena. Before conducting the experiments, the mice have been familiarized with the food reward. We have added this information to section 3.1, lines 251–257.

We hope that we were able to properly address your questions and remarks, and appreciate additional feedback.

[1] Weinreb et al. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Nature Methods, vol. 21, no. 7, pp. 1329–1339, 2024.

[2] Hsu et al. B-SOiD: an open source unsupervised algorithm for discovery of spontaneous behaviors. BioRxiv, 770271, 2019.

[3] Luxem et al. Identifying behavioral structure from deep variational embeddings of animal motion. Communications Biology, vol. 5, no. 1, pp. 1267, 2022.

[4] Mathis et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, vol. 21, pp. 1281–1289, 2018.

Dear reviewer, we are thankful for your insightful feedback as it allowed us to improve our manuscript. Please allow us to address both the weaknesses and questions you stated in your review.

Any changes that we have introduced to the text of our revised manuscript have been marked with blue color.

As weakness 1, you argue that providing behavior labels for a larger split of our dataset would be helpful. We agree. However, our goal is not to provide an exhaustively labeled dataset for supervised learning, but to contribute to the ongoing shift towards unsupervised methods by providing a high-quality test set. We provide more detailed action labels than most other datasets (i.e., 5 interactions x 4 mechanisms) and assess their quality (i.e., inter-rater reliability) systematically. This resonates with the ongoing debate in ML on whether to prefer larger amounts of noisy or smaller amounts of precise label data. Providing substantially more high-quality label data is currently beyond our capacity.

As weakness 2, you point out that our data exclusively shows female mice and that this may bias the analysis. We agree. However, the mice are group-housed and male groups often suffer from aggressive conflicts, while mixed groups come with their own challenges (e.g., growing number of animals). These challenges may bias the analysis as well.

As weakness 3, you point out that the labels are not distributed evenly. This is rooted in the mice behaving freely in the arena. Their preference for different actions and mechanisms is naturally occurring and reflected in the statistics we report. We have added this brief explanation to our manuscript. We have also added an “any” column to Tbl. 4 describing how often each of the action labels is active in total, and fixed a mistake in our calculations.

As weakness 4, you criticize that we provide grayscale videos. That’s the result of us recording in infrared. Infrared light is invisible to the mice. Recording in RGB requires bright light sources in the visible spectrum of the mice, which is often aversive to mice and biases their behavior. This is a standard approach when conducting animal experiments in controlled environments. We added a brief explanation to the first paragraph of section 3.1.

As weakness 5, you point out that we do not provide homography parameters. We unfortunately did not record them. But we aim to mitigate this problem as we will provide the lockboxes’ CAD model files, which provide ground truth anchor points for their keypoints. We will provide these files upon acceptance as they could possibly deanonymize us.

In question 1, you asked us to provide a baseline method. While there exist methods for behavior identification on individual mice (e.g., Keypoint-MoSeq [1], B-SOiD [2], and VAME [3]), these methods extract behavioral primitives of a mouse in isolation, rather than interactions with its environment as we do here. It is nontrivial to extend them such that they couple an action (e.g., touch) with an environmental element (e.g., a lockbox mechanism) and there is currently no established standard that could serve as a benchmark. Therefore, we decided against using these existing methods as a benchmark for our dataset. Instead, we applied an established keypoint tracking method (i.e., DeepLabCut [4]) to extract the 2D keypoints from the videos and implemented a generic processing pipeline, based on (Extended) Kalman filters, to detect proximity, touching, and biting in 3D space. We describe this method and evaluate its results in sections 2.1 and 3.4.

In question 2, you asked us to add a label dictionary and an exemplary notebook. We have added the label-dictionary.txt file to our dataset and its preview. We have considered providing a notebook with our submission, but decided against it. Reason being that our data can be applied with a broad spectrum of contexts (e.g., keypoints vs abstract representations, machine learning vs neuroscience) and we could not think of a way to provide a universal notebook.

We hope that we were able to properly address your questions and remarks, and appreciate additional feedback.

[1] Weinreb et al. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Nature Methods, vol. 21, no. 7, pp. 1329–1339, 2024.

[2] Hsu et al. B-SOiD: an open source unsupervised algorithm for discovery of spontaneous behaviors. BioRxiv, 770271, 2019.

[3] Luxem et al. Identifying behavioral structure from deep variational embeddings of animal motion. Communications Biology, vol. 5, no. 1, pp. 1267, 2022.

[4] Mathis et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, vol. 21, pp. 1281–1289, 2018.

Dear reviewer, we are thankful for your insightful feedback as it allowed us to improve our manuscript. Please allow us to address both the weaknesses and questions you stated in your review.

Any changes that we have introduced to the text of our revised manuscript have been marked with blue color.

As weakness 1, you criticize the lack of concrete use cases and research questions that our dataset is relevant for. We have added a short paragraph to the introduction, in which we outline how action labels enable computational analyses of complex behaviors. While we acknowledge the charm of giving a granular analysis of both its potentials and the literature available today, we refrained from doing so as this could otherwise distract readers from the core contribution of our work.

A weakness 2 and question 3, you point out that the temporal pseudo-synchronization that we also report on in section 3.5 may be an additional challenge in some contexts. We have considered post-hoc synchronizing the videos. However, we consider the temporal desynchronization of 1.39±1.5 frames in our data not critical given the temporal accuracy of ±3 frames of our behavioral labels. We could also not think of any method that would produce reliable results without substantial amounts of labor. Given this cost-benefit trade-off, we decided not to alter the raw data.

As weakness 3, you point out that the inter-rater reliability of the biting label is lower than for other labels and suggest that we use more standardized annotation methods. We would like to emphasize that we follow best-practise for labeling. Reporting the inter-rater reliability is the gold standard, but unfortunately the vast majority of work that presents new data falls short to do so. We also follow the best practice by having exclusively skilled human raters annotate the data that also have been specifically instructed prior to working with our data. The low inter-rater reliability is the mere reflection that biting is hard to detect even for expert human raters, e.g., due to occlusions. We hope that both the reviewers and readers of our work appreciate our transparency regarding the low inter-rater reliability of a subset of our labels.

In question 1, you ask us whether alternative post-processing methods were considered to improve our baseline method’s accuracy. We are sure that the results can be improved by specialized and fine-tuned methods. Our benchmark rather reflects the current performance of current standard methods. Proposing a new state-of-the-art algorithm for action classification is not the goal of the paper. We are looking forward to seeing what the field will develop.

In question 2, you ask us if we are planning to extend the dataset and if there are approaches under development that would further automate behavior annotation. We agree that providing more behavior labels would be beneficial in some contexts. However, our main goal was not to provide an exhaustively labeled dataset for supervised learning, but rather contribute to the ongoing shift towards self- and unsupervised methods by providing a high-quality test set. To this end, we provide more detailed action labels than most other datasets (i.e., 5 interactions x 4 lockbox components + reward consumption) and estimate their quality (i.e., inter-rater reliability) systematically. This resonates with the ongoing debate in ML on whether to prefer larger amounts of noisy or smaller amounts of precise label data. Labeling a substantially larger fraction of the videos at this quality is currently beyond our capacity. As we stated above, proposing a new state-of-the-art algorithm for action classification is not the goal of the paper.

We hope that we were able to properly address your questions and remarks, and appreciate additional feedback.

Thank you for your detailed response, which has addressed my concerns. I am willing to raise the score for this paper.

Dear reviewer, we are thankful for your insightful feedback as it allowed us to improve our manuscript. Please allow us to address both the weaknesses and questions you stated in your review.

Any changes that we have introduced to the text of our revised manuscript have been marked with blue color.

You mention that the use cases and potentially addressed research questions of our datasets are not entirely clear to you. The new paragraph in lines 052–057 outlines how action labels enable computational analyses of complex behaviors. We acknowledge the charm of reviewing applications in the diverse literature available today, but refrained from doing so as this could distract readers from the core contribution of our work.

➝ L55 -- you say that it is a disadvantage that poses are low dimensional. But there are many scenarios where this is advantage.

➝ L59-60 -- there is actually an ideal set of keypoint locations -- skeletal joints

➝ Relatedly, you spend a lot of the intro saying how bad it is to use keypoints for behavioral analysis, but then go on to use keypoints for your benchmark action recognition results. What am I missing?

By submitting to ICLR we hope to fuel the emerging adoption of representation learning in computational ethology. That is why we point out challenges of keypoint-based methods, although they are undoubtedly state-of-the-art. Methods for behavioral studies that rely on representation learning are—despite their potentials—currently under-researched. Therefore, keypoint methods remain the only reasonable benchmark. To adequately portrait keypoint-based methods we changed our phrasing in lines 057 and 060.

➝ L58 -- I don't understand what you mean when you write, "It impedes mapping of keypoint locations from different perspectives into a common coordinate system."

We have rephrased our statement to be more precise and clear.

➝ Please provide a clear breakdown of how much of your dataset video time is the animals engaged with the puzzle.

We have added an “any” column to Tbl. 4 describing how often each of the action labels is active in total. We have furthermore fixed a mistake in our calculations and updated the values in this table.

➝ What is your definition of "playtime"? I would prefer if these data were in the table and not in a ring/pie chart that is harder to read.

We have added our definition of total playtime, the number of perspectives multiplied by the real time recorded, to our manuscript. We have also added a new table (Tbl. 1) to compare other datasets against ours.

➝ How precise is your keypoint tracking (3D error in mm)?

➝ How many keypoints were tracked exactly?

We added Fig. 5 to our appendix A.2 showcasing the keypoints we used. There we also report our tracker’s 2D training and test errors.

➝ What tracking method did you use?

➝ What method/model did you use to go from keypoints to frame labels in your benchmarking?

→ There is a lot of missing detail that is required of a datasets / benchmarks paper that the authors must provide for the work to be useful to others.

➝ Similarly, as a dataset/benchmark paper, it is standard to show results from several different methods, or at least some internal comparisons over different model configurations/hyperparamters.

We have now extended the description of the used benchmark used in section 2.1.

While there exist methods for behavior identification on individual mice (e.g., Keypoint-MoSeq [1], B-SOiD [2], and VAME [3]), these methods extract behavioral primitives of a mouse in isolation, rather than interactions with its environment as we do here. It is nontrivial to extend them such that they couple an action (e.g., touch) with an environmental element (e.g., a lockbox mechanism) and there is currently no established standard that could serve as a benchmark. Therefore, we decided against using these existing methods as a benchmark for our dataset. Instead, we applied an established keypoint tracking method (i.e., DeepLabCut [4]) to extract the 2D keypoints from the videos and implemented a generic processing pipeline, based on (Extended) Kalman filters, to detect proximity, touching, and biting in 3D space. We describe this method and evaluate its results in sections 2.1 and 3.4.

We hope that we were able to properly address your questions and remarks, and appreciate additional feedback.

[1] Weinreb et al. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Nature Methods, vol. 21, no. 7, pp. 1329–1339, 2024.

[2] Hsu et al. B-SOiD: an open source unsupervised algorithm for discovery of spontaneous behaviors. BioRxiv, 770271, 2019.

[3] Luxem et al. Identifying behavioral structure from deep variational embeddings of animal motion. Communications Biology, vol. 5, no. 1, pp. 1267, 2022.

[4] Mathis et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, vol. 21, pp. 1281–1289, 2018.

Dear reviewer, we are thankful for your insightful feedback as it allowed us to improve our manuscript. Please allow us to address both the weaknesses and questions you stated in your review.

Any changes that we have introduced to the text of our revised manuscript have been marked with blue color.

As weaknesses 1 and 4, you argue that we lack concrete use cases and research questions that our dataset is relevant for and the fragmentation of our introduction. We have added a short paragraph to the introduction, in which we outline how action labels enable computational analyses of complex behaviors. And while we acknowledge the charm of giving a granular analysis of both its potentials and the literature available today, we refrained from doing so as this could otherwise distract readers from the core contribution of our work.

As weakness 2, you criticize the lack of a table comparing our dataset against others. We have added such a table (Tbl. 1).

As weakness 3, you point out that the benchmark method is not well described in the manuscript. We have now extended its description in section 2.1.

As weakness 5, you point out that our work would benefit from introducing additional baseline methods. While there exist methods for behavior identification on individual mice (e.g., Keypoint-MoSeq [1], B-SOiD [2], and VAME [3]), these methods extract behavioral primitives of a mouse in isolation, rather than interactions with its environment as we do here. It is nontrivial to extend them such that they couple an action (e.g., touch) with an environmental element (e.g., a lockbox mechanism) and there is currently no established standard that could serve as a benchmark. Therefore, we decided against using these existing methods as a benchmark for our dataset. Instead, we applied an established keypoint tracking method (i.e., DeepLabCut [4]) to extract the 2D keypoints from the videos and implemented a generic processing pipeline, based on (Extended) Kalman filters, to detect proximity, touching, and biting in 3D space. We describe this method and evaluate its results in sections 2.1 and 3.4.

In your question, you ask us if we will release the keypoint tracking results alongside our dataset. Yes, we will also publish the raw DeepLabCut tracks for all three perspectives for the entire dataset, so that users can freely decide on their methods for 3D triangulation, smoothing etc. There, we hope to make the dataset attractive also to researchers that wish to get started with a keypoint-based approach.

We hope that we were able to properly address your questions and remarks, and appreciate additional feedback.

[1] Weinreb et al. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Nature Methods, vol. 21, no. 7, pp. 1329–1339, 2024.

[2] Hsu et al. B-SOiD: an open source unsupervised algorithm for discovery of spontaneous behaviors. BioRxiv, 770271, 2019.

[3] Luxem et al. Identifying behavioral structure from deep variational embeddings of animal motion. Communications Biology, vol. 5, no. 1, pp. 1267, 2022.

[4] Mathis et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, vol. 21, pp. 1281–1289, 2018.