Thanks for engaging with the paper, and for giving us the opportunity to clarify why we see this dataset – both the first release and the ongoing collection – of this dataset as important. First, please see the response to Reviewer #F6HH, where we elaborate on the utility of this dataset for the ML community.

We would also like to emphasize that the first release of our dataset is still the largest and the most diverse developmental egocentric dataset to date—in fact, for scientists seeking to model and understand human cognition, this is likely more important than the fact that these videos are higher-resolution videos than older datasets or contain accelerometer/gyroscope data. The SAYCam dataset (Sullivan et al., 2020), while unprecedented in size at the time, only captured everyday experiences from three children of psychologists (who were also the first children in their families), limiting the scope of the generalizability of the conclusions that can be drawn from these data (this is also true of the Ego-SingleChild subset). In contrast, the full BabyView dataset seeks to capture the child’s perspective across a much greater number and more diverse set of families across the U.S.

With respect to the family demographics, we note that the BabyView videos include many families where both speech from mothers/fathers/grandparents and other care-takers occurs. The example in the SI just happened to highlight the speech of a mother. Some of our families do have relatively higher incomes; recruiting low income families remains a major challenge in developmental research (Singh et al., 2023). As we build the BabyView dataset, we are actively engaged in recruiting participants from a wide variety of backgrounds. By collecting from a combination of convenience samples as well as targeted samples of families from under-studied backgrounds, our goal is to maximize the number of hours of data while striving for as representative a sample as possible. In future releases of the dataset, these demographic variables will be made available within Databrary.

Because of the slow pace of data collection for BabyView, our goal is to release the dataset in chunks as it becomes available in order to maximize scientific progress across all research groups, not only within our own. We schedule each release after we have had time to clean each portion of the dataset and ensure that parents have been given ample opportunity to retract any portion of their videos. We currently are planning to collect data through September 2025, pending funding, and will release all of the data via Databrary after the six month embargo expires (Spring 2026).

As per your last comment on the differences in domains, please see our response to Reviewer es8L, which emphasizes that we elected not to evaluate image-trained models on egocentric video benchmarks.

In our revised manuscript, we will (1) emphasize these points in both the Introduction and the General Discussion, (2) provide an overview of why we have chosen the particular annotation and training tasks to characterize the dataset as a first pass before diving into details, (3) provide additional clarifying details throughout the methods/results, including details on the datasets in Table 1,AP/AR (which refer to average precision/recall), the ages of the child participants (0;5 represents 0 years, 5 months, and this is a standard shorthand in developmental psychology papers), and the procedure for diarization, and (4) clarify why we chose not to evaluate image-trained models on egocentric video benchmarks.

References:

Gilmore, R. O., Adolph, K. E., & Millman, D. S. (2016, August). Curating identifiable data for sharing: The Databrary project. In 2016 New York Scientific Data Summit (NYSDS) (pp. 1-6). IEEE.

Sullivan, J., Mei, M., Perfors, A., Wojcik, E., & Frank, M. C. (2021). SAYCam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective. Open Mind, 5, 20-29.

Thanks for your comments and for engaging with the paper. First, please see our responses to Reviewer F6HH and Reviewer es8L, where we elaborate why we think this dataset is of value for researchers in the machine learning community and has scientific value more broadly as well.

Your review highlights that our current models do not show a difference in results using data trained on older, lower-resolution datasets (like SAYCam) vs. our more diverse dataset (BabyView). We also found this surprising! This runs counter to the hypothesis that many researchers held (including us!) that we might see dramatically better performance with more data and with images from better, higher-resolution videos. The fact that this was not the case further puts the onus on algorithmic development for the field to make progress on building algorithms that can learn as humans do; any model that is supposed to learn the way humans do should perform better when it is provided with a more complete set of “training data” from the child’s perspective. In a revised paper, we will emphasize that the primary contribution of the paper is the dataset itself —which stands a challenge to the community—rather than any of our particular modeling results.

We will also clarify why we believe our annotations are of particular value. The gold-standard annotations that we will provide are careful, hand-annotated, gold-standard data from ourselves looking at the data that can then be used to benchmark both speech transcription and pose annotation algorithms. To be clear, we will provide all model annotations as well as gold-standard annotations for the subset that was hand-annotated by humans. With respect to pose annotations, we will clarify that the highest performing models reach comparable performance with these egocentric frames vs. state-of-the-art benchmarks on allocentric images. With respect to the diarization results, we will also clarify that diarization was conducted with a state-of-the-art model (Lavechin et al., 2020), noting also that there are very few diarization models designed to handle child and child-directed speech, which are out-of-distribution for most speech processing models. Furthermore, the lower precision annotations were primarily for child speech, in which the model typically confused the key child and other children; this is perhaps unsurprising given that child utterances tend to be much shorter (and thus harder to identify).

Finally, with respect to access to the dataset, we will be providing access via Databrary, a well-established database platform for video data which provides a user-friendly interface that allows for easy viewing and browsing of the dataset (e.g., videos are playable within the browser) and selection of individual videos by demographic and participant variables. We will be providing detailed metadata to accompany the dataset, including family demographics as well as parent-report vocabulary scores for a subset of the children for whom it’s available. The metadata as well as the public Databrary API will allow researchers to download and process particular portions of the dataset that meet their own research requirements or interests. Finally, in a revised paper, we will clarify the procedures that researchers should follow to access these videos and their associated metadata.

References:

Lavechin, M., Bousbib, R., Bredin, H., Dupoux, E., & Cristia, A. (2020). An open-source voice type classifier for child-centered daylong recordings. arXiv preprint arXiv:2005.12656.

Gilmore, R. O., Adolph, K. E., & Millman, D. S. (2016, August). Curating identifiable data for sharing: The databrary project. In 2016 New York Scientific Data Summit (NYSDS) (pp. 1-6). IEEE.

Lasky, E. Z. & Klopp, K. (1982). Parent–child interactions in normal and language-disordered children. Journal of Speech and Hearing Disorders, 47: 7–18.

Thanks for your comments and engaging with our paper. We generally agree with many of your comments but we want to highlight that we see the primary contributions of our paper to be the dataset. The BabyView dataset is the largest and most diverse dataset of egocentric videos from the child’s perspective. These data are costly and time-consuming to gather, but – as we highlight in our response to Reviewer F6HH above – they represent our best estimate of the true training data for natural intelligence (babies). Any model that is supposed to learn the way humans do must be able to succeed using human data!

Your review highlights that there are many possible learning targets to evaluate based on egocentric video. In the vision domain, our paper focused on transfer to third-person object recognition because we know that babies are good at object recognition and this task has become an important and challenging transfer task in the literature (e.g., Zhuang et al., 2020; Vong et al., 2024; Orhan & Lake, 2024). Babies do not show any “domain gap” as far as we know – even though they are “trained” on continuous views of the world from an egocentric perspective, they have no trouble recognizing objects in pictures (e.g., Deloache et al., 1979)!

Nevertheless, you are right that there are other possible video benchmarks that we could use that would require less domain shift, including several from EPIC-Kitchens and Ego4D. We have not yet trained video models on the BabyView data, so we elected not to evaluate on these benchmarks, but we view this as an important next step.

In a revised paper, we will also take care to clarify some of the methodological points raised in this review. We will add EgoExo4D as an additional dataset to the table, and clarify that “long” refers to “longitudinal” (videos by the same person over time). In addition, we will add details on the pose annotations, which were manually completed by an author on a small portion of frames in order to evaluate the pose detection models mentioned in the text. We will show pose annotations on example frames from families who have provided public access to their videos; we are able to share this video which has the automated annotations overlaid on a sample video. In addition, we will clarify that we extracted Ego4D at 1 FPS in order to have a comparable amount of frames across training regimes between this dataset and the developmental datasets.

We extracted Ego4D at 1 FPS to make it feasible to train on that data using our hardware I/O and capacity constraints. Semantic information in video is highly redundant at a small temporal scale. As a result, for image-based representation learning and tasks like category recognition or semantic segmentation, we expect that 1 FPS sampling will not result in meaningful loss of learning signal.

References: DeLoache, J. S., Strauss, M. S., & Maynard, J. (1979). Picture perception in infancy. Infant behavior and development, 2, 77-89.

Orhan, A. E., & Lake, B. M. (2024). Learning high-level visual representations from a child’s perspective without strong inductive biases. Nature Machine Intelligence, 6(3), 271-283.

Vong, W. K., Wang, W., Orhan, A. E., & Lake, B. M. (2024). Grounded language acquisition through the eyes and ears of a single child. Science, 383(6682), 504-511.

Zhuang, C., Yan, S., Nayebi, A., Schrimpf, M., Frank, M. C., DiCarlo, J. J., & Yamins, D. L. (2021). Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3), e2014196118.

Thank you for engaging with our paper. We believe that our viewpoint on learning is similar to yours, namely that both human and artificial intelligences likely learn best through active engagement with the world. Indeed, the BabyView corpus consists of videos from the perspective of infants actively engaged in exploration and interaction with the world, making them a rich source of insights about how this process unfolds.

Babies in our dataset are in their own homes, looking at the things that they are interested in, selecting what to look at (presumably via the mechanisms posited in reference [A]). But remember, we are studying human babies, so we cannot “control” them or know exactly why they take the actions they take! That said, our dataset is the first dataset with children to include data from inertial motion units (accelerometer and gyroscope) to shed light on the role of the child’s actions in changing their visual exposure.

Video data about human babies’ experiences are a critical part of understanding the growth of human intelligence: these videos are as close as the field of developmental psychology has come to capturing the natural training data that humans receive (Smith et al., 2015). So, these data stand as a challenge to current machine learning algorithms. Human children learn robust and generalizable representations for vision and language with similar data to those we present (see e.g., Gopnik, 2022). The lack of strong results from standard algorithms in either the visual or language domain is evidence that we need to make new innovations (perhaps including some from the works on video learning cited in your review).

In sum, the importance of our data is that, if we want to build artificial intelligences that learn like humans, they must be able to learn from the same data that humans learn from.

References: Gopnik, A. (2022). Children, creativity, and the real key to intelligence. APS Observer, 35.

Smith, L. B., Yu, C., Yoshida, H., & Fausey, C. M. (2015). Contributions of Head-Mounted Cameras to Studying the Visual Environments of Infants and Young Children. Journal of Cognition and Development, 16(3), 407–419. https://doi-org.stanford.idm.oclc.org/10.1080/15248372.2014.933430

Thanks for your review and for your acknowledgement of the effort involved in collecting this dataset and its general scientific value.

In our revision we will absolutely attempt to make clear to the ML community why this dataset is valuable. Please see our response to Reviewer F6HH below.

In our revised paper, we will provide better justification for camera selection decision (which was present in the journal paper that first described the BabyView camera; Long et al., 2024) as well as additional details about the camera (e..g, the field of view, and resolution of the video, as suggested). In brief, developmental psychologists have used head mounted cameras with different fields of view in their research, recognizing that the majority of attention is focused on the central parts of the field of view because head movements are roughly synchronized with eye-movements (Yoshida & Smith, 2008). But several groups have argued that a large vertical field of view is critical for capturing 1) what children are doing with their hands, which tends to be in the lower periphery, and 2) caregivers’ faces for social signals, which tend to be in the upper periphery (e.g., Long et al., 2022; Franchak & Yu, 2022). Thus, the BabyView camera is designed to capture these specific peripheral experiences.

The use of a logistic fit is motivated by the nature of the BLIMP benchmark, which is a two-alternative forced choice format. Thus, scores must range between .5 (chance) and 1 (perfect accuracy). A logistic (sigmoid) is the appropriate functional form for extrapolation to avoid predictions above and below the range of the measure, and has been used in related work on scaling laws (e.g., Ruan et al., 2024).

Finally, regarding your comment about image vs. video models, we agree that the difference in view types between egocentric images vs. photographs may have an impact on model accuracy. However, there are two relevant dimensions of variation. First, egocentric vs. non-egocentric; second, image vs. video. Given that we only trained models on images, we did not evaluate them using video evaluation methods, although we intend to train and evaluate video models on this dataset in the future. Further, ImageNet classification accuracy provides a broad standard for comparison between our trained models vs. other models, allowing for direct comparison to the current state of the art (e.g., Orhan & Lake, 2024).

References: Franchak, J. M., & Yu, C. (2022). Beyond screen time: Using head-mounted eye tracking to study natural behavior. Advances in child development and behavior, 62, 61-91.

Long, B., Goodin, S., Kachergis, G., Marchman, V. A., Radwan, S. F., Sparks, R. Z., Xiang, V., Zhuang, C., Hsu, O., Newman, B. and Yamins, D.L., & Frank, M. C. (2024). The BabyView camera: designing a new head-mounted camera to capture children’s early social and visual environments. Behavior Research Methods, 56(4), 3523-3534.

Long, B. L., Sanchez, A., Kraus, A. M., Agrawal, K., & Frank, M. C. (2022). Automated detections reveal the social information in the changing infant view. Child Development, 93(1), 101-116.

Orhan, A. E., & Lake, B. M. (2024). Learning high-level visual representations from a child’s perspective without strong inductive biases. Nature Machine Intelligence, 6(3), 271-283.

Ruan, Y., Maddison, C. J., Hashimoto, T. (2024). Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938.

We appreciate that Databrary access is challenging, however, this is a necessary layer of protection for the privacy of the families involved. These families spend a considerable amount of time and energy recording private moments in their everyday lives, and we have an ethical and legal obligation to prioritize our participants' privacy and well-being in this research endeavor. To that end, Databrary is the state-of-the-art repository for sharing video data that contain identifiable video and audio information. The developers of Databrary have created a platform that ensures that the correct procedures are in place so that the confidentiality of the families are protected. As such, it is our best and only resource for sharing identifiable video data via our Institutional Review Board agreement.

Thank you for your response. While I do not agree that Databrary is the ideal platform for sharing the dataset and have had negative experiences with them in the past, I will not argue this point here, as it is not relevant to this paper. Moreover, I fully understand the importance of safeguarding participants' privacy and agree that protecting the families’ confidentiality is a top priority. In light of this, would it be possible for you to release an anonymized version of the dataset (e.g., by blurring faces, removing other privacy-sensitive information, and muting any personally identifiable audio) without using Databrary?

Thanks for your interest in the BabyView dataset. Unfortunately, two issues prevent us from posting the dataset. First, our platform (Databrary.org) is currently upgrading their upload functionality and so we cannot upload at this time. Second, because our manuscript was rejected from ICLR we have prepared a larger release of the dataset for another conference and will be releasing these data mid-summer pending availability of upload functions.