RanDumb: Random Representations Outperform Online Continually Learned Representations

We thank the reviewer for their time and feedback. We appreciate the recognition that our idea is simple and easy to understand, as well as the recognition of the comprehensive nature of our ablation study and the improvements demonstrated across multiple benchmarks. The reviewer also pointed out that our meta-study, which highlights weak points of the entire field rather than specific works, is important for the community. We seek to address the concerns raised by the reviewer below. Below we have merged different parts of the review that come under the reviewer’s “meta-claim” and “technical claims” perspectives of the paper to provide a thorough reply in these aspects.

---

Meta Claim

"... judging the 'meta' claim of the paper, that we should re-think CL methods, as they are not optimal for many of the existing CL benchmarks. I think that the additional benefits of this paper over g-dumb [1] are somehow limited, as it has been shown before that CL methods do not optimally solve the CL benchmarks in many cases. Can you please elaborate on the problems in CL that this paper introduces, and the difference between them and the points raised in g-dumb?"

• Our meta-claim, as consistently stated across the work, is that random representations of the raw pixels of the images consistently outperform the deep-learning based learned representations from methods specifically designed for online continual training.

• In contrast, the meta-claim made in GDumb is that continual learning methods forget all previous online samples, and their performance can be recovered by using the latest memory. Additionally, GDumb still uses deep models to justify their claims.

Main differences between GDumb and RanDumb meta-claims

• GDumb primarily argues about forgetting of online samples: GDumb relies entirely on memory and learns nothing from the online samples themselves. This ablates the effect of online samples on performance. In contrast, we only learn from online samples and use no memory.

• RanDumb primarily argues about inadequate representation learning: GDumb learns representations similar to the experience replay (ER) baseline and does not make any claims regarding the quality of the representations learned via online representation learning. We specifically ablate representations and discuss the effects.

Overall, there are critical fundamental differences between RanDumb and GDumb in their meta-points.

Differences between the experimental settings of GDumb and RanDumb

• RanDumb focuses mainly on low-memory settings, whereas GDumb primarily shines in high-memory settings. For example, it is trivial to note that in our primary rehearsal-free setting with no exemplars stored (currently popular in continual learning), GDumb would, by design, effectively provide random performance.

• RanDumb is entirely complementary to GDumb, i.e., RanDumb shines in benchmarks where GDumb performs poorly and vice-versa. Therefore, just like GDumb, we believe that RanDumb would be a critical addition to the continual learning literature.

Overall, we request the reviewer not to simply dismiss the meta-point because of a broad claim that both RanDumb and GDumb are simple baselines (with no other points in common), and we re-iterate that our meta-point is emphasized above and in the title.

---

Technical Claim

"From the technical point of view, the suggested method is simply an implementation of known methods on several different benchmarks, and as such has very limited originality. I find the technical point of view good and convincing, but not significant and novel enough to merit a publication in a top-tier conference."

We agree that a part of our work heavily relies on an existing and very well-cited work of Rahimi and Recht [52] to support our arguments. However, we believe that it is absolutely fine to do so for various reasons:

• It is well accepted in the research community to use fundamental insights and findings from a related field. The above work is a fundamental work with thousands of citations of top-tier conference papers, which have been built on top of this work.
• We are the first to show the effectiveness of the above approach in continual learning literature.
• We had to combine various insights from the continual learning literature to make the above work effective across a variety of experiments we presented in this work.

Just the fact that an approach seems simple should never be the reason to undermine its effectiveness, and we hope that the reviewer agrees with us.

---

Presentation

"While the main idea is clear, the paper is very confusing to read. The different benchmarks are marked by letters, which makes it hard to follow which is which..."

We would like to highlight that other reviewers found the presentation of the paper to be "good" or “excellent.” They found that "This paper is well-written," "The motivation, method, and results are very clear and easy to follow," and even specifically, "The paper is well-structured; it is relatively easy to browse the results and sections."

However, we do understand that it can sometimes be difficult to follow notations and references. Hence, we had summarized the important details about the different benchmarks relevant to our paper in a table alongside the “Benchmarks” paragraph in Section 3. We also describe critical aspects of the benchmarks in detail there.

Having said the above, we do acknowledge that labeling experiments by alphabets might affect readability, and we will improve this aspect by replacing these labels with short, explanatory phrases in our revised draft.

Could the reviewer suggest further ways to improve the structure of experiments? We are happy to make necessary changes.

---

We hope we have addressed the major concerns of the reviewer, and are happy to answer any further questions/concerns. We look forward to a fruitful reviewer-author discussion phase.

We would like to thank the reviewer for their time reviewing our work and providing encouraging remarks such as "no major technical flaws," "interesting and novel," and "extensive benchmarks." We also appreciate giving the presentation of our work a remark of being "excellent."

Below, we seek to address the potential weaknesses/questions raised by the reviewer in the same order as noted by the reviewer.

---

"I think the main novelty lies in the usage of random Fourier features because, without it, the proposed method is very similar to SLDA. Therefore, I'm not entirely convinced by the motivation. It makes sense that representation learning from scratch performs poorly in the low-data regime. In what cases do we not want to use a frozen pretrained representation directly?"

When strong pre-trained features are available for a specific application (e.g., where the test data distribution is more or less known during training), practitioners should use them as they are likely to perform better. However, in the absence of such pre-trained representations, as is the case with online continual learning where the future datapoints/classes are unknown, we argue that a baseline, such as ours in this work that simply uses random features, should be first evaluated thoroughly before relying too much on the online representation learning methods that exist in the literature.

We have made the above point empirically as well via extensive experiments where we show that several of the existing continual learning methods (including VR-MCL—the ICLR'24 best paper honorable mention [70]) fail to outperform our simple non-deep-learning based random feature baseline! We strongly believe that our random transform as a baseline will help in guiding future research directions. We are thankful to the reviewer for raising this question as we believe that adding a small discussion and related implications around this in our revised draft would strengthen our arguments further.

---

"The method involves using a very large embedding space (25K in most experiments), which raises concerns on efficiency. Kernel trick is efficient because you can compute the dot product in a high-dimensional space without explicitly projecting the data to that space, but I'm not sure how the dot product can be computed efficiently here."

We will expand our discussion on implementation details of our method versus SLDA [27]. Briefly, computing the dot product was extremely fast. For our use case, the runtime (on an Alienware x17 R2 laptop for 25K dimensions) was 0.3±0.02 seconds.

However, as you asked for details in Q1, the major bottleneck is not the inner product, it is computing $S^{-1}$. We use a standard approach of solving linear systems $S \cdot x = \mu$ for this using a heavily optimized function in PyTorch called torch.linalg.lstsq. The solution $S^{-1} \mu$ is then used to compute $x^T (S^{-1} \mu) - \mu^T (S^{-1} \mu)$.

Certainly, one could just invert the matrix $S$ at the beginning and then use the Sherman-Morrison formula for online updates as each update in our case is rank-1.

---

"I wonder if it's worth providing the definition of OCL clearly so that readers not familiar with OCL can grasp the ideas more easily. Also, I recommend adding details on the method explicitly (even if it's just a few equations in the appendix) rather than only providing references. For example, I'd appreciate seeing how the inverse of the covariance matrix is computed online."

Thank you for this question. We agree that providing a more formal definition of OCL will improve the readability of the paper. We will provide that in our revised draft. For details about matrix inversion, please check the answer above.

---

"What part of the results are the reported numbers in prior work? Is it all the numbers except for the proposed method?"

We clarify—all numbers in tables where the caption says (Ref: table and citation) except our method are taken from the aforementioned table in the cited paper. We ensure that the experimental settings are exactly the same. Please note that reporting numbers directly from prior works makes the baselines stronger as these are already heavily optimized by the authors of their corresponding papers, which most of the time is difficult to reproduce as the specific design choices (hyper-parameters, etc.) to achieve such numbers are not always well explained in the paper.

---

"I thought Co2L was a contrastive learning-based method. Why is it included in benchmark B.1 and not B.2?"

Thank you for bringing this to our notice. We will correct this right away!

---

"Could the authors provide some hypothesis on why fine-tuning/prompt-tuning based methods collapse in Table 6?"

Thank you for the great question. Based on recent research, and our observation that the collapse is early across tasks:

Parallel work {1} (will be cited and included in the revised draft) suggests that most current prompt-tuning methods often suffer from a lack of prompt diversity and can be characterized with a single prompt. As a result, the effectiveness of classification depends heavily on the quality of that prompt.

When designing a prompt for a large number of classes (e.g., 50 or 20), the prompt leads to generally discriminative representations (even for future tasks), whereas a prompt for just 2 classes, in our case, is limited in its discriminative power. This causes all prompt-tuning methods to collapse across tasks.

{1} Thede et. al., Reflecting on the State of Rehearsal-free Continual Learning with Pretrained Models, CoLLAs 2024

---

We hope we have addressed the major concerns of the reviewer, and are happy to answer any further questions/concerns. We look forward to a fruitful reviewer-author discussion phase.

We thank the reviewer for their time in reviewing our work and for providing their concerns and suggestions. We appreciate the recognition of the work's contribution in "challenging the prevailing assumptions about effective representation learning in online continual learning." The acknowledgment of our "extensive experiments" and results "extending the investigation to popular exemplar-free scenarios with pre-trained models" is highly valued.

Below, we address the reviewer’s request for clarification on RanDumb’s structure.

P1 [Structure of RanDumb needs Clarity]: (i) Could the authors depict the architecture of input, RBF-Kernel, pre-trained model (ViT), Decorrelate(D), and NCM(C)? Or include an additional figure to illustrate the RanDumb structure.

P2 [Embedding]: The embedding (RBF-kernel) is unclear; the authors could better explain how the random Fourier projection affects continual learning input representation with simple math.

We would like to clarify the structure of RanDumb across benchmarks (and will add a figure in the revised draft to clarify this).

• For training from scratch:
  Raw flattened pixels are inputted, followed by random projection, decorrelation, and then classification using the NCM (as described in Equation 3). No input representations are learned; we use raw pixels as inputs, as illustrated in Figure 1.

• For pretrained models:
  The input image is passed through the pretrained model to obtain features, followed by decorrelation, and then NCM classification. Mathematically, this would be the same as Equation 3 if we overload $\phi$ as the pretrained model instead of random projection.

We acknowledge our initial description was inadequate and appreciate you bringing this to our attention. We will clarify this in the paper with a separate figure for pretrained models and additional equations to avoid overloading $\phi$.

We hope we have addressed the major concerns of the reviewer, and are happy to answer any further questions/concerns. We look forward to a fruitful reviewer-author discussion phase.

Thank you for your insightful question. To compare the effect, we will use two baselines: ER [13] and iCARL [54].

Effect of gradient-free classifiers: The primary difference between these methods is that ER trains the last layer along with the previous network, while iCARL learns a non-gradient based Nearest Class Mean (NCM) classifier. Similar to iCARL, RanDumb also employs an NCM classifier, but differs as it does not train a deep network.

Training the last layer in a gradient-based manner, as noted by the reviewer, can lead to an additional degree of forgetting. However, even after addressing this issue, there remains a significant gap in performance that needs to be addressed, as highlighted by our comparison with RanDumb.

The experiment proposed by the reviewer requires two passes over the data, not permissible in the OCL setting. We suggest that our experiment above might provide a valid comparison to clarify the concern while maintaining the online continual learning scenario.

We hope we have addressed the major concerns of the reviewer, and are happy to answer any further questions/concerns. We look forward to a fruitful reviewer-author discussion phase.