Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement

Reviewer Question 1

Could the author explain the rationality of replacing $P$ (the prior distribution) and $S$ (sampled data) with $\mathcal{W}\_{src}$ (the pre-trained network) and $\mathcal{W}\_{trg}$ (fine-tuned network), respectively, in the context of PAC-Bayesian generalization theory? The original theory is designed to measure the generalization error between the empirical and expect loss, but for the same distribution.

Author Response

Thank you for raising this critical point of clarification. It is absolutely correct that the PAC-Bayesian generalization theory is traditionally designed to measure the generalization error between the empirical and expected losses for the same distribution. In our context, the PAC-Bayesian bound we provide pertains specifically to the target adaptation step, which solely operates on the target distribution. This ensures compliance with the "same distribution" constraint inherent in PAC-Bayesian theory. The validity of this approach stems from the fact that, by the time the target adaptation step begins, we have already pre-trained the network on the source distribution and have access to the pre-trained source weights. These source weights serve as a prior in the adaptation phase, enabling the PAC-Bayesian bound to operate effectively on the target distribution alone.

Reviewer Question 2

Could the author include an ERM baseline in Table 1 to provide readers with a clearer understanding of the performance improvements achieved by the proposed method?

Author Response

We sincerely thank the reviewer for the thoughtful suggestion to include an ERM baseline in Table 1 to provide readers with a clearer understanding of the performance improvements achieved by our proposed method.

We would like to bring to the reviewer's attention that the ERM baseline for the target performance of the source-pretrained model is already included in our manuscript. This is presented in the ablation analysis in Tables 2, 11, 12, and 13 under the row corresponding to the "No-Adaptation" method. If this aligns with what the reviewer is referring to, we would be happy to incorporate these results into Table 1 for consistency and easier reference.

However, if the reviewer is suggesting a different or additional ERM baseline, we kindly request further clarification so that we can address the feedback appropriately. We greatly appreciate your input and are eager to ensure the manuscript fully meets the reviewer's expectations.

We sincerely appreciate the reviewer’s recognition of the simplicity and efficiency of our proposed metric decomposition-based approach for target domain adaptation. We are grateful for the acknowledgment of our detailed and thorough experiments, which clearly demonstrate the effectiveness and efficiency of the method. Additionally, we thank the reviewer for appreciating the well-structured and clearly written manuscript, which facilitates easy comprehension for readers. Below we provide the responses to the concerns raised by the reviewer. All the changes in the revised version of the manuscript are highlighted in ${\color{blue}\text{Blue}}$.

Reviewer Weakness 1

The description of target domain adaptation is insufficient. Specifically, which loss function is utilised? And how to optimise the parameters?

Author Response

We thank the reviewer for pointing out the need for a more detailed description of the target domain adaptation process. In response to this suggestion, we have included a comprehensive explanation of the loss functions and adaptation objectives in the updated manuscript. Specifically, the details regarding the adaptation objectives, including the precise loss functions and the parameter optimization process, have been added in Section A.8 of the appendix. This updated section outlines the mathematical formulation of the adaptation losses. We hope this additional information addresses the reviewer’s concern and provides greater clarity on our adaptation methodology.

Reviewer Weakness 2

The novelty of the approach is limited, as parameter decomposition for domain adaptation is a prevalent technique.

Author Reponse

We appreciate the reviewer’s feedback . However, based on our extensive survey of the literature and works discussed, we are not aware of any prior works that leverage parameter decomposition for DA. Specifically, to the best of our knowledge, no existing work employs parameter decomposition in conjunction with selective fine-tuning to achieve both parameter and sample efficiency in domain adaptation.

If there are relevant studies or references that the reviewer is aware of, we would be grateful if they could direct us to these works, as it would help further contextualize our contributions. In our view, the primary novelty of our approach lies in demonstrating that parameter decomposition, when combined with selective fine-tuning, can simultaneously address the dual challenges of parameter and sample efficiency—a unique and compelling insight.

We respectfully request clarification and supporting references for the reviewer’s claim that parameter decomposition is a widely used method in domain adaptation. This would greatly assist us in strengthening the discussion in our manuscript.

Reviewer Weakness 3

The connection between the proposed method and the characteristics of time-series data is somewhat weak. The method appears to be a general SFDA approach rather than being specifically tailored for time-series data.

Author Response

We acknowledge that the connection between our proposed method and the characteristics of time-series data may not have been sufficiently explicit. Our primary motivation was to address the challenges of parameter and sample efficiency in Source-Free Domain Adaptation (SFDA) for time-series data—a pressing research gap we identified. In our analysis (see Figure 2A), we observed significant parameter redundancy along the channel dimension in source models, indicating strong inter-channel dependencies among variables in time-series features as they propagate through deep networks.

To leverage these characteristics, we proposed decomposing the model parameters along the channel dimension (Equation 3) and focusing on the principal channels or variables. This approach exploits the inherent inter-variable dependencies in time-series data, achieving parameter efficiency without sacrificing performance by concentrating on the most significant channels.

Our goal was to develop a method applicable across various SFDA settings and diverse time-series datasets. Therefore, we deliberately avoided introducing specific inductive biases. We conducted extensive experiments with both general SFDA methods (SHOT, NRC, AAD) and a time-series-specific method (MAPU), following Ragab et al [1]. We also expanded our empirical evaluation to all five datasets from the AdaTime benchmark [2]—unlike previous works that typically used only three—and tested ten source-target pairs, including a many-to-one adaptation setting.

References:

[1] Ragab, Mohamed, et al. "Source-free domain adaptation with temporal imputation for time series data." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.
[2] Ragab, Mohamed, et al. "Adatime: A benchmarking suite for domain adaptation on time series data." ACM Transactions on Knowledge Discovery from Data 17.8 (2023): 1-18.

We sincerely thank the reviewers for their valuable suggestions and thoughtful comments. We would greatly appreciate any additional feedback that could further improve our work and are happy to incorporate further suggestions to ensure its quality. If our responses have satisfactorily resolved your concerns, we kindly request confirmation of whether the provided clarifications and changes address the issues raised and, if applicable, reconsideration of the scores in light of the responses and the updated manuscript.

Thank you again for your time and effort in reviewing our work.

Understanding the Original PAC-Bayesian Framework:

The PAC-Bayesian theory provides bounds on the generalization error—that is, the difference between the empirical loss (on sampled data) and the expected loss (over the entire distribution)—when both are measured with respect to the same data distribution. In this framework:

• Prior Distribution ($P$): Represents our initial belief about the model parameters before seeing any data.
• Posterior Distribution ($Q$): Represents the updated belief after training on the sampled data $S$.
• Sampled Data ($S$): A set of examples drawn from a single data distribution $D$.

Our Context with Pre-trained and Fine-tuned Networks:

In our scenario, we're adapting the PAC-Bayesian framework to the process of fine-tuning a pre-trained network:

1. Prior as Pre-trained Weights ($P = W_{\text{src}}$):

   • We consider the pre-trained network's weights $W_{\text{src}}$ as our prior distribution.
   • These weights encapsulate knowledge from previous training on a source dataset but serve as our starting point before fine-tuning.

2. Fine-tuning on Target Data ($S = D_{\text{trg}}$):

   • We fine-tune the model using data $S$ sampled from the target distribution $D_{\text{trg}}$.
   • This process adjusts the model weights to better fit the target data.

3. Posterior as Fine-tuned Weights ($Q = W_{\text{trg}}$):

   • The fine-tuned weights $W_{\text{trg}}$ represent our posterior distribution.
   • They reflect our updated belief after seeing the target data.

Maintaining the Same Distribution:

A key point is that during fine-tuning, we're exclusively using data from the target distribution $D_{\text{trg}}$. This means:

• Both the empirical loss (computed on $S = D_{\text{trg}}$) and the expected loss (over $D_{\text{trg}}$) are with respect to the same distribution.
• The prior and posterior are connected through data sampled from this single distribution, satisfying the PAC-Bayesian assumption of operating within one distribution.

Replacement Rational:

• Alignment with PAC-Bayesian Assumptions: By treating the pre-trained weights as the prior and the fine-tuned weights as the posterior, we adhere to the framework's requirement of measuring generalization over the same distribution.
• Applicability of Generalization Bounds: Since we're fine-tuning solely on $D_{\text{trg}}$, we can validly apply PAC-Bayesian generalization bounds to assess how well our fine-tuned model will perform on unseen data from the same distribution.

In Summary:

Replacing the prior distribution with the pre-trained network $W_{\text{src}}$ and the posterior with the fine-tuned network $W_{\text{trg}}$ is justified because:

• We're operating entirely within the target distribution $D_{\text{trg}}$.
• The PAC-Bayesian theory's conditions are met, allowing us to measure the generalization error appropriately.

By maintaining the focus on a single distribution throughout the fine-tuning process, we ensure that our application of the PAC-Bayesian theory remains sound.

We sincerely appreciate the reviewer’s recognition of our novel use of Tucker-style factorization for model decomposition, offering a fresh perspective beyond conventional feature selection and BN adaptation methods. We are grateful for the acknowledgment of our solid results, the inclusion of a PAC-Bayes generalization bound for the selective fine-tuning stage, and the comprehensive evaluation of our method on real-world datasets such as the AdaTime benchmark. Below we provide the responses to the concerns raised by the reviewer.

Reviewer Question 1

Some domain adaptation methods only adapt the BN layers [1], normalization [2], or only feature modules [3]. They also selectively fine-tune the model, what is the relations between them and the proposed one. Can the proposed method explain or unify these methods in some cases?
[1] Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963, 2020.
[2] Tent: Fully Test-time Adaptation by Entropy Minimization. ICLR 2021.
[3] Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. ICML 2020.

Author Response

Thank you for your insightful question regarding the relationship between methods that adapt BN layers, normalization techniques, or feature modules and our proposed approach. We appreciate the opportunity to clarify this.

While domain adaptation methods that focus on adapting batch normalization (BN) layers [1], normalization techniques [2], or selectively fine-tuning feature modules [3] have shown effectiveness, they are orthogonal to our approach, which means our method can be combined with them.

Moreover, BN adaptation and feature normalization methods do not inherently provide interpretability regarding how and what aspects of the model are being adapted. In contrast, our method offers the potential for interpretability. For example, in a toy scenario illustrated in Figure 4, we demonstrate that the negated target domain can be adapted by learning an inversion of the core tensor. Although we did not extensively explore interpretability in more complex settings, our approach allows for insights into the source-target transformations that are being learned. Moreover, the decomposition also makes the backbone inference efficient, which is an added benefit.

We also explored the possibility of combining our method with BN adaptation and feature normalization techniques. However, our analysis did not show a significant improvement over using our fine-tuning strategy alone. The standalone fine-tuning of BN layers was tested in our ablation analysis (Tables 2, 11, 12, and 13, row 3), which supports this observation. Moreover, the normalization methods also suffer form the same issue. Additionally, our method can also be considered as a seletive finetuning strategy however, we alleviate the dilemma of what module to freeze and what to finetune.

We hope this clarifies how our method relates to and differs from existing approaches. Thank you again for your thoughtful question and for giving us the chance to elaborate on this aspect of our work. Please let us know if you need more insights into these aspects.

Reviewer Weakness 2

Can the proposed method be transferred to the general DA task? If not, could you please explain the reasons?

Author Response

Thank you for your insightful question. Our proposed method is designed with the Source-Free assumption in mind, where we have access to a pre-trained source model but do not have access to the source data during adaptation to the target domain. In the conventional Domain Adaptation (DA) framework, it is assumed that both source (labeled) and target (unlabeled) data are available during training. A single network is typically trained jointly using both datasets, and there is no distinction between a source model and a target model in this context. Since there is no separate fine-tuning step involved, there is no clear motivation or reason we see to decompose the backbone network and conduct selective fine-tuning in the typical DA setup.

Reviewer Weakness 2

It seems that the proposed theorem can be applied to any method with selective fine-tuning. What is the advantage of the proposed decomposition method analyzed from this theorem?

Author Response

Thank you for your insightful observation. You are correct that the presented theorem explaining the PAC-Bayesian bound is general and applicable to any decompose method. Notheless, we emphasize that the bound provided in the manuscript is presented to support our empirical observation and not define a foolproof tight bound. Our contribution lies in demonstrating how the combination of decomposition and selective fine-tuning enhances sample efficiency in the adaptation process. Specifically, our approach effectively constrains the parameters, and we verifiy this for our decomposition and selective finetuning strategy in Figure 5. By simplifying the PAC-Bayesian bound, we observe the emergence of a parameter distance factor, which highlights the why such selective finetuning contributes to the sample efficiency of our method. While the parameter efficiency might seem apparent due to the low-rank decomposition and selective fine-tuning of only a subset of parameters, the sample efficiency is less obvious. Our work seeks to using PAC-Bayesian bound as an analytical tool, providing insights into why decomposing and selectively fine-tuning the model not only reduces the number of parameters but also requires fewer samples for effective adaptation.

We sincerely thank the reviewers for their valuable suggestions and thoughtful comments. We would greatly appreciate any additional feedback that could further improve our work and are happy to incorporate further suggestions to ensure its quality. If our responses have satisfactorily resolved your concerns, we kindly request confirmation of whether the provided clarifications and changes address the issues raised and, if applicable, reconsideration of the scores in light of the responses and the updated manuscript.

Thank you again for your time and effort in reviewing our work.

We sincerely appreciate the reviewer’s recognition of our novel application of tensor decomposition for domain adaptation in time-series data. We are also grateful for the acknowledgment of our comprehensive experimental results across multiple datasets, which clearly highlight the computational benefits of our approach. Below we provide the responses to the concerns raised by the reviewer. All the changes in the revised version of the manuscript are highlighted in ${\color{blue}\text{Blue}}$.

Reviewer Weakness 1

While the theoretical analysis provides valuable insights into the trade-off between target data fit and source-target distribution divergence, the assumptions have limitations:
The posterior $Q(S)$ is assumed to be an isotropic Gaussian with variance matching the prior $P$. This assumption doesn't reflect real-world scenarios and is more like a typical variational inference solutions. A more realistic upper bound should incorporate: (1) Weight differences; (2) Distance between isotropic Gaussian posterior and true posterior. While such an extension appears feasible, the distance metric between isotropic Gaussian and true posterior remains undefined

Author Response

We sincerely thank you for your thoughtful feedback on our theoretical analysis. Your comments raise important points about the assumptions in our PAC-Bayesian generalization bound, and we appreciate the opportunity to clarify and expand upon these aspects.

We appreciate your concern regarding the assumption of an isotropic Gaussian posterior with variance matching the prior. We acknowledge that this simplification may not fully capture the complexities of real-world posterior distributions in deep neural networks. However, we believe that our approach provides a meaningful and tractable way to understand the generalization behavior in the context of fine-tuning.

1. Incorporating Weight Differences: Our theoretical framework does incorporate weight differences between the source (pre-trained) and target (fine-tuned) models. Specifically, in Theorem 1 of our paper, we derive a PAC-Bayesian generalization bound that includes the distance term. This term explicitly quantifies the sum of squared Frobenius norms of the differences between the target and source weights across all layers. By including this term in the bound, we directly account for the weight differences, addressing your first point about incorporating weight differences into a more realistic upper bound.

2. Distance Between Isotropic Gaussian Posterior and True Posterior: We agree that the true posterior in deep neural networks is likely complex and not accurately represented by an isotropic Gaussian. Hence, we are not able explicitly define a distance metric between the isotropic Gaussian posterior and the true posterior (due to the intractability of the true posterior).

3. Justification for the Isotropic Gaussian Assumption: The assumption of an isotropic Gaussian posterior is indeed a simplification, and it is a common practice in PAC-Bayesian analyses to enable analytical tractability. This choice allows us to derive explicit bounds and gain insights into the factors affecting generalization. Moreover, recent works [1,2,3] have successfully employed similar assumptions to obtain generalization bounds and practical insights into neural network training.

4. Support for Empirical Observations: The bound derived is not a strict one and is intended more as an analytical tool to gain insights into and support the underlying fine-tuning phase, rather than being a primary contribution of our work. The included theoretical analysis aims to complement our primary empirical findings. Therefore, we also empirically visualize the source-target weight parameter distance in Figure 5.

Nonetheless, incorporating the error term between the isotropic posterior and the true posterior, is an excellent suggestion. However, defining these metric/error rigorously for decomposed tensors under the Tucker framework remains a non-trivial extension, which we leave as future work.

References:

[1] Dziugaite, Gintare Karolina, and Daniel M. Roy. "Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data." Conference on Uncertainty in Artificial Intelligence (UAI), 2017.
[2] Li, Dongyue, and Hongyang R. Zhang. "Improved Regularization and Robustness for Fine-Tuning in Neural Networks."NeurIPS, 2021.
[3] Wang, Zifan, et al. "Improving Robust Generalization by Direct PAC-Bayesian Bound Minimization." CVPR, 2023.

Reviewer Question 1

Related Work: The paper would benefit from discussing connections to recent NLP transformer architectures, particularly:

• Dao et al. (2022), "Monarch: Expressive Structured Matrices for Efficient and Accurate Training"
• Qiu et al. (2024), "Compute Better Spent: Replacing Dense Layers with Structured Matrices"

Author Response

We appreciate the reviewer’s suggestion to discuss connections to recent NLP transformer architectures, particularly the works by Dao et al. (2022) and Qiu et al. (2024). In response, we have added a detailed discussion in the extended related work section of the revised manuscript, specifically in Section A.12 (Parameter Redundancy and Low-Rank Subspaces).

We sincerely thank the reviewers for their valuable suggestions and thoughtful comments. We would greatly appreciate any additional feedback that could further improve our work and are happy to incorporate further suggestions to ensure its quality. If our responses have satisfactorily resolved your concerns, we kindly request confirmation of whether the provided clarifications and changes address the issues raised and, if applicable, reconsideration of the scores in light of the responses and the updated manuscript.

Thank you again for your time and effort in reviewing our work.

Reviewer Weakness 4

Cite and compare with: • Tang et al, Source-Free Domain Adaptation with Frozen Multimodal Foundation Model, CVPR 2024
• Litrico et al, Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation, CVPR 2023
• Tang et al, Source-free domain adaptation via target prediction distribution searching, ICLR 2023
• Gao et al, UNITS: A Unified Multi-Task Time Series Model, NeurIPS 2024
• Ding et al, Source-Free Domain Adaptation via Distribution Estimation, CVPR 2022 It's important to compare against foundation model style approaches since they offer SoTA adaptation efficiency.

Author Response

We appreciate the reviewer’s observation on the importance of comparing against foundation model-style approaches for achieving state-of-the-art (SoTA) adaptation efficiency. Our work, however, intentionally addresses a different but complementary challenge in the domain of Source-Free Domain Adaptation (SFDA). As emphasized in Line 079-083 of the manuscript:

“It is important to emphasize that our contribution does not introduce a novel SFDA method for time-series data. Instead, we focus on making the target adaptation process more parameter- and sample-efficient and demonstrating that our framework can be seamlessly integrated with existing, and potentially future, SFDA techniques.”

Our work specifically targets parameter efficiency and sample efficiency for time-series data in the context of SFDA, a challenge that remains unexplored. This focus is critical for real-world applications in computationally constrained or low-data environments, where deploying large foundation models may not be practical due to their computational overhead or lack of niche domain-specific representation.

Citation and Comparison

To address the reviewer’s suggestion, we have incorporated citations for the following works in the added extended related works section (see Appendix Section A.12) and have discussed their relevance in the context of our research:

• Tang et al., "Source-Free Domain Adaptation with Frozen Multimodal Foundation Model," CVPR 2024.
• Litrico et al., "Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation," CVPR 2023.
• Tang et al., "Source-free domain adaptation via target prediction distribution searching," ICLR 2023.
• Gao et al., "UNITS: A Unified Multi-Task Time Series Model," NeurIPS 2024.
• Ding et al., "Source-Free Domain Adaptation via Distribution Estimation," CVPR 2022.

In terms of experimental comparisons, we adopted the setup and benchmarks established by Ragab et al. for a fair evaluation of our framework. We have clarified in the main text and Appendix Section A.9, Lines 1440-1442 that Tables 5–9 in the appendix present predictive performance comparisons with prior methods and SoTA results from Ragab et al. (Tables 2, 3, and 4 in [1] ). As these comparisons demonstrate, our framework achieves SoTA predictive performance while being significantly more parameter- and sample-efficient than existing methods.

Relevance to Foundation Model-Based Approaches

We acknowledge that foundation models, such as the ones referenced in Tang et al. and Gao et al., offer substantial benefits in learning generalized features across diverse domains and facilitating adaptation with limited target samples. However, our work provides a complementary perspective by focusing on fine-tuning efficiency , ensuring minimal computational overhead and adaptability to specialized domains. These aspects are particularly critical when foundation models are either unavailable or inefficient due to domain-specific mismatches, as noted in our extended related works section (Appendix Section A.12). Additonal the combination of utilizing foundation models along with our method could be an interesting direction to investigate and we will definitely be exploring it our future works (Appendix Section A.14).

We hope this response clarifies our position and demonstrates how our work complements existing foundation model-based approaches while addressing a distinct challenge adaptation efficiency.

References

[1] Ragab, Mohamed, et al. "Source-free domain adaptation with temporal imputation for time series data." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.

Reviewer Weakness 5

The method is not time series specific. Why not compare with vision models and datasets to show universality of the method?

Author Response

Thank you for your insightful comment highlighting the potential universality of our method. You are correct that the core principles of our approach are not inherently specific to time-series data. However, our primary motivation was to address specific challenges within the domain of Source-Free Domain Adaptation (SFDA) for time-series data—that has a research gap in terms of parameter and sample efficiency of the adaptation process.

In our research, we observed significant parameter redundancy along the channel (or variable) dimension in models trained on time-series data (see Figure 2A in the paper). This redundancy suggests substantial inter-channel dependencies that time-series data demonstrate as they propagate through deep networks. Previous studies [1,2] have explored leveraging these dependencies by applying convolutions along the variable dimension, reinforcing the notion that such inter-variable relationships are inherent and particularly pronounced in time-series data.

Building on these insights, we proposed decomposing the model along the channel dimension (as detailed in Equation 3 of our paper), and focusing on the principal channels or variables. This decomposition leverages the inherent inter-variable dependencies specific to time-series data, allowing us to achieve parameter efficiency without sacrificing performance.

While it is true that the mathematical underpinnings of our method could be applied to other domains like computer vision, the characteristics and dependencies in those domains may differ. For instance, vision models often deal with spatial and hierarchical features, and the inter-channel redundancies may manifest differently compared to time-series data. Therefore applying our method to vision datasets may or may not require addressing these different types of dependencies and might necessitate additional modifications beyond the scope of our current work.

Our goal was to keep the method general within the context of time-series SFDA to ensure broad applicability across various time-series datasets and adaptation scenarios. To this end, we avoid introducing specific inductive biases tied to datasets or the adaptation method. We validated our approach extensively on all five datasets from the AdaTime benchmark—conducting experiments on ten source-target pairs, including a many-to-one setting—providing a comprehensive evaluation within the time-series domain.

We appreciate your suggestion to explore the universality of our method by applying it to vision models and datasets. This is indeed an interesting direction for future research and could help in understanding the broader applicability of our approach across different domains. To this end we have added a Future Work section (Section A.14) to discuss explore the unversality of our method.

Reference

[1] Lai, Guokun, et al. "Modeling long-and short-term temporal patterns with deep neural networks." The 41st international ACM SIGIR conference on research & development in information retrieval. 2018.
[2] Luo, Donghao, and Xue Wang. "Moderntcn: A modern pure convolution structure for general time series analysis." ICLR. 2024 (spotlight).

We sincerely appreciate the reviewer’s recognition of the adaptability and versatility of our proposed method, particularly its ability to integrate seamlessly with several baseline SFDA methods to enhance adaptation efficiency. Below we provide the responses to the concerns raised by the reviewer. All the changes in the revised version of the manuscript are highlighted in ${\color{blue}\text{Blue}}$.

Reviewer Weakness 1

Writing needs to be improved. The tables with results and plots are very very tiny, and zoom 200% is needed to read them and evaluate. Please choose the most important results to present.

Author Response

Thank you for your valuable feedback. We have made revisions to prune the tables to improve its readability and visibility. We have carefully selected the most important results to present, streamlining the tables for better focus and clarity.

Reviewer Weakness 2

The loss functions for target adaptation are not presented, please be specific and self-contained.

Author Response

Thank you for your thoughtful feedback and for highlighting the need for more specificity regarding the target adaptation loss functions. We truly appreciate your attention to detail and your suggestion to make the manuscript more self-contained.

In response, we have added a new Section A.8 in the appendix, where we provide a comprehensive description of all the adaptation objectives used in our work. We believe this addition enhances the clarity and self-contained nature of the manuscript.

Reviewer Weakness 3

The authors do not compare against Lokra, Lora with the same style of parameter block freezing and tuning. This would improve fairness of comparisons.

Author Response

Thank you for your insightful suggestion to compare against LoRA and LoKrA with the same style of parameter block freezing and tuning. We greatly appreciate your recommendation to enhance the fairness of our comparisons.

In response to your feedback, we have incorporated this analysis and presented the results in the newly added Section A.10 in the appendix of the updated manuscript. We believe this addition provides a more comprehensive and balanced evaluation of our method.

We sincerely thank the reviewers for their valuable suggestions and thoughtful comments. We would greatly appreciate any additional feedback that could further improve our work and are happy to incorporate further suggestions to ensure its quality. If our responses have satisfactorily resolved your concerns, we kindly request confirmation of whether the provided clarifications and changes address the issues raised and, if applicable, reconsideration of the scores in light of the responses and the updated manuscript.

Thank you again for your time and effort in reviewing our work.

We sincerely appreciate the reviewer’s acknowledgment of the significance of the research problem, particularly its relevance to the challenges of fine-tuning full models in on-device settings. We are grateful for the recognition of our efforts in motivating the problem and articulating its importance in the introduction and methodology sections. The reviewer’s appreciation of the clarity in our definitions, equations, and the soundness of the proposed method is highly valued. We also thank the reviewer for highlighting our effort to connect the proposed approach to the Lottery Ticket Hypothesis, which provides an intuitive justification for its effectiveness. Lastly, we are pleased that the thoroughness of our experiments, including the diverse datasets, models, and state-of-the-art comparisons showing clear improvements, has been noted. Below we provide the responses to the concerns raised by the reviewer. All the changes in the revised version of the manuscript are highlighed in ${\color{blue}\text{Blue}}$.

Reviewer Weakness 1

A weakness of this work can be characterized as a missing link between the entire method and the time-series aspect of the problem. Specifically, it’s not entire clear why the method here is specifically applied to time-series, as the authors have setup here. Perhaps the authors can clarify how their method has leverage properties in time-series data to their advantage. If this connection is indeed not mentioned, then it seems reasonable to add a sentence or a section in the Appendix justifying this.

Author Response

Thank you for your thoughtful feedback and for highlighting an important aspect of our work. We genuinely appreciate your attention to detail and your efforts to help improve the interpretability of our method within the context of time-series data.

You are right in pointing out the missing link in our work to time series. Our primary motivation is to address the simultaneous parameter and sample efficiency in Source-Free Domain Adaptation (SFDA) for time-series, a pressing research gap we identified in the community. In our study, we observed significant parameter redundancy along the channel dimension of source models (see Figure 2A in the paper). This redundancy suggests substantial inter-channel dependency among variables in multivariate time-series features as they propagate through a deep network. Works by [1,2] has explored using convolution along the variable dimension to capture such cross-variable dependencies, reinforcing the notion that these dependencies are inherent in time-series data.

Building on this insight, we proposed decomposing the model along the channel dimension (as demonstrated in Equation 3 of our paper) and operating on the decomposed factors corresponding to the principal channels or variables. This approach leverages the inherent inter-variable dependencies in time-series data, allowing us to achieve parameter efficiency by focusing on the most significant channels without sacrificing performance.

Our aim was to keep the method as general as possible to ensure its applicability across different SFDA settings and diverse time-series datasets. To this end, we avoided introducing specific inductive biases and experimented with both generalized SFDA methods (such as SHOT, NRC, AAD) and a time-series-specific SFDA method (MAPU), following the initial work of Ragab et al [3]. Hence, we are empirically thorough, experimenting on all five datasets from the AdaTime benchmark [4]—where previous works typically focused on only three—and conducted experiments on ten source-target pairs, including a many-to-one setting where the source model is trained on multiple sources and then adapted to other targets.

To that end, we have added a Future Work section (Section A.14) in the revised version addressing the above discussion.

References
[1] Lai, Guokun, et al. "Modeling long-and short-term temporal patterns with deep neural networks." The 41st international ACM SIGIR conference on research & development in information retrieval. 2018.
[2] Luo, Donghao, and Xue Wang. "Moderntcn: A modern pure convolution structure for general time series analysis." ICLR. 2024 (spotlight).
[3] Ragab, Mohamed, et al. "Source-free domain adaptation with temporal imputation for time series data." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.
[4] Ragab, Mohamed, et al. "Adatime: A benchmarking suite for domain adaptation on time series data." ACM Transactions on Knowledge Discovery from Data 17.8 (2023): 1-18.

Reviewer Question 1

Can the authors comment on the time for inference before and after factorization? Does the smaller number of parameters reduce the time required for inference? Or does the increased number of matrix factorization increase the time required?

Author Response

The factorization also helps improve the inference (reduced inference time) by reducing the total number of computations involved. We appreciate the reviewer's insightful question and are happy to provide a detailed analysis of the inference analysis before and after applying Tucker decomposition to a convolutional neural network (CNN) layer. We also quantify the inference computation using multiply-accumulate operation counts (MACs), as mentioned in the manuscript.

Original Convolutional Layer:

Consider a convolutional layer with the following dimensions:

• Input Channels ($C_{\text{in}}$): 128
• Output Channels ($C_{\text{out}}$): 256
• Kernel Size ($K$): 8
• Input Length ($L$): Length of the input sequence
• Output Length ($L'$): Length of the output feature map The computational complexity of the original convolution operation is:

After Tucker Decomposition:

We apply Tucker decomposition along the channel dimensions (input and output channels) of the weight tensor. The decomposition factorizes the original weight tensor into smaller tensors, introducing two rank parameters:

• Input Rank ($R_{\text{in}}$): Reduced dimension for input channels
• Output Rank ($R_{\text{out}}$): Reduced dimension for output channels The computational complexity after Tucker decomposition becomes:

Components Computation Explanation:

1. Input Projection: Project the input feature maps from $C_{\text{in}}$ channels to $R_{\text{in}}$ channels:

2. Core Convolution: Perform convolution using the core tensor of size $R_{\text{out}} \times R_{\text{in}} \times K$:

3. Output Projection: Project the result from $R_{\text{out}}$ channels to $C_{\text{out}}$ output channels:

Computational Efficiency Demonstration:

For demonstration purposes, let's assume equal reduced ranks for input and output channels:

• $R_{\text{in}} = R_{\text{out}} = R$, where $R = 32$

Computations After Decomposition:

1. Input Projection:

2. Core Convolution:

3. Output Projection:

Total Computations After Decomposition:

Comparing Computational Costs:

Assuming the input and output lengths are approximately equal ($L \approx L'$), we can directly compare the total computations.

1. Total Computations Before Decomposition:

2. Total Computations After Decomposition:

3. Computational Reduction:

This indicates a reduction of approximately 92% in computational cost after Tucker decomposition.

Impact on Inference Time:

• Reduced Computation: The substantial decrease in computations directly translates to faster inference times, as the model performs significantly fewer operations.
• Efficient Projections: The additional projection operations involve smaller matrices due to the reduced rank $R$, making them computationally efficient.

The reduced ranks $R_{\text{in}}$ and $R_{\text{out}}$ lead to a lower computational complexity compared to the original convolutional layer.

We sincerely thank the reviewers for their valuable suggestions and thoughtful comments. We would greatly appreciate any additional feedback that could further improve our work and are happy to incorporate further suggestions to ensure its quality. If our responses have satisfactorily resolved your concerns, we kindly request confirmation of whether the provided clarifications and changes address the issues raised.

Thank you again for your time and effort in reviewing our work.

We sincerely appreciate the reviewer’s recognition of the cutting-edge nature of our work and the rich methodological framework, particularly our use of PAC-Bayes to demonstrate the effectiveness of the proposed approach. We are grateful for the acknowledgment of the detailed supplementary materials, including the derivation of the formal bound for parameter distance, as well as the sufficiency and quality of our experiments. Furthermore, we value the reviewer’s appreciation of our thorough elaboration and analysis of the limitations of our approach. Below we provide the responses to the concerns raised by the reviewer. All the changes in the revised version of the manuscript are highlighed in ${\color{blue}\text{Blue}}$.

Reviewer Weakness 1 and 4

In my opinion, in the Related Work section, what is shown is more accurately ablations rather than the so-called observations in the title. Compared to this experimental result, it is more necessary to simply explain why Tucker-style decomposition is used instead of other decomposition methods you have exemplified.

On line 238, “Building on these insights, we decompose the pre-trained source model weights using Tucker decomposition”. Although the expression here is rigorous. Logically, the former is a necessary condition for the latter. From the beginning of the article until now, there is no sufficient reason given for why Tucker decomposition is used instead of other decomposition methods. Even if it is mentioned earlier, it is better to emphasize it again here.

Author Response

Thank you for your insightful comments and for taking the time to review our manuscript. We acknowledge your point that the content in the Related Work section is more accurately described as ablations rather than observations. We haved revised the content accordingly to better reflect this. Additionally, we understand the importance of explicitly explaining why we chose Tucker-style decomposition over other decomposition methods. We appreciate your feedback and have carefully considered your suggestions. To that end, we have modified Section 2 (L146-152), Section 3 (L242-245) and have added Section A.12 in the Appendix (Extended Related Works).

1. Justification for Using Tucker Decomposition:

a. Natural Tensor Representation of Multi-Dimensional Time Series Data

• Multi-Modal Data Accommodation: Time series classification often involves multi-dimensional data, such as multiple time steps (temporal mode) and features (channel mode). Convolutional operations extract features from this data by leveraging its inherent structure.

• Effective Modeling of Interactions: Tucker decomposition naturally accommodates this multi-modal structure by disentangling temporal and channel information. For example, when dealing with EEG signals over time across different channels, Tucker decomposition effectively models interactions between time and channels. This is precisely what we exploit in defining Equations (4), (5), and (6).

b. Flexibility in Rank Selection

• Mode-Specific Ranks: Tucker decomposition allows for specifying different ranks for each mode. This flexibility is crucial when different dimensions of the data have varying levels of complexity.

c. Dimensionality Reduction and Feature Extraction

• Higher-Order Extension of PCA: Tucker decomposition acts as a higher-order extension of Principal Component Analysis (PCA), reducing the dimensionality of the data while preserving the most significant information.

• Improving Classification Performance: Lower-dimensional representations can enhance the performance of classification algorithms by reducing noise and mitigating the curse of dimensionality.

d. Interpretability

• Understanding Principal Components: The factor matrices obtained from Tucker decomposition represent the principal components in each mode (e.g., temporal and channel).

• Insight into Significant Features: These matrices could provide insights into which time points or features in the parameter space are most significant for classification, potentially aiding in interpretability and feature selection.

2. Comparison with Other Decomposition Methods:

a. CP Decomposition

• Limitation of Uniform Ranks: CP decomposition uses the same rank across all modes, which may not capture the varying complexities in the parameter space.

• Lack of Flexibility: While CP decomposition is highly interpretable, it lacks the flexibility of Tucker decomposition to model inter-modal interactions effectively.

b. Tensor-Train Decomposition

• Suitability for High-Order Tensors: Tensor-Train decomposition is more suitable for very high-order tensors and is less interpretable due to its chain-like core tensors.

• Ineffective Global Interaction Modeling: It may not capture global interactions as effectively as Tucker decomposition.

References:

[1] Tucker, Ledyard R. "The Extension of Factor Analysis to Three-Dimensional Matrices." Contributions to Mathematical Psychology, edited by Norman Frederiksen and Harold Gulliksen, Holt, Rinehart & Winston, 1964, pp. 110–127.
[2] Lathauwer, Lieven De, Bart De Moor, and Joos Vandewalle. "A Multilinear Singular Value Decomposition." SIAM Journal on Matrix Analysis and Applications, vol. 21, no. 4, 2000, pp. 1253–1278.
[3] Rabanser, Stephan, Oleksandr Shchur, and Stephan Günnemann. "Introduction to tensor decompositions and their applications in machine learning." arXiv preprint arXiv:1711.10781 (2017).
[4] Kolda, Tamara G., and Brett W. Bader. "Tensor Decompositions and Applications." SIAM Review 51 (2009): 455-500.

Reviewer Weakness 2

I think the insights in lines 201-204 need a more complete explanation. From Equation 1, I can only understand that the linear operation in deep learning is the so-called sequence of linear suboperations in the forward process. However, I cannot directly confirm whether such a description is also accurate for the backward process. If there are relevant formula derivations or proofs, references can be provided. If not, considering the space limitation, it is best to prove it in the Appendix. Because this claim is the premise of Equations 4, 5, and 6 later, which is necessary and important for understanding this method.

Author Response

Thanks for the suggestion. In the updated manuscript we have added Appendix Section A.3 (highlighted in ${\color{blue} \text{Blue}}$), that mathematically describes the forward pass and the backward pass equivalence through the decomposed factors

1. Forward Propagation

Case 1: Using $\mathbf{W} = \mathbf{U}^{(1)} \mathbf{G} (\mathbf{U}^{(2)})^\top$ directly

Let the input to the layer be $x$. Forward propagation through $\mathbf{W}$ is:

Case 2: Using $\mathbf{U}^{(1)}, \mathbf{G}, (\mathbf{U}^{(2)})^\top$ separately

We compute the forward pass in steps:

1. Compute $y_1 = (\mathbf{U}^{(2)})^\top x$,
2. Compute $y_2 = \mathbf{G} y_1 = \mathbf{G} \big( (\mathbf{U}^{(2)})^\top x \big)$,
3. Compute $z = \mathbf{U}^{(1)} y_2 = \mathbf{U}^{(1)} \big( \mathbf{G} \big( (\mathbf{U}^{(2)})^\top x \big) \big)$.

By associativity of matrix multiplication, this is equivalent to:

Conclusion (Forward Pass): Forward propagation through $\mathbf{U}^{(1)}, \mathbf{G}, (\mathbf{U}^{(2)})^\top$ in sequence is equivalent to propagating through $\mathbf{W}$ directly.

2. Backward Propagation

Let the gradient of the loss $\mathcal{L}$ with respect to the output $z$ be $\frac{\partial \mathcal{L}}{\partial z} = g_z$. The task is to propagate this gradient backward.

Case 1: Using $\mathbf{W}= \mathbf{U}^{(1)} \mathbf{G} (\mathbf{U}^{(2)})^\top$ directly

The gradient of the loss with respect to $x$ is:

Case 2: Using $\mathbf{U}^{(1)}, \mathbf{G}, (\mathbf{U}^{(2)})^\top$ separately

Backward propagation proceeds as:

1. Compute $g_{y_2} = (\mathbf{U}^{(1)})^\top g_z$,
2. Compute $g_{y_1} = \mathbf{G}^\top g_{y_2} = \mathbf{G}^\top \big( (\mathbf{U}^{(1)})^\top g_z \big)$,
3. Compute $g_x = \mathbf{U}^{(2)} g_{y_1} = \mathbf{U}^{(2)} \big( \mathbf{G}^\top \big( (\mathbf{U}^{(1)})^\top g_z \big) \big)$.

By associativity of matrix multiplication, this simplifies to:

Gradients with respect to $\mathbf{U}^{(1)}, \mathbf{G}, (\mathbf{U}^{(2)})^\top$

The gradients of $\mathcal{L}$ with respect to the parameters are:

1. Gradient with respect to $\mathbf{U}^{(1)}$: $$\frac{\partial \mathcal{L}}{\partial \mathbf{U}^{(1)}} = g_z y_2^\top = g_z \big( \mathbf{G} \big( (\mathbf{U}^{(2)})^\top x \big) \big)^\top$$
2. Gradient with respect to $\mathbf{G}$: $$\frac{\partial \mathcal{L}}{\partial \mathbf{G}} = g_{y_2} y_1^\top = \big( (\mathbf{U}^{(1)})^\top g_z \big) \big( (\mathbf{U}^{(2)})^\top x \big)^\top$$
3. Gradient with respect to $(\mathbf{U}^{(2)})^\top$: $$\frac{\partial \mathcal{L}}{\partial (\mathbf{U}^{(2)})^\top} = g_{y_1} x^\top = \big( \mathbf{G}^\top \big( (\mathbf{U}^{(1)})^\top g_z \big) \big) x^\top$$

These computations are consistent with the chain rule applied to $\mathbf{W} = \mathbf{U}^{(1)} \mathbf{G} (\mathbf{U}^{(2)})^\top$.

3. Equivalence of Forward and Backward Passes

• Forward Pass: The output $z$ is identical whether computed via $\mathbf{W}$ or via $\mathbf{U}^{(1)}, \mathbf{G}, (\mathbf{U}^{(2)})^\top$ separately.
• Backward Pass: The gradient $g_x$ and the parameter gradients ($\frac{\partial \mathcal{L}}{\partial \mathbf{U}^{(1)}}, \frac{\partial \mathcal{L}}{\partial \mathbf{G}}, \frac{\partial \mathcal{L}}{\partial (\mathbf{U}^{(2)})^\top}$) are identical whether computed via $\mathbf{W}$ or via $\mathbf{U}^{(1)}, \mathbf{G}, (\mathbf{U}^{(2)})^\top$ separately.

Thus, forward and backward propagation through the decomposed factors $\mathbf{U}^{(1)}, \mathbf{G}, (\mathbf{U}^{(2)})^\top$ is mathematically equivalent to propagating through $\mathbf{W}$ directly.

Reviewer Weakness 3

A small mistake. In the sentence after Eq3, on line 221, after “where”, it should be an explanation of the symbol V in Equation 3 instead of U.

Thank you for bringing this oversight to our attention. You are absolutely correct. We have corrected this error in the manuscript to accurately reflect the notation used in Equation (3). We appreciate your attention to detail.

Reviewer Question 1

Lines 248-252, why is fine-tuning for 2-3 epochs? Is this number closely related to the dataset and model scale? I am not very familiar with this field and hope that relatively rigorous evidence can be given here in the text.

Author Response

Thank you for your insightful question regarding the fine-tuning process after the decomposition step. The necessity for a limited additional fine-tuning arises because decomposing the weight parameters introduces approximation errors, due to the decomposition conducted, that accumulate across layers, leading to a degradation in the model's predictive performance on the source data. Such similar observation have been prominent in the literature too [1,2]

We empirically determined that fine-tuning the model for 2-3 epochs effectively restores its original performance. Specifically, we fine-tuned for additional 3 epochs on the SSC dataset and for 2 epochs on the UCI-HAR, MFD, HHAR, and WISDM datasets. This duration was chosen by monitoring the source empirical risk during fine-tuning until they returned to their pre-decomposition levels.

The number of fine-tuning epochs depends on the specific characteristics of the dataset and the complexity of the model architecture. In Figure 11 (appendix) of the updated manuscript, we present empirical evidence showing that the model's empirical risk converges back to its initial value within the specified epochs, demonstrating effective performance recovery. We emphasize that the optimal number of fine-tuning epochs should be determined empirically for each case to balance performance restoration and computational efficiency. This approach ensures that the model quickly recovers its original capabilities.

We hope this explanation provides the rigorous evidence you were seeking and clarifies the rationale behind our fine-tuning strategy.

References:

[1] Kim, Yong-Deok, et al. "Compression of deep convolutional neural networks for fast and low power mobile applications." ICLR 2016
[2] Dong, Xin, Shangyu Chen, and Sinno Pan. "Learning to prune deep neural networks via layer-wise optimal brain surgeon." NeurIPS 2017

We sincerely thank the reviewers for their valuable suggestions and thoughtful comments. We would greatly appreciate any additional feedback that could further improve our work and are happy to incorporate further suggestions to ensure its quality. If our responses have satisfactorily resolved your concerns, we kindly request confirmation of whether the provided clarifications and changes address the issues raised and, if applicable, reconsideration of the scores in light of the updated manuscript.

Thank you again for your time and effort in reviewing our work.

• Based on reviewer 56c7's suggestion, we modify Section 2 (Lines 146-152) and Section 3 (Lines 242-245) to provide a clearer and proper justification of the use of Tucker-Decomposition. Additionally we extend Section 2 in Appendix Section A.12 (Parameter Redundancy and Low-Rank Subspaces.), for an elaborative description.

• Based on reviewer 56c7's suggestion, we prove/show the forward and backward pass equivalence, with the reparametrized parameter in Section A.3

• Based on reviewer 56c7's concern, in Figure 11, we provide the justification to our brief finetuning during the source-model preparation (for 2-3 epochs) after the decomposition.

• Based on reviewer bZkd, gGV1, and G1Yq's suggestion, in Appendix Section A.12 (Our Motivation and Link to Time-Series.) discuss the motivation of our work and the potential properties in time-series data that we leverage and links that we exploit.

• Based on reviewer bzkd's question, we include an elaborative analysis of the reduction in computation, that translates to faster inference in Appendix Section A.2.3.

• Based on reviewer gGV1's suggestion, we have made revisions to prune the tables to improve its readability and visibility (see Table 1 and Table 2). we have carefully selected the most important results to present, streamlining the tables for better focus and clarity.

• Based on reviewer gGV1 and G1Yq's suggestion, we have added a new Section A.8 in the appendix, where we provide a comprehensive description of all the adaptation objectives used in the paper.

• Based on reviewer gGV1's suggestion, in section Appendix Section A.10 (Figure 13, 14, and 15), we have incorporated the comparison of our method against LoRA and LoKrA with the same style of parameter block freezing and tuning.

• Based on reviewer gGV1's concern, we cite and provide justification of our comparisons with the work referenced by reviewer gGV1. We have discussed the referenced works in newly added Appendix Section A.12 and Section A.14.

• Based on reviewer vYWA's suggestion, we have added a discussion on the relation of our work to Monarch and Block Tensor Train parameterization in Appendix Section A.12 (Parameter Redundancy and Low-Rank Subspaces.).

Below, we address a few shared concerns on the application of the PAC-Bayesian Analysis framework to our method. Note: Our contribution is not to innovate on PAC-Bayesian theory itself; rather, we use it as a lens through which the impact of decomposition and selective fine-tuning on sample efficiency can be understood.

Simplifying Assumptions in PAC-Bayesian Analysis
The choice of an isotropic Gaussian posterior and prior is a widely used simplification in PAC-Bayesian analyses, primarily to ensure analytical tractability. This assumption enables the derivation of explicit bounds, offering insights into the factors influencing generalization. Despite being a simplification, its practical utility has been validated in prior work [1, 2, 3], where similar assumptions were leveraged to obtain meaningful generalization bounds and actionable insights into neural network training. This forms a solid theoretical basis for our approach.

Contextualizing the PAC-Bayesian Framework
The PAC-Bayesian framework is fundamentally designed to measure generalization error—quantifying the gap between empirical loss (on sampled data) and expected loss (over the true data distribution). In this context:

1. Prior Distribution ($P$) encapsulates our belief about model parameters before observing data.
2. Posterior Distribution ($Q$) reflects the updated belief after training on a sampled dataset.
3. Sampled Data ($S$) provides examples drawn from the data distribution ($D$), forming the basis for training and evaluation.

In our specific application, we adapt this framework to the process of fine-tuning pre-trained neural networks, establishing a clear correspondence between these theoretical elements and practical scenarios.

Relating PAC-Bayesian Concepts to Our Fine-Tuning Setup
Fine-tuning a pre-trained model aligns naturally with the PAC-Bayesian framework. Here’s how:

1. Prior as Pre-trained Weights ($P = W_{\text{src}}$):
   The pre-trained model weights ($W_{\text{src}}$) serve as the prior, encapsulating knowledge from the source dataset. This initialization provides a structured starting point for adaptation.

2. Fine-tuning on Target Data ($S = D_{\text{trg}}$):
   The sampled target data ($S$) updates the model through fine-tuning, effectively bridging the gap between the source and target distributions.

3. Posterior as Fine-tuned Weights ($Q = W_{\text{trg}}$):
   The fine-tuned weights ($W_{\text{trg}}$) represent the posterior, reflecting the model’s state after integrating knowledge from the target dataset.

This mapping ensures coherence between the PAC-Bayesian framework and our practical fine-tuning scenario, preserving the integrity of theoretical assumptions.

Applicability to a Single Data Distribution
A critical aspect of our setup is that fine-tuning occurs exclusively within the target distribution ($D_{\text{trg}}$). This adherence guarantees that both the empirical loss (measured on $S = D_{\text{trg}}$) and the expected loss (over $D_{\text{trg}}$) are defined with respect to the same distribution. Consequently:

• The prior and posterior remain connected through data sampled from a single distribution, aligning with PAC-Bayesian principles.
• This coherence ensures that generalization bounds derived under this framework remain valid, bolstering the applicability of our approach.

Rationale for the Replacement
By modeling the pre-trained weights as the prior and the fine-tuned weights as the posterior, we achieve alignment with PAC-Bayesian assumptions while providing a framework for measuring generalization. This approach justifies the use of PAC-Bayesian bounds to evaluate fine-tuned models, especially in scenarios emphasizing sample efficiency.

Connecting to Our Empirical Contributions
The use of PAC-Bayesian bounds is integral to contextualizing our empirical findings. Specifically:

• Insight into Sample Efficiency: PAC-Bayesian analysis helps elucidate why decomposing the model and selectively fine-tuning certain parameters enhances sample efficiency.
• Theoretical Support for Observations: The simplified bound highlights the role of parameter distance, bridging the gap between decomposition strategies and their implications for adaptation.

Conclusion
By staying within the same distribution throughout the fine-tuning process and adhering to PAC-Bayesian principles, we ensure that our theoretical approach remains robust. The analysis not only complements our empirical results but also provides a solid theoretical foundation for understanding the observed improvements in sample efficiency and regularization.

Motivation and Research Gap
The primary motivation for this work was to improve the parameter and sample efficiency of different Source-Free Domain Adaptation (SFDA) techniques as applied to time-series data. Many of the reviewers accurately pointed out that our method is not specific to time-series data and urged us to test this approach on other modalities (e.g. vision). We intentially limited the scope of our method to time-series data for two reasons: 1. there is very little work in making SFDA methods for time-series models parameter and sample efficient (this is less true in vision for example), and 2. we wanted to exploit properties that are unique to time-series datasets. On the latter point, time-series models often exhibit significant parameter redundancy along the channel dimension, as evidenced in Figure 2A of our paper. This redundancy reflects strong inter-channel dependencies among variables. Related works [1, 2] have demonstrated the effectiveness of leveraging these dependencies through convolution along the variable dimension, further supporting this observation.

Proposed Methodology
Building on this insight, we proposed decomposing the model along the channel dimension (Equation 3) and focusing on the most significant components of the decomposition. This selective operation allows us to exploit inter-variable dependencies while reducing parameter redundancy. By concentrating on principal channels—those most critical to the parameter space—we achieve a compact model representation that maintains performance. The empirical results in Section 4 confirm the effectiveness of this approach, demonstrating consistent performance across diverse datasets, SFDA methods, and sample-ratios.

Generality and Applicability
Although our method is grounded in time-series data, its underlying mathematical principles are general and could extend to other domains, such as vision. Each domain however has its own unique characteristics that must be taken into consideration, such as spatial and hierarchical dependencies in the case of vision datasets. As a result, inter-channel redundancies may manifest differently compared to time-series data, and may require special considerations that while interesting (and pave the way for future research), are beyond the scope of this work.

Empirical Validation and Thoroughness
We aimed to ensure the broad applicability of our method across various SFDA settings by avoiding specific inductive biases. To this end, we evaluated our method using both generalized SFDA techniques (SHOT [3], NRC [4], AAD [5]) and time-series-specific methods (MAPU [6]), conducting extensive experiments on five datasets from the AdaTime benchmark [7]. These datasets include diverse time-series tasks and source-target settings, moreover we exten the experiment scenario to many-to-one adaptation setting, showcasing the robustness and generality of our approach.

Positioning as a Foundational Framework
The proposed method can be seen as a foundational framework that balances simplicity with versatility. It serves as a strong baseline or starting point for more advanced methods incorporating domain knowledge. Its generality ensures applicability across different SFDA settings while providing a solid platform for future research.

Conclusion
By addressing the simultaneous challenges of parameter and sample efficiency in SFDA for time-series, our work bridges a critical research gap. The combination of theoretical rigor, empirical thoroughness, and practical versatility ensures that our contributions provide a meaningful step forward for the community. We are grateful for the reviewers' recognition of these efforts and their constructive feedback to enhance the work.

References
[1] Lai, Guokun, et al. "Modeling long-and short-term temporal patterns with deep neural networks." SIGIR, 2018.
[2] Luo, Donghao, and Xue Wang. "Moderntcn: A modern pure convolution structure for general time series analysis." ICLR, 2024 (spotlight).
[3] Liang, Jian, Dapeng Hu, and Jiashi Feng. "Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation." In ICML, 2020.
[4] Yang, Shiqi, et al. "Exploiting the intrinsic neighborhood structure for source-free domain adaptation." In NeurIPS, 2021.
[5] Yang, Shiqi, Shangling Jui, and Joost van de Weijer. "Attracting and dispersing: A simple approach for source-free domain adaptation." In NeurIPS, 2022.
[6] Ragab, Mohamed, et al. "Source-free domain adaptation with temporal imputation for time series data." KDD, 2023.
[7] Ragab, Mohamed, et al. "Adatime: A benchmarking suite for domain adaptation on time series data." TKDD, 2023.