TELEPORTATION WITH NULL SPACE GRADIENT PROJECTION FOR OPTIMIZATION ACCELERATION

Response to “The approximation scheme is not very clearly described. The goal of the approximation is only defined in words, and the error we should expect is not analyzed”:

The pseudocode for teleportation with null space gradient projection is already provided in Appendix A.1. Unlike the linear approximation approach proposed by Mishkin et al. (2024), our algorithm does not necessarily involve any form of approximation. When the SVD threshold is set to 1, as in our experiments (except for Penn Treebank, where a threshold of 0.99 is used), the gradient is projected onto the exact null space, ensuring that the loss remains exactly invariant. The SVD threshold serves as a hyperparameter that introduces additional flexibility, and its behavior is empirically analyzed in Section 4.5.

Response to “A major claim is the improved speed over past teleportation approaches, such as the one using symmetry or the one using linear approximation”:

Comparison among algorithms in the context of teleportation presents significant challenges. This difficulty arises because the mechanisms for maintaining the invariance of loss during teleportation differ among the approaches, such as the group-action-based method proposed by Zhao et al. (2022), our method, and the approach described in Mishkin et al. (2024). Designing a fair comparison across these baselines is hard. The only reasonable approach we can think of is to compare the methods using their respective best hyperparameter settings. However, grid-searching for the optimal set of hyperparameters is highly impractical, if not infeasible, due to the sensitivity and versatility of teleportation hyperparameters. Moreover, the "best" hyperparameters often involves leveraging significantly more computational resources than a typical or reasonable setting would allow. For instance, Zhao et al. (2022) demonstrated that scheduling five teleportation epochs performs better than one epoch, and using 32 teleportation steps outperforms eight steps. While, in principle, performing as many teleportation steps as possible before each main task update would yield optimal performance, this approach is computationally prohibitive.

Given these considerations, we believe the more critical contribution lies in increasing the applicability of teleportation by designing a more efficient and generalizable algorithm. Accordingly, our experiments focus on comparing the efficiency and generalizability of the methods, which we believe provides a more meaningful and practical evaluation.

Response to “The observed improvement is mainly in training performance, with minimal improvement in generalization”:

The primary focus of this paper is to extend the teleportation technique to a broader class of functions while proposing a more efficient algorithm. The use of gradnorm as the teleportation objective function was intended as a demonstration of the effectiveness of our method in accelerating convergence and is not necessarily designed to enhance generalization. We appreciate your observation regarding this point, as it highlights an important branch of research concerning the choice of teleportation loss. As demonstrated in Zhao et al. (2023), the selection of teleportation loss can indeed yield different outcomes; for instance, employing a teleportation loss related to curvature has been shown to improve generalization on test sets.

While our work does not explore teleportation with alternative objective functions that may enhance generalization, we believe that the acceleration in training convergence itself is a noteworthy contribution. As computational efficiency becomes increasingly critical in the era of big data and modern machine learning, our findings address a pressing concern in the field and warrant further attention.

Response to “What does “enhance the gradient norm” mean?”:

This simply refers to an "increase" in the gradient norm.

Response to “In what sense is symmetry teleportation a state-of-the-art algorithm?”:

To date, there are only two teleportation algorithms: symmetry teleportation (Zhao et al., 2022) and the linear approximation method (Mishkin et al., 2024). None of the existing works compare against any other teleportation algorithms, as conducting a fair comparison is extremely challenging. We consider both methods to be state-of-the-art, as they are the only established algorithms in this domain, and no qualitative performance comparison has been conducted between the two approaches.

Response to “It is strange to cite a 2018 applications paper for MLPs ”:

We sincerely thank you for pointing this out. We have carefully reviewed and updated all citations to ensure they are accurate and appropriately representative of the referenced work.

Response to “The authors mix layer-wise and global operations without a clear explanation or justification”:

We admit that the shift from the general explanation in Equations (4) and (5) to the layerwise notation in Section 3 and beyond may indeed cause some confusion. It is important to note that the null space gradient projection operation only works at the layerwise level, as each layer's weight is linearly multiplied by the layer's input. Specifically, the gradient of each layer is projected onto the null space of the input of respective layer. This ensures that each layer's output remains invariant, thereby preserving the invariance of the input to the subsequent layer, and ultimately, the final output and the loss of the entire model. To address this, we have revised all relevant expressions to consistently use layerwise notation.

Response to “Relatedly, reusing 𝜋 in Equations (11-13) is incorrect, as these equations seem to imply different dimensionality”:

This issue should now be resolved, as we have updated all projections to explicitly use layerwise notation. Additionally, to avoid ambiguity, we have revised the notation for π, adding a subscript l to clearly indicate the layer-specific projection. We have modified the manuscript accordingly.

Response to “The statement that "the gradient of the teleportation objective function resides within the space spanned by the input data" is incorrect and misleading":

You are absolutely correct that the statement, along with related ones, all refers to the input data of each specific layer. We initially believed the statement was clear, as we used layerwise notation from Section 3 onward, but we did not fully account for potential confusion arising from the earlier sections. To address this, we have revised the paper to ensure the statements are precise and explicitly refer to each layer where appropriate.

Response to “Can the authors explain whether this order affects the theoretical claims or the projected outcome, and if so, in what ways?”:

The order of operations is solely determined by the column definition of x in the MLP and the definition of the matrix X, with row features in CNNs and self-attention, as established in Section 3. If we were to redefine x in the MLP as a row vector, the order of multiplication would be the same as in MLP and self-attention.

Response to “Explanation on null space gradient projection”:

Updating the model weights with the gradient of the teleportation loss projected onto the RGS effectively ensures that the output of each layer remains invariant. Consequently, the original loss also remains invariant, as it depends solely on the output and the corresponding labels. While the original loss remains unchanged, the teleportation loss can still be minimized during the teleportation step because it is dependent only on the weights, rather than on the outputs of the layers.

Response to “Lack of comparison with other state-of-the-art methods”:

To date, there are only two teleportation algorithms: symmetry teleportation (Zhao et al., 2022) and the linear approximation method (Mishkin et al., 2024). None of the existing works compare against any other teleportation algorithms, as conducting a fair comparison is extremely challenging.

This difficulty arises because the mechanisms for maintaining the invariance of loss during teleportation differ among the approaches. Designing a fair comparison across these baselines is hard. The only reasonable approach we can think of is to compare the methods using their respective best hyperparameter settings. However, grid-searching for the optimal set of hyperparameters is highly impractical, if not infeasible, due to the sensitivity and versatility of teleportation hyperparameters. Moreover, the "best" hyperparameters often involves leveraging significantly more computational resources than a typical or reasonable setting would allow. For instance, Zhao et al. (2022) demonstrated that scheduling five teleportation epochs performs better than one epoch, and using 32 teleportation steps outperforms eight steps. While, in principle, performing as many teleportation steps as possible before each main task update would yield optimal performance, this approach is computationally prohibitive.

Given these considerations, we believe the more critical contribution lies in increasing the applicability of teleportation by designing a more efficient and generalizable algorithm. Accordingly, our experiments focus on comparing the efficiency and generalizability of the methods, which we believe provides a more meaningful and practical evaluation.

Response to “A runtime comparison with non-teleportation optimizers counterparts (SGD, Momentum, Adagrad, Adam) is absent”:

Since you are specifically referring to non-teleportation optimizers, we assume this pertains to the graphs in Section 4.4, where only the runtime of group-action-based teleportation and our method is included. This is because these graphs focus on the runtime of teleportation steps, which are not applicable to non-teleportation algorithms as they do not involve teleportation runtime.

However, since your comment pertains to different optimizers, it is possible that you are referring to the loss trajectory graphs, which include both teleportation and non-teleportation optimizers.

Alternatively, you may be referencing the wall-clock runtime versus loss graph, which is currently absent for both teleportation and non-teleportation optimizers. Zhao et al. (2022) demonstrated that symmetry teleportation achieves faster convergence in loss versus wall-clock time compared to non-teleportation algorithms, consistent with the trends observed in loss versus epoch graphs. Given that our method has been shown to be significantly more efficient than their symmetry teleportation algorithm, we are confident that our method would also demonstrate superior performance on the loss versus wall-clock time convergence graph.

To address this gap, we will include these graphs across all architectures in the future revised version of the paper.

Response to “Wall-clock comparison of convergence”:

Zhao et al. (2022) demonstrated that the convergence of symmetry teleportation with respect to loss versus wall-clock time is faster compared to non-teleportation algorithms, consistent with the trends observed in the loss versus epoch graph. Since our results have shown that our method significantly outperforms their symmetry teleportation algorithm in terms of efficiency, it can be reasonably inferred that our approach would exhibit superior performance on the loss versus wall-clock time convergence graph as well. We acknowledge the importance of presenting these graphs and will incorporate them in appendix in the future revised version of the paper to provide a comprehensive comparison.

Response to “Difference in arrived solutions”:

The primary focus of this paper is to extend the teleportation technique to a broader class of functions while proposing a more efficient algorithm. The use of gradnorm as the teleportation objective function was intended as a demonstration of the effectiveness of our method in accelerating convergence and is not necessarily designed to enhance generalization. We appreciate your observation regarding this point, as it highlights an important branch of research concerning the choice of teleportation loss. As demonstrated in Zhao et al. (2023), the selection of teleportation loss can indeed yield different outcomes; for instance, employing a teleportation loss related to curvature has been shown to improve generalization on test sets.

While our work does not explore teleportation with alternative objective functions that may enhance generalization, we believe that the acceleration in training convergence itself is a noteworthy contribution. As computational efficiency becomes increasingly critical in the era of big data and modern machine learning, our findings address a pressing concern in the field and warrant further attention.

Response to “Stability of the hyperparameters”:

The sensitivity of teleportation hyperparameters is indeed high, as noted in the discussion section. For instance, Zhao et al. (2022) demonstrated that performance can improve significantly by scheduling more teleportation epochs and steps. The hyperparameters used in our experiments primarily follow the conventions established in their work. The sensitivity of hyperparameters also highlights the need to carefully study the tradeoff between performance and computational cost, which makes algorithmic efficiency even more critical. Therefore, the primary motivation behind our work is to provide an algorithm that is significantly more efficient than existing approaches. Nevertheless, we will include additional ablation studies in the future revised version of the paper.

The ablation studies on the threshold of SVD is present in section 4.5.

Response to “Comparison to group action based method”:

To date, there are only two teleportation algorithms: symmetry teleportation (Zhao et al., 2022) and the linear approximation method (Mishkin et al., 2024). None of the existing works compare against any other teleportation algorithms, as conducting a fair comparison is extremely challenging.

This difficulty arises because the mechanisms for maintaining the invariance of loss during teleportation differ among the approaches. The only reasonable approach we can think of is to compare the methods using their respective best hyperparameter settings. However, grid-searching for the optimal set of hyperparameters is highly impractical, if not infeasible, due to the sensitivity and versatility of teleportation hyperparameters. Moreover, the "best" hyperparameters often involves leveraging significantly more computational resources than a typical or reasonable setting would allow. For instance, Zhao et al. (2022) demonstrated that scheduling five teleportation epochs performs better than one epoch, and using 32 teleportation steps outperforms eight steps. While, in principle, performing as many teleportation steps as possible before each main task update would yield optimal performance, this approach is computationally prohibitive.

Given these considerations, we believe the more critical contribution lies in increasing the applicability of teleportation by designing a more efficient and generalizable algorithm. Accordingly, our experimental section focuses on comparing the efficiency and generalizability of the methods, which we believe provides a more meaningful and practical evaluation.

Response to “Poor referencing of related work”:

We sincerely thank you for pointing this out. We have carefully reviewed and updated all citations to ensure they are accurate and appropriately representative of the referenced work.

Additionally, we have revised the input space to be explicitly described as the layerwise input space for improved clarity. We have also corrected the title to reflect "Tiny ImageNet" instead of "ImageNet" for consistency.