Hyper-Connections

Weakness3: Sorry for any confusion regarding the structure and flow of our paper. Let us provide a clearer overview of the organization and the rationale behind each section:

Section 1: Introduction

(1) We begin by outlining our motivation: enabling neural networks to autonomously learn the optimal strength of connections to improve performance.

(2) We introduce our key idea: expanding the layer input to $n$ input vectors, connecting different input vectors (width-connections), feeding them into the layer to get the output vector, and further connecting the input vectors and the layer output vector (depth-connections).

Section 2: Method

2.1: We formally define the mathematical formulation of $\mathcal{HC}$ for controlling depth-connections and width-connections mentioned in Section 1

2.2: We present a dynamic version where $\mathcal{HC}$ depends on the input, achieving even better performance.

2.3: We explain the initialization for $\mathcal{HC}$, which is crucial for training convergence.

Section 3: Further Analysis

3.1: We compare our approach with ordinary PostNorm/PreNorm, demonstrating that they are special cases of our hyper-connections, thereby making our method broadly applicable.

3.2: We discuss sequential and parallel duality, showing that hyper-connections can learn to arrange layers, which is a promising direction for designing more representative foundation models.

Section 4: Experiments

(1)We present comprehensive ablation studies and comparisons with OLMo and OLMoE models using 6 tables and 6 figures.

(2)We include visualization analysis to demonstrate that our model learns parallel block patterns and highlights some interesting findings.

Section 5 and Section 6

We review relevant literature and summarize our contributions

We greatly appreciate the positive feedback from reviewers VJDJ, DUBh, and HHYn, who praised the paper for being “a clear and systematic extension”. We are committed to ensuring that our paper is as clear and well-structured as possible. If you have any specific sections that you find difficult to understand or feel could be improved, please let us know. Your feedback will be invaluable in enhancing the readability and clarity of our paper.

Question1: We sincerely appreciate the feedback and recognize that our original explanation may not have been sufficiently clear. In the paper, we rephrased it as follows: "In contrast, Post-Norm applies normalization after the output of each residual block, reducing the influence of a hidden state on subsequent layers."

What we intended to convey is that the influence of the output hidden state of each layer on the subsequent layers decreases. Specifically, suppose $h_i$ is the output of the i-th layer, and $h_j$ is the output of the j-th layer, where i>j.

For PreNorm, we have: $h_i = L_{i-1}(h_{i-1}) + h_{i-1} = L_{i-1}(h_{i-1}) + L_{i-2}(h_{i-2}) + h_{i-2} = \Sigma_{k=j}^{i-1} L_k(h_k) + h_j$

For PostNorm, we have:$h_{i}=`Norm`(L(h_{i-1})+h_{i-1}) =\frac{L(h_{i-1})+h_{i-1}}{\sqrt{var(L(h_{i-1}))+var(h_{i-1})+2\cdot covar(L(h_{i-1}), h_{i-1}))}}$

Let: $w_{i-1}=\frac{L(h_{i-1})+h_{i-1}}{\sqrt{var(L(h_{i-1}))+var(h_{i-1})+2\cdot covar(L(h_{i-1}), h_{i-1}))}}$,

Typically, we assume $covar(L(h_{i}), h_{i}))=0$ , as stated in Section 4.1 of the paper.

Since $h_{i}$ has already been normalized, $var(h_{i})=1$, and therefore $w_{i-1}<1$. Finally, we have: $h_i=w_{i-1}L_{i-1}(h_{i-1}) + w_{i-1}h_{i-1} =w_{i-1}L_{i-1}(h_{i-1}) + w_{i-1}(w_{i-2}L_{i-1}(h_{i-2}) + w_{i-2}h_{i-2})=\Sigma_{k=j}^{i-1}(\Pi_{a=k}^{i-1}w_a)L_k(h_k)+\Pi_{k=j}^{i-1}w_kh_j.$

$\Pi_{k=j}^{i-1}w_k<1$ represents the influence factor of $h_j$ on $h_i$. Since the product of several values less than 1 decays rapidly, the influence of $h_j$ on subsequent outputs diminishes as the number of layers increases.

Question2: Yes, we have added comparisons of the parameter count and FLOPs, as well as the formulas for calculating the parameter count, in Appendix B. Additionally, we have included the table in response to Weakness 2.

We greatly appreciate the time you have taken to review our manuscript. In response to your comments, we address each point individually.

Weakness1: Thank you for your suggestions. We have revised Figure 2 and added an additional architecture diagram in Figure 8 of Appendix A for more intuitive illustration. You can also check the special $n = 2$ case in Figure 4 for more clear understanding. For the pseudocode, see Algorithm 1. The hyper-connection is designed to address the gradient vanishing and representation collapse challenges stemming from Pre-Norm and Post-Norm. Our motivation is to enable neural networks to autonomously learn the optimal strength of connections to improve performance. Unlike typical residual connections that only connect the input vector and the layer output vector, we expand the layer input to $n$ input vectors, connect different input vectors (width-connections), feed them into the layer to get the output vector, and then further connect the input vectors and the layer output vector (depth-connections).

We greatly appreciate specific suggestions for enhancement. Reviewer VJDJ praised the paper as “well detailed and mathematically sound,” DUBh noted “thorough analysis” and “nice visualizations,” and HHYn commended it as “a clear and systematic extension”. We kindly request that if you encountered any sections that were difficult to understand or felt could be improved, you could point these out specifically. This feedback would be invaluable in enhancing the readability and clarity of our paper.

Weakness2: To clarify, the number of parameters and the computational cost of our model are almost the same as the original model, with the additional parameters and computation being at most a fraction of a thousandth compared to the original model. We have added a table of parameter counts and computational costs in Appendix B. Since there is still space in the table for the 7B experiments, we have added the parameters and computation (FLOPs) to the 7B experiment table. The detailed data is presented as follows.

Table: Comparison of number of parameters

Table: FLOPs per token in forward pass

We sincerely appreciate your thoughtful and constructive feedback. Your insights are invaluable in improving our work, and we remain open to further discussion should you have any additional questions or concerns about our response.

As all reviewers have recognized, our results are highly promising while adding virtually no extra computational cost. We believe our approach will become increasingly practical and significant in the era of large language models. We hope these insights and outcomes will contribute meaningfully to the community. We truly appreciate your time and would be very grateful if you could re-evaluate the paper’s rating.

Supplementary Response to Weakness1: Regarding the pseudocode, actually, we have provided it in Appendix G and H of the original version. In the new version, these pseudocodes can be found in Appendix I and J.

Given the author does address most of my concerns raised in the original review, I will raise my rating to 5 to reflect the new manuscript.

Thank you for taking the time to reevaluate our work. We noticed that your rating remains on the negative side. We are concerned that there may be additional points we have not fully aligned with or other issues that remain unresolved. If so, we would be more than happy to provide further clarification or address these concerns.

Question1: This is a very interesting and profound question. In fact, we conducted a related analysis in Figure 7 and Section 4.5 of the paper. We can study this issue by unfolding the hyper-connections. After unfolding the hyper-connections, we can observe the influence of the hidden state from layer $j$ on layer $i$. In Figure 7, the intensity of the red color represents the magnitude of the influence. In the lower triangular, the uncolored areas represent redundant connections.
Overall, we observe the following:

1. There are fewer redundant connections in the shallow and deep layers, but more in the intermediate layers. Notably, the shallow layers have fewer redundant connections, which prevents the issue of vanishing gradients.
2. The attention layers exhibit a short-term pattern in their influence on subsequent layers, mainly affecting nearby layers. In contrast, the feedforward network (FFN) has a long-term influence on subsequent layers. It’s important to note that the reduction in the short-term pattern indicates a lower risk of collapse.

Additionally, since our initialization is equivalent to Pre-Norm, the connection pattern, as shown in the third diagram of Figure 7, exhibits a fully-connected pattern. This ensures smooth gradient flow in the early stages of training, while dynamically balancing vanishing gradients and representation collapse as training progresses.

Question2: We consider this proposal very interesting, and we have included this part of the visualization analysis in Appendix E. We randomly select three categories from the ImageNet dataset and sample the corresponding examples from the validation set. These samples are fed into the ViT-Base/16-DHC$\times$2 model to compute the dynamic connection weights of the DHC in the final layer. As shown in Fig. 12, we visualize the distribution of these weights. We observe that the intra-class distribution of beta is highly concentrated, indicating that samples within the same category tend to have similar beta values. In contrast, the distribution of alpha is less concentrated, but the differences between the distributions of different categories are more pronounced, as exemplified by $\alpha_{2,0}$.

We greatly appreciate the time and effort you have taken to review our manuscript. In response to your insightful comments, we address each point individually.

Weakness1: Sorry for the confusion. To clarify, the hidden state is duplicated into $n$ copies only once at the beginning of the network input. Then, each layer of our hyper-connections accepts $n$ hidden vector inputs, feeds them into the transformer layer, and applies residual connections with reweighting, controlled by the $\mathcal{HC}$ matrix of $\mathbb{R}^{(n+1) \times (n+1)}$. The process outputs $n$ hidden vectors. This is detailed in Section 2 and Algorithm 1.

We have revised Figure 2 and added an additional architecture diagram in Figure 8 of Appendix A for more intuitive illustration.

Weakness2:

If the goal of creating multiple copies is just to make sure multiple depth connections can be modelled parallelly, is creating such copies actually necessary?

We would like to point out that the hidden vector only needs to be duplicated into $n$ copies when it is first input into the network, and the subsequent $n$ hidden vectors are actually different, as shown in Figure 8.

The subsequent response will explain that these n(>1) hidden vectors are the key to making this method work.

Can't a single copy be used with different residual strengths? The only difference would be in gradient computation , were additional terms for each depth connection would be added to the singular copy.

I guess the approach mentioned by the reviewer refers to our experiment with n=1 in Figure 14. It does not work, and we have included a detailed analysis explaining why this is the case.

We conducted an analysis using the unfolded hyper-connections method, as we did in Section 4.5. We found that when the rate = 1, the unfolded connection graph is fundamentally different from other cases, as shown in Figure 14 in Appendix F.

In Figure 14, compared to HC$\times4$ models, the $\Lambda$-shaped pattern does not appear. Note that HC$\times1$ does not support the $\Lambda$ pattern in its mathematical formulation, in which the connections to previous layers must be either weakened or strengthened simultaneously. Thus, the lack of connections from the early layers to the final layers may lead to gradient vanishing, similar to post-norm style transformers, which results in performance degradation. For models with rate ≥ 2, this issue does not arise, resulting in improved performance.

Can an ablation be conducted between DHC (n=4 and n=1) and SHC (n=4 an n=1) to show the importance and need for additional copies.

A corresponding ablation study has been conducted for the DHC method proposed in our paper. Please refer to Figure 5 and Table 1, where the performance of n=1 degrades at 500 tokens but gradually approaches the baseline as training continues. Please also refer to Figures 14 and 15 in Appendix L of the revised version.

Furthermore, we believe this issue is very critical, and we have been thinking about it as well. In fact, in our subsequent research, we have indeed found that it is possible to achieve gains without using n copies, although the gains are smaller. The core idea here is not to copy n times but to split the hidden state into n parts. We will disclose the related results of this research in our future work.

Weakness3: Thank you for the suggestion; we have carefully revised this figure.

I read the rebuttal and respective updations in the paper, I am mostly satisfied with the rebuttal. My rating remains unchanged.

I have edited my review to correct the wrong notion that the algorithm recursively creates n copies. I thank the authors for answering my concerns and questions. I also understood the argument for the need of multiple copies. I don't raise my score to 10 because, the memory footprint is still on the higher end. I also note the public comment and its claims but I think this paper produces a novel contribution and the statement made on DHC only improving because of width to be incorrect (note my strength no 3). Ultimately, I suggest acceptance of the paper and maintain my score of 8.

Question2: Theoretically, our method is independent of the model architecture and is compatible with CNNs, as it primarily improves residual connections, which are also present in ResNet. Expanding the application of Hyper-Connections to large-scale vision-language models (including CNNs) and text-to-image/video generation tasks is a key direction for our future work. We leave the detailed exploration of these applications to future research.

Question3: Fixed, thanks.

Weakness2: We appreciate the suggestion to train on more tokens as a means of enhancing the impact of this work. And in fact, we are actively coordinating resources to apply the methodology to the current production-level training of large language models (e.g. models like GPT-4o, Claude, Gemini). However, for academic research endeavors, it is challenging to directly train a model from scratch until convergence, as it would demand an immense amount of computational resources. For instance, Meta's training of the LLaMA-3.2 1B model (https://huggingface.co/meta-llama/Llama-3.2-1B) utilized up to 9T tokens and consumed 370,000 H100-GPU hours, equivalent to 128 H100 GPUs for 4 months or 128 A100 GPUs for 1 year.

And it is crucial to note that for the majority of research related to pre-training improvements, the performance gap between methods stabilizes once a certain threshold of training corpus is reached. To further elucidate this point, we provided the pre-training loss curves up to 500B tokens , as shown in Figure 1 and Figure 6. For some of the 1B model experiments, we extended the training trajectory to even 1T tokens (each experiment requires 64 A100 GPUs for 15 days), where the gains from hyper-connections were consistently maintained. If the reviewers are interested in these additional loss curves, we have included them in Appendix L.

Moreover, based on our past experience in training production-level LLMs, we are confident in the effectiveness of this method, even for model scales with hundreds of billions of parameters.

Weakness3: In section 3.2 several non-learnable patterns are presented but are not tried in practice. It is not clear whether learning the hyper-connection patterns is really better that having simple fixed patterns, and an analysis on that would be interesting. response： section 3.2 is SEQUENTIAL-PARALLEL DUALITY We believe this is an excellent suggestion, and we would also like to point out the following:

1. The "sequential configuration" is exactly equivalent to the baseline we are comparing against.

2. As for the "parallel configuration," which is an almost parallel transformer block (PTB), it is commonly used in Google-related models with the primary goal of speeding up inference. This technique is mainly applied to overlap the computation of the FFN (Feed-Forward Network) and memory access in attention, thereby achieving inference acceleration. This is also the reason why the flagship model Gemini 1.5 Pro adopts the "sequential configuration," while only the lightweight model Gemini 1.5 Flash employs this technique. We have tried PTB in LLMs for other projects, and while the training loss can be brought to parity, the reasoning ability significantly deteriorates.

Nevertheless, we believe that the "parallel configuration" is not always inferior to the "sequential configuration" for all problem instances. Therefore, allowing the model to learn and decide which configuration to lean towards based on the input is a reasonable design.

For this work, we have started training the parallel configuration experiments, but the results may not be available until after the rebuttal period. If the paper is accepted, we plan to include them in the camera-ready version.

Question1: We believe this issue is very critical, and we have been thinking about it as well. We have included our analysis of the rate=1 case in Appendix F.

We conducted an analysis using the unfolded hyper-connections method, as we did in Section 4.5. We found that when the rate = 1, the unfolded connection graph is fundamentally different from other cases, as shown in Figure 14 in Appendix F.

In Figure 14, compared to HC$\times4$ models, the $\Lambda$-shaped pattern does not appear. Note that HC$\times1$ does not support the $\Lambda$ pattern in its mathematical formulation, in which the connections to previous layers must be either weakened or strengthened simultaneously. Thus, the lack of connections from the early layers to the final layers may lead to gradient vanishing, similar to post-norm style transformers, which results in performance degradation. For models with rate ≥ 2, this issue does not arise, resulting in improved performance.

Similar experiments and conclusions can be found in Table 1 of https://arxiv.org/pdf/1603.05027: namely, that a shortcut in the residual connection with weights predicted by a network does not perform better than the standard residual connection.

In fact, in our subsequent research, we have found that it is possible to achieve gains without using n copies, although the gains are smaller. The core idea here is not to copy n times but to split the hidden state into n parts. We will disclose the relevant results of this research in our future work.

We greatly appreciate the time and effort you have taken to review our manuscript. In response to your insightful comments, we address each point individually.

Weakness1: We have provided an analysis of the parameter count, computational cost and memory footprint of Hyper-Connections in Appendix B. Since there is still space in the table for the 7B experiments, we have added the parameters and computation (FLOPs) to the 7B experiment table. The detailed data is presented as follows.

Table: Comparison of number of parameters

Table: FLOPs per token in forward pass

The introduction of HC results in a minor increase in activation memory usage during training . This contributes less than 15% , as we analyzed in Appendx . Furthermore , we have developed highly effective engineering optimizations. Since Hyper-Connections introduce very little additional computation, their computational cost is minimal. As a result, during the training phase, memory usage can be reduced through recomputation, while training speed can be maintained by leveraging Triton operators. Based on our current optimizations, we have reached the following conclusions:

• Training phase: With recomputation and Triton operator optimization, when n=2, peak memory increases by 8%, and training speed reaches 90% of the baseline.
• Inference phase: The hidden states generated in each layer can be immediately freed, making the impact on memory usage during inference negligible. We will provide the final engineering solutions and detailed numbers in the open-source repository.

We are greatly appreciative of your insightful and constructive feedback, which has been instrumental in elevating the quality of our work. And we remain open to any further questions or concerns you might have regarding our response.

We strongly believe that our approach holds great potential to become increasingly practical and impactful in the era of large language models. We are optimistic that our findings and contributions will provide meaningful value to the community. As the discussion phase progresses, we would greatly appreciate it if you could share any additional thoughts or re-evaluate the paper’s rating at your earliest convenience.

The rebuttal partially answers my main concern regarding the computational cost of hyper-connexions. Training flops and additional #parameters are not impacted, however memory using is still unclear. Why not showing a Table similar to your two tables for flops and #parameters, but for peak memory usage ? 15% peak memory increase for n=2 is not negligible, what is the increase for n>2 ?

Thank you for the new table on memory usage. I believe the memory usage overhead is reasonable and I understand the argument of activation checkpointing. For these reasons I am increasing my score to 6. Please include the new tables in future versions of the paper.

The complete memory footprint results without HC hidden states checkpointing are as follows:

Table: Comparison of Memory Footprint Without HC hidden states checkpointing on 8 A100 GPUs

We would like to reiterate that:

1. memory usage is significantly reduced by leveraging the HC hidden states checkpointing technique. Specifically, for activations, only the inputs and outputs of each layer are stored, while the expanded hidden states are recomputed during training through the HC module. This approach not only minimizes memory consumption but also maintains computational efficiency.

2. HC is particularly effective for models such as MoE, which combine large parameter counts with relatively small hidden sizes.

Thank you for your suggestion. We conducted experiments on 1B models using 8 A100 GPUs to observe their memory usage.

Table: Comparison of Memory footprint of 1B models

We hope this addresses your concerns. If you have any further questions, please let us know—we would be more than happy to clarify.

We greatly appreciate the time and effort you have taken to review our manuscript. In response to your insightful comments, we address each point individually.

Weaknesses1: It should be noted that our hyper-connections make all connections learnable, as detailed in Section 2. The performance under this setting is compared in Tables 1 and 2 (SHC/DHC). The different lines in Figure 2 are for illustrative purposes to correspond to the captions and do not indicate actual connections. We have revised Figure 2 and added an additional architecture diagram in Figure 8 of Appendix A for more intuitive illustration.

Weaknesses2: We would like to point out that the visual gains are only relatively smaller compared to LLMs, but they are still significant. Table 7 hyper-connections in DiT performance comparable to the 1.5 larger model; Table 8 has significant improvement from 76.38/77.25 to 77.60/79.94.

Regarding this phenomenon, we have some intuitive analysis:

1. The gain from Hyper-Connections in reasoning ability is hard to manifest in these vision tasks. Hyper-Connections, to some extent, unlock the potential of network depth (alleviating representation collapse), and network depth has a significant impact on reasoning ability (https://arxiv.org/abs/2407.20311). This is why we see a very robust gain on tasks like HellaSwag.

2. The relatively small scale of the datasets used in these vision tasks may diminish the effect of Hyper-Connections (HC). For example, datasets like ImageNet are smaller compared to those used for large language models, and the training process spans many epochs—for instance, ViT is trained for 300 epochs, and DiT for 1400 epochs. In contrast, large language models typically pass through the data only once, and we observe that the gain from Hyper-Connections does not diminish as the number of training tokens increases. However, in vision tasks, especially in the later stages of training, we notice that the gain from Hyper-Connections tends to decrease as training continues. This may be due to the fact that these vision tasks involve repeated passes over the same dataset across many epochs, which could lead to diminishing returns from the additional capacity provided by Hyper-Connections. Despite these limitations, Hyper-Connections still exhibit notable performance gains in vision tasks. We believe that the full potential of Hyper-Connections may be realized in large-scale vision-language models or in tasks such as text-to-image/video generation, where larger datasets and more complex reasoning may come into play.

Expanding the application of Hyper-Connections to large-scale vision-language models and text-to-image/video generation tasks is a key direction for our future work. We believe that these areas, with their larger datasets and more complex reasoning requirements, provide an ideal environment to further unlock the potential of Hyper-Connections and explore their full capabilities.

We have included the loss curve for training on ViT in Figure 11, Appendix E. Please refer to it for additional details.

Question1: Thanks for your suggestion. We have included the loss curves in Figure 13, Appendix L.

We sincerely appreciate your thoughtful and constructive feedback, which has been invaluable in improving our work. As the deadline for submitting a revised version of the PDF (November 27, AoE) is approaching, we kindly remind you that if there are any remaining questions or suggestions for improvement, we would be more than happy to address them.

Once again, we deeply appreciate your time and effort, and we would be truly grateful if you could re-evaluate the paper’s rating.

Thank you for recognizing our work and engaging in this constructive dialogue with us.

After carefully reviewing the hyperZZW paper, I would like to clarify the distinctions between our works.

We will now proceed to address each question individually:

Motivation of the Paper: The primary goal of residual connections is to minimize information loss at each layer, as referenced in https://arxiv.org/pdf/2401.17948.pdf. Your proposed framework shares similarities in that it increases the model's width by duplicating the hidden features $n$ times.

We would like to compare Figure 8 in Hyper-Connections with Figure 3 in HyperZZW. We believe that the differences between this work and ours are significant.

1. For Hyper-connections, the approach involves duplicating the hidden vector $n$ times only once at the beginning of the network. At each layer, a linear combination of the $n$ hidden vectors is used as the layer's input, while maintaining the $n$ hidden vectors with information exchanged through width connections. Then, through deep connections, the layer's output is linearly combined with the n hidden vectors, resulting in $n$ hidden vectors. The coefficients for these linear combinations are scalars.

2. For HyperZZW, it appears that each SFNE block processes the input $x$ through multi (9) branches (similar to GoogleNet (https://arxiv.org/pdf/1409.4842)), and then concatenates the outputs of these 9 branches.

There is a fundamental difference between retaining n hidden vectors throughout the network and repeating the input into n parts for each layer. The former can retain diverse combinations of information from earlier layers across different hidden vectors, while the latter merges the information early on.

Hyper-connections effectively transform the input using a small number of training parameters. This aligns with the concept of Hyper Interaction, as introduced on page 6 of the above paper.

We would like to compare Figure 2 (D) in Hyper-Connections with Figure 5 in HyperZZW.

Hyper-connections aim to construct both width connections (across the $n$ input hidden vectors) and depth connections (between the layer output and the processed hidden vectors). The coefficients generated by Hyper-connections involve $n \times (n+1)$ scalars, without any constraints on their value ranges.

In contrast, in the HyperZZW paper, the proposed SFNE block processes the input through 9 branches. One of these branches is Hyper Interaction, which aims to generate a gating mechanism with the same shape as the input ($B \times C \times H \times W$), where the values are in the range $[0, 1]$. This gate then multiplies with the input to extract useful information. This technique is conceptually similar to https://openaccess.thecvf.com/content_ECCV_2018/papers/Sanghyun_Woo_Convolutional_Block_Attention_ECCV_2018_paper.pdf.

Additionally, as proven in Section 3 of our paper, Hyper-connections exhibit the Sequential-Parallel Duality property, meaning they can dynamically adjust the sequential or parallel arrangement of layers. Hyper Interaction does not seem to possess this property.

Conclusion

We thank the reviewers again for their thorough evaluation and constructive suggestions. We believe that this paper makes important contributions to the design of network architectures by introducing Hyper-Connections, a simple, effective, and broadly applicable alternative to residual connections. Through additional experiments and clarifications, we addressed most key concerns, and reviewers broadly recognized the significance of our theoretical insights, empirical results, and practical utility. We are confident that Hyper-Connections will inspire further research and advancements in deep learning architectures.

We would like to emphasize that accusations of plagiarism are extremely serious. If you still believe that we have plagiarized your work, we request that you provide direct and concrete evidence, as we do not find the arguments you have presented so far to be convincing. Furthermore, we urge you to consider the following: our methodology and the problem we address differ significantly from yours. If our work had indeed been inspired by yours, citing it would not diminish the impact of our research in any way. Given this, what possible motivation would we have for omitting a citation to your work? There is no logical reason for such an omission.

We believe that this public comment is not made in the spirit of constructive discussion. Without providing substantial evidence, the commenter maliciously speculates about and defames our work, which is deeply frustrating. Furthermore, the intent to solicit citation is quite apparent.

Our team has carefully reviewed the paper mentioned by Harvie ZHANG and acknowledges that, in terms of relevance (exploring alternatives to residual connections), it could be cited but is by no means essential. The scope of our research, the underlying network architecture, and our methodological approach differ significantly from this study.

Given the lack of sufficient experimental validation, proper peer review for the paper, and the substantial differences in our research tasks, we are inclined not to cite it.

Moreover, we would like to emphasize that the most effective way to enhance the impact of one's research is by refining and strengthening the work itself, rather than pressuring others into citing it.

We will not respond to this baseless comment unless specific evidence is provided to support the accusation.

We appreciate your interest in our work and wish you success in your future research endeavors.