I have multiple concerns in relationship to this work. In particular, the modules here introduced, namely Layer-Wise Mapping, Rectify SideNet and the SWEAP operator, seem to have been introduced by different previous papers under different names.

• First, the paper claims throughout the paper that “current long-range convolutions have problems with excessive parameter usage”. However, this is not the case for most long convolutional models. To give an example, Hyenas [1] rely on the parameterization of convolutional kernels of CKConv [2], which uses implicit neural representations to parameterize the kernel. This --which to the best of my understanding is at least analogous to the LayerWise mapping proposed here (see below for more discussions on this)--, also produces a kernel parameterization which is sublinear wrt the length of the sequence considered. Note that several other existing parameterizations of long conv models also exhibit the same behavior, e.g., S4 [3], H3 [4], S5 [5], etc.

  In fact, the layer-wise mapping proposed in this paper seems to be a less powerful replication of what has been done in CKConv [2] and Hyenas [1]. Specifically, both of these use an MLP to map coordinates to kernel values, which –just as here– allows for the generation of long conv kernels that can be made dynamic based on the input length. With less powerful I refer to the conclusion of the study in CKConv regarding high-frequencies. Note that both CKConv and Hyenas rely on “implicit neural representations” for the parameterization of the kernel. CKConv depicts that using normal MLPs leads to very bad parameterizations unable to depict high-frequencies.

• Secondly, the rectify sideNet is very similar to what is done in the Hyena operator of level 1. Specifically, the rectify sideNet can be understood as a combination of two input dependent mappings and a long conv. Given that typically level-2 Hyena’s are used, I do not see how this could be more powerful than Hyenas and, to the best of my understanding, the rectify sideNet seems to be a more constrained construction.

• Finally, the paper proposes to use a learnable exponentially decreasing mapping to mask on the generated kernel. However, the idea of using learnable masks to restrict the receptive field of the kernel has been proposed in FlexConv [6] with Gaussian masks, and exactly explored in Hyena and several other SSM papers in a learnable, exponentially decreasing form.

Altogether, I feel that the method proposed in this paper is a restricted form of Hyenas with multiple heads that have different nonlinearities. However, all these different relationships have not been properly described in the paper. I would like to note that it is not per se a problem that methods are related. However, if this is the case, I consider it important to discuss how the proposed method is different and why these simplifications make sense.

• Given the previous observations, I do not understand the reasoning behind the “partial dependency” and “fully dependency” in Table 1. Based on the discussion in the previous points, the only difference I see between these two classes is the use of a different nonlinearity in the gating operation.

• Experimental results. In addition, I am not sure that the experimental experiments support the claim of the paper in the abstract. Specifically, the paper states that: “Extensive experiments have demonstrated that DCL-Net surpasses the state-of-the-art in processing lengthy sequences, even when trained in short sequences.” However:

  First, in Table 2, this is not the case.

  Next, in Table 3, one would say that Hyenas and DLCNets are roughly on pair.

  On Table 4, this claim is also not supported.

  Finally, on Table 5, this is not compared to Hyena’s, which –based on my previous observations about the method– should be able to do this as well, I feel this comparison is incomplete.

• Since the use of long range convolution is rather recent, the authors should pay extra efforts to introduce this line of work in detail, especially in the related works section. Also, a method-wise comparison against prior works such as Hyena and TNN should be provided to help the readers understand and make fair comparisons.
• Performance-wise, the results on downstream NLP tasks show that DLCNet is mostly second to other baselines such as RWKV-4, and in the cases where DLCNet outwins the margin is comparably small than in the opposite case.

I found this paper challenging to read and understand the proposed methods. The text through page 4 does little to explain the three mechanisms that have been proposed and I had to re-read a number of times to understand how the three proposed changes align with the descriptions of existing long-range convolutional architectures. One portion of the method that I am still unclear about is the decay used in the “layer-wise mapping”. It’s not clear to me if this is from existing long-range convolutional architectures or if this is a new idea the authors are introducing.

I also found the experimental results to be not very compelling. First, I think the authors should include the number of math operations that are done by each model and not just their parameter counts (e.g., in Table 2). Convolutional architecture often reuse parameters, which can lead to having higher quality per parameter but not necessarily per FLOP. This seems to be potentially true for the lower-parameter bottleneck in the proposed method as well. I’m particularly interested in these numbers for comparison with Hyena, which is quite close to the proposed architecture in quality and parameter count across pre-training and downstream tasks.