Efficient Heterogeneous Meta-Learning via Channel Shuffling Modulation

1. We provide below the results of varying the $\lambda$ parameter in the MRN-GB/MRN-GS objective function. We used the 1-shot 5-way Omniglot dataset and 200 training epochs for each experiment (Note that in the full experiment in our manuscript we used 2000 training epochs, hence the accuracy values reported below are somewhat lower).

The above result seems to suggest that the parameter $\lambda$ cannot be too large, causing the loss function to behave similarly to the ProtoNet loss; or too small, causing the task embedding module to capture unmeaningful patterns.

2. In all of our experiments, we used a base architecture with 4 convolution blocks. For our methods, we added one routing layer after every block, hence the routers are distributed at various depths of the network. The influence of the routers at each depth is an interesting experiment to conduct. We have provided some preliminary results below (also on the 5-way 1-shot Omniglot dataset), in which we compared the performance of our methods using all 4 routers vs. the performance of using only one router at different depths (e.g., larger depth means being closer to the output layer).

Our result suggests that having the routing layer placed earlier on in the architecture is the most effective practice (e.g., putting a routing layer after the second convolution block gives the best performance). However, combining many routing layers might improve performance, such as in the case of MRN-GB. We will investigate this behavior further in future work.

3. Parameter count and training efficiency comparison have been provided in Table 3 of our manuscript.

4. The ProtoNet loss ($\mathcal{L}_{proto}$) is computed for each class in a meta task. The embedding vector phi is the average of the ProtoNet embeddings for all classes in the task. This is a well-known strategy to compute task embedding. Thank you for pointing it out, we will add an algorithm box to clarify this in our revision.

1. We will provide an overview of related concepts in the revision of our paper.

2. Thank you for the interesting suggestion regarding the use of sorting networks. It is true that other sorting networks can be used in place of the Benes routing network for our strategy. This approach would be very similar in spirit to our method. Whereas our Gumbel Benes layer is conditioned on the task embedding vector, a sorting layer would theoretically derive contextual information from learning a value function that scores the convolution channels. Furthermore, a bitonic sorting network will also require $C\log^2 C$ comparators, so we would expect this mechanism to have, at best, similar complexity to our method.

1. Regarding significance, even though the margin of improvement is not as large as you would prefer, we would like to highlight that our contribution also includes a parameter efficient meta learning formulation, as demonstrated in Table 3.

2. Following your suggestion, we have conducted follow up experiments on the more difficult 1-shot, 5-way Mini-ImageNet setting and its Jigsaw counterpart. We adopt similar training settings to the 5-shot, 5-way experiments in the main manuscript. That is, we trained every baseline for 2000 epochs, sampling 32 training tasks of 16 images per epoch (1-shot, 15-queries). For testing, we sampled 5 test tasks with 1 training and 15 test images per task. Due to the limited rebuttal time, we only focus on comparing with other heterogeneous Meta learning baselines (i.e., MMAML, HSMAML, URL). We have also provided the result using ResNet-18 on these scenarios to answer your other question. The results of these experiments are detailed in the Table below:

3. We have compared our method to a lot of benchmarks in the manuscript, one of which (i.e. the URL method) was published very recently in 2022. Due to the limited rebuttal time, we may not be able to provide results to the suggested methods. In particular, we couldn’t find an implementation of reference [1]. On the other hand, FEAT (reference [2]) adopts a Transformer architecture that does not immediately fit with our permutation framework, thus it would require significant changes to provide an apples-to-apples comparison. That being said, we will try our best to provide such results in our future revision.

4. Thank you for your suggestion. We will provide a discussion of other DL works that use the Benes Network in our revision. We however hope you agree that our work is the first that uses this concept in a meta-learning context.

Dear reviewer,

We have re-run our method for 12000 training epochs. The table below presents our results side by side with the results quoted from Table 1 of Ref [1].

As you can see, given similar training budget our method is capable of achieving much better performance than the quoted results. We hope this would give you sufficient evidence to support our method's potential for now. Given time, we will thoroughly compare with these previous methods on the exact same setting (train/test split, learning rate, layer size, etc.)

First of all, thank you for clarifying your points. We do understand your concerns better now, and appreciate your constructive feedback. To add some discussion:

We acknowledge that ConvNet layers with more than 64 channels are not as practical in practice, and there are better architectures to try our method on. Some, such as ResNet, are immediately compatible as shown in the rebuttal experiments and achieve reasonably good performance. We will heed your advice and experiment with Transformer/ViT architectures in the future, possibly by permuting the multi-head structure of transformer. We would also like to highlight that the heterogeneous meta-learning baselines that adopt the weight modulation strategy will become more impractical with increasing model size, regardless of the architecture type. In our manuscript we chose to increase the number of channels to achieve this effect. Although it was not meant to favor our method, we do understand how confusion can arise.

Regarding the rerun experiments on 12k epochs, the backbone that we used is the CNN architecture with 4 conv blocks and 4 routing layers as described in the paper. Each conv block has a 64 channel conv layer, followed by BatchNorm, ReLU and MaxPool, which is exactly similar to [1]. What we meant in our previous response was that we couldn't find their implementation, so we cannot reproduce exactly their learning rate, train/test split, weight initialization (our apology, we meant to say layer weight, not layer size). Reference [1] also did not describe how many test images are there in one test task, and how many query images are there in one training task. In our experiments, each task consists of 1-shot/15-query or 1-shot/15-test. Due to limited time, we also did not go into each method listed in Table 1 of reference [1] to check their experimental setting to make sure our results are as fair as possible.

Finally, we will try our best to provide the accuracy of our methods on the Omniglot dataset with 12k epochs in the remaining time of the rebuttal.

1. Thank you for your suggestion; we will provide a discussion of the suggested related works in our revision.

2. We strongly agree that the intuition of modulation assumes a strong shared backbone, and each dataset/distribution performs task-specific adjustments for its data. However, the degree of adjustments will depend on the heterogeneity of the task distribution. Additive modulations might suffice for task distributions consisting of closely related tasks. On the other hand, more challenging distribution of tasks tend to require more significant modifications, as explored in previous studies of heterogeneous meta learning via modulation. Our hypothesis is that structural modulations, such as a shuffling of the embedded information channels, would be more effective at modulating between tasks of different modalities. In fact, this is not an arbitrary design choice as it is also supported by previous empirical evidence. For example, random shuffling has been used in the context of compact neural architecture due to its efficiency in encoding information.

3. It would be possible to visualize the learned permutations. However, we would like to note that the identity of the permutations themselves are not quite important. We believe that it is more critical to show that the network can learn to assign similar tasks to similar permutations, and different tasks to different permutations, and thus improve knowledge organization. To show this, we have provided the t-SNE plots of the learned Benes switch configurations for 1000 tasks in two different scenarios (see our Appendix A), which show distinct clusters that correspond to the task identities.