Stochastic Adversarial Networks for Multi-Domain Text Classification

1. The motivation is trivial. It is hard to say that the model size is the bottleneck of the training process according to Table.1 and 9. 342.91M is absolutely fine in current period. Further, inference process may gain nothing in the aspect of computational acceleration as we only choose one private extractor from the Domain Discriminator D.
2. The baselines are outdated and improvements on two benchmarks are limited. According to Table 2,3 and 4, it can hardly convince me that the proposed model exactly outperforms the SOTA models. It is worth noting that the author points out this limitation in Appendix E.
3. The writing and organization need to be improved. a) The emphasis in writing has been misplaced. As the author highlights the role of multivariate Gaussian distribution in Abstract, you are supposed to tell more story of it instead of the regularization term, which is the idea of others. b) The effectiveness is not the focus of this article, efficiency is. Therefore, moving D. 5 to the main body of the article perhaps make your contribution more prominent. c) Some tools can be utilized effectively to optimize sentence structure and composition.

I have some concerns about this work.

1. Assuming the design of proposed model is sensible (in fact I have doubts on this; see 2), the work heuristically puts together a bunch of well-known techniques to improve performance. Works of primarily such a nature, although potentially valuable in practice, do not possess enough novelty that justifies a publication in ICLR.

2. I have doubts on the proposed approach in the "stochastic" part. Let us track the parameter $W_1$ of the domain-specific feature extractor for domain 1. In the beginning it is drawn from the prescribed Gaussian, say, its value is $W_1^{(0)}$, and after the first iteration, the Gaussian parameter gets updated (using the reparametrization trick) -- well, whether Gaussian parameter is updated or not is not critical here. Then in the next iteration, $W_1$ is drawn again, let us call it $W_1^{(1)}$. If this understanding is correct, then $W_1^{(0)}$ and $W_1^{(1)}$ can be very different. That is, along the training process, $W_1$ will randomly hop everywhere as long as the Gaussian variance is not vanishing. How would such a scheme work at all? Bringing the parameter $W_2$ of the second domain-specific extractor into the picture would show an even more absurd picture: at each iteration $t$, $W_1^{(t)}$ and $W_2^{(t)}$ are random variables following the same Gaussian distribution. How would $W_1$ and $W_2$ track their respective domain specific features? If this structure were to work, it would have to be the case where the Gaussian variance is very small (which might be the case as shown in Figure 3 of the appendix). In that case, all domain-specific extractors are more or less the same, i.e, all equal to the Gaussian mean, only subject to some tiny domain-nonspecific random perturbation. That would defeat the entire purpose of having domain specific feature extractors. -- I could misunderstood the paper and I am willing to hear the authors' defence on this. In your defence, please also show the initial and final values of the Gaussian mean vector $\mu$ (say, in terms of its L1-norm divided by its dimension), I would like compare it with $\sigma$.

1. Though the proposal seems to be effective and achieving strong performance, the model itself still uses a relative old adversarial backbone, with the discriminator approach for removing the domain invariant feature. The two-feature-extractor approach is interesting, but that is mainly to deal with parameter increase in the MDTC problem. It would be great to see other design improvement in the model.
2. The performance gain in using the proposed model is marginal on the Amazon review/FDU-MTL datasets. Also, it would be great to have some analysis on adjusting the setting between the two feature extractors.