Text2Data: Low-Resource Data Generation with Textual Control

Comment 1: There is no comparison to other techniques like data augmentation or transfer learning for low-resource text-to-data generation. Comparisons would be useful to know if Text2Data outperforms them.

Response 1:

• Data Augmentation: we employed GPT-4 to modify the textual descriptions to augment the text-data pairs (EDM-DA), and we directly performed classifier-free guidance diffusion on molecular generation using EDM. Below are the results that evaluate the controllability of EDM on properties of generated molecules (Alpha and HOMO).

Based on the results above, data augmentation-based approach slightly improves performance of directly finetuning the model. This is potentially due to the improvement of generalization of text descriptions.

• We have already performed transfer learning based approach in our paper. For instance, EDM-Finetune represents the model that EDM is pretrained on a large amount of molecular structures, and then finetuned on a small amount of text-molecule pairs. Based on the results above, EDM-Finetune has much worse performance compared with the proposed approach. This is expected due to the potential catastrophic forgetting of transfer learning-based approach.

Comment 2: The threshold $\xi$ for the constrained optimization is set heuristically based on the minimum value of unsupervised loss L1.

Response 2:

• The threshold does not have to be set strictly based on the minimum value of unsupervised loss L1, but can be further relaxed as long as it lies in the confidence bound of the minimum value of unsupervised loss L1.

• To guide the relaxation of the threshold, we have derived a generalization bound as shown in Theorem 1. In Theorem 1, we have proved that the threshold can be further relaxed as long as it lies in the summation of empirical minimum value of unsupervised loss L1 and the confidence bound $\epsilon=\epsilon_N+\epsilon_{N_p}$. In practice, we multiply a hyperparameter before the threshold to relax it. We have highlighted relevant part in red in the main paper.

Comment 3: Catastrophic forgetting is identified as an issue, but not experimentally verified by comparing with unconstrained finetuning. This would further prove the need for constrained optimization.

Response 3:

We have compared with unconstrained finetuning. For instance, as shown in the table below, EDM-Finetune represents the model that EDM is pretrained on a large amount of molecular structures, and then finetuned on a small amount of text-molecule pairs without applying any constraints. Based on the results above, EDM-Finetune has much worse performance compared with the proposed approach. This is expected due to the potential catastrophic forgetting issue.

Comment 1: As for the method, the proposed method is relevant to the classifier-free guidance, which both involves a conditional model and an unconditional model (and the two models share the parameters). The difference is that the proposed method: finetune the unconditional model while adding constraint for catastrophic forgetting and the classifier-free guidance: guidance the denoising direction (in diffusion model) of conditional model with the help of unconditional model. I think the discussion of why the proposed method is better than the classifier-free guidance is necessary.

Response 1: We target the situation while labeled data is very limited. Directly learning a conditional model on such limited data may not have satisfactory generalizability and suffer from the limited diversity of generated data. Additionally, as there are too many parameters in the diffusion model (e.g., 14 million for Motion Diffusion Model), the lack of labeled data may cause issues such as overfitting. By contrast, our model is first pre-trained on unlabeled data, which usually ensures enough sample size to avoid overfitting.

Comment 2: As for the related work, the paper should discuss the related work about transfer learning, where catastrophic forgetting is an important topic. Since one of the contributions of the paper is preventing catastrophic forgetting. I think the paper should compare the proposed method with other works about catastrophic forgetting.

Response 2:

• We have discussed representative works of transfer learning and its potential drawbacks in causing catastrophic forgetting. We have highlighted this part in red for easier reference:

“Tu et al. (2019) proposes to learn a mapping between source and target linguistic symbols and employs transfer learning to transfer knowledge from a high-resource language to low-resource language.”

“Nevertheless, all those strategies have their own limitations, such as high computational complexity for data augmentation, difficulty in maintaining the correct interpretation of text when leveraging unlabeled data during semi-supervised learning, and the potential catastrophic forgetting issues in transfer learning.”

• We have also conducted additional experiments for data augmentation, another approach to overcome catastrophic forgetting.

Specifically, we employed GPT-4 to modify the textual descriptions to augment the text-data pairs (EDM-DA), and we directly performed classifier-free guidance diffusion on molecular generation using EDM. Below are the results that evaluate the controllability of EDM on properties of generated molecules (Alpha and HOMO).

Based on the results above, data augmentation-based approach slightly improves performance of directly finetuning the model. This is potentially due to the improvement of generalization of text descriptions.

Comment 3: The proposed method should be verified on popular tasks such as image generation. Currently the 3 task in paper is not quite convincing.

Response 3: We did not apply our approach to domains such as image generation because those domains usually have huge labeled dataset (e.g., ImageNet, CelebFaces, COCO) that can be leveraged to directly train the diffusion model conditional on labels. Instead, we focus on modalities such as time series, molecules or motions that lack textual descriptions due to costly annotations or intricate data structures.

Comment 1: What’s the estimated manual data collecting effort and procedure? How do the authors design a system to do simulation? If data synthesis is based on an existing generative model, to train such a high-quality generative model already requires a lot of data.

Response 1:

• Manual collection can be the existing unlabeled datasets such as QM9, ZINC, etc. Those datasets usually contains a huge amount of unlabeled samples to pretrain our model. For instance, QM9 has 134k stable organic molecules, and ZINC has 250k commercially available drug-like chemical compounds.

• Simulation can be, for example, simulating time series data from a statistical model, such as mixture autoregressive model [1], autoregressive moving-average model [2] etc. From those statistical models we can simulate as many as samples of time series, but they are still unlabeled.

[1] Kang et al., GeneRAting TIme Series with diverse and controllable characteristics.

[2] Benjamin et al., Generalized autoregressive moving average models.

Comment 2: The authors claim they use a cross-attention layer, but some implementation details are missing.

Response 2:

We use the cross-attention to enforce the alignment between data and text descriptions. Specifically, we maximize the cosine-similarity between text embeddings obtained from the LLMs encoder and data embeddings diffused from original data. We have added the explanation to Appendix C.1 and the relevant part has been highlighted in red.

Comment 3: Pre-training is done on the full training dataset, that involves both unlabeled and labeled data. Labeled data is treated as unlabeled without using their labels in the pre-training process. Finetuning relies on only those small proportion of labeled data in the training set.

Response 3: Pre-training is done on the full training dataset, that involves both unlabeled and labeled data. Labeled data is treated as unlabeled without using their labels in the pre-training process. Finetuning relies on only those small proportion of labeled data in the training set.

Comment 4: The authors should have a baseline that trained on the 100% of the labeled data. -- Please include proportions that are larger than 40 as reference.

Respone 4: We have added experiments for up to 100% of labeled data (e.g., 60%, 80%, 100%), the results of which are as below. The experiments are performed on QM9 dataset by leveraging EDM baseline.

Based on the results in the table above, both finetuned method without constraint (i.g., EDM-Finetune) and the direct conditional model (EDM) converge when the proportion goes up to 100%.

Comment 5: What if the model is not in-domain pre-trained? For example, a generative model really pre-trained on large-scale data, but does not use the downstream dataset.

Response 5: We have added experiments for our model to pre-train using just unlabeled data and finetune on labeled data (i.e., EDM-In).

The experiments are performed on molecular generation under the proportion of labeled data from 2% up to 10%. The results show that the performance of EDM-In is slightly worse than Text2Data, but much better than EDM-Finetune and EDN. This is expected since the only drawback of using only unlabeled data in pretraining is the reduction of sample size to learn the general data distribution. But from 2% to 10%, the loss of sample size is pretty small, which does not harm the performance too much.

Perhaps I am missing something, but if you minimise only $\hat{L}_2$ in equation (6), won't it automatically minimise $\hat{L}_1'$ In particular, if p(x|c) is very high for a particular x, doesn't it automatically imply that p(x) is automatically going to be very high.

We don't need that extra term for $\hat{L}_1'$ at all unless we compute it for unlabelled data for which we don't know $c$. Am I wrong here?

This is why in semi-supervised learning, the marginal-loss term is always over the unlabelled data

Comment 1: The approach of optimising marginal likelihood on unlabelled data and conditional likelihood on labelled data is very common in semi-supervised learning literature. The primary difference is that here the authors have also included the marginal likelihood on unlabelled data as a constraint. This would have been interesting if the authors had discussed how that constraint is used during training batch-by-batch. Unfortunately the algorithm for training is missing from the paper.

Response 1:

• We train the model following a pretrain-finetune manner. We firstly pretrain the model on unlabeled data to capture the general data distribution $p_\theta(x)$, as usually people have enough unlabeled data. Then we finetune our model on (usually small amount of) labeled data while, based on Eq. (2), ensuring the finetuned distribution not far from the one learned during the pretrain process. This is not strictly semi-supervised learning as we do not infer labels of unlabeled data, but just leverage unlabeled data to learn a good distribution to initiate finetuning process.

• We have added an algorithm of the training process in Appendix C.3 of revised paper.

Comment 2: There are lots of other details missing from the paper as well. For instance, how is \epsilon_theta modelled? How can it be a function of x at one point and x and c at the other point. Perhaps the authors have assumed that a lot of stuff to be obvious.

We just borrowed the score function of the traditional classifier-free diffusion guidance [1]. We have added the detailed derivation of the diffusion score in Appendix C.2:

“For time series generation, we use the same score function as in [2] that consists of conditional 1-dim dilated ConvNets with residual connections adapted from the WaveNet [3] and Diffwave [4]. For molecule generation, we employ the same score function from EDM [5] which is a Multilayer Perceptron. For motion generation, we also leverage the same score function in [6] as a straightforward transformer.”

We also highlighted this part in red for easier reference of the reviewer.

[1] Ho and Salimans, Classifier-free Diffusion Guidance.

[2] Rasul et al., Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting.

[3] Oord et al., Wavenet: A generative model for raw audio.

[4] Kong et al., Diffwave: A versatile diffusion model for audio synthesis.

[5] Hoogeboom et al., Equivariant diffusion for molecule generation in 3d.

[6] Tevet et al., Human motion diffusion model.

Comment 3: There are no novel insights from the paper. It is not clear why a constrained optimisation objective is superior over other forms of semi-supervised learning. This weakens the paper a lot.

Response 3:

• We have explained the drawbacks of semi-supervised learning approaches as follows:

“For semi-supervised learning, text inherently carries nuances, ambiguities, and multiple meanings. Ensuring that the model maintains the correct interpretation when leveraging unlabeled data is not straightforward.”

We have highlighted this part in our paper for easier reference.

• Our approach is not a strictly semi-supervised learning-based approach as our approach leverages unlabeled data just to learn a general distribution of data, without inferring any of its labels. Therefore, this will not introduce any text-data ambiguity during the learning process. Then during finetuning, we uses (usually) a small amount of labeled data to finetune the model conditional on their text descriptions. This process will not cause any ambiguities either as we are using existing textual descriptions, not inferring their textual semantics.