Tracing Hyperparameter Dependencies for Model Parsing via Learnable Graph Pooling Network

We thank the reviewer for constructive feedbacks. Reviews are positive about our empirical performance and the learnable pooling algorithm, and we address all concerns raised as follows:

Brief Recap:

We adhere to the problem statement of model parsing [R3], in which the algorithm takes one image as the input and predicts 37 hyperparameters used in the generative model (GM) that generates the input image. These 37 predictable hyperparameters are clearly defined in [R3] and detailed in our supplementary, including discrete architecture hyperparameters, continuous architecture hyperparameters, and loss functions.

Q1: The difference between RED116 dataset and RED140 dataset & Ground truth vector:

RED116 is a subset of RED140, with RED140 serving as a more advanced model parsing dataset. The key distinctions between RED116 and RED140 are as follows:

1. RED116 does not have diffusion model images, whereas RED140 includes images from 24 distinct diffusion models (details in supplementary Sec.D and E). Consequently, RED140 provides a richer variety of samples, enhancing the dataset's utility in developing algorithms capable of identifying hyperparameter patterns in images generated by diffusion models.

2. RED140 has real images covering different semantics (supplementary Sec.D details). These real images are crucial for training models to only predict hyperparameters for generated images, rather than real ones. Without these real images, models might erroneously assign hyperparameters to real images when used as inputs

Moreover, each GM has an annotated ground truth vector for discrete architecture hyperparameters. As defined in [R3], these vectors can supervise the training. FEN-PN and our LGPN use the same train and test protocol.

Q2: The comparison on RED116 & comparison on face and non-face GM:

Thank you very much for the suggestion. Please refer to the general rebuttal response, in which we use Table R.1, R.2, R.3, and R.4 to report various performances on RED116. Also, the main paper's Table 1 shows the performance of RED140. These five tables show that LGPN surpasses [R3] on all metrics (i.e., discrete architecture hyperparameters, continuous architecture hyperparameters, and loss functions) on both RED116 and RED140 datasets. Such performance superiority indicates our proposed method's better generalization ability and robustness.

Q3: Image attribution performance:

The main paper's Fig.6 (b) shows a relatively easy protocol in which different models have similar saturated performances. Therefore, we extend this image attribution experiment into Table R.9 with a more challenging protocol, showcasing our method's performance superiority to FEN-PN[R3]:

• Protocol 1: as Fig.6 (b) reports (detailed in Supplementary Fig.8), the image attribution is a 5-way classification task --- classifying if the image is real or generated by which one from 4 GMs (e.g., SNGAN, MMDGAN, CRAMERGAN, and ProGAN).

• Protocol 2: On top of protocol 1, we add 2 more generative methods, resulting in a more challenging task: a 7-way classification task, classifying whether the image is real samples or generated by which one from 6 GMs (e.g., styleGANv2, styleGANv3, SNGAN, MMDGAN, CRAMERGAN, and ProGAN).

Table R.9 Image attribution performance.

Q4: Hyperparameter dependencies:

Sorry for the inconvenience. One common dependency is that GMs with more layers usually have more parameters (lines 30-32). This is illustrated by lines 270-274 and Tab. 2(b):

• Lines 270-274 detail that, with capturing the dependency between Param. Num. and Layer Num., the GTC with GCN refinement block achieves a better performance than GTC with MLP layers, on predicting these two continuous hyperparameters.

• Tab. 2(b) shows that the GCN refinement block helps GTC achieve better prediction performance than GTC with MLP layers on all continuous architecture hyperparameters. Such performance improvements are attributed to the proposed GCN refinement block, which effectively captures dependencies among continuous hyperparameters, such as FC layer numbers, model parameter numbers, pooling layer numbers, etc.

Q5: Graph construction:

Sorry for the confusion. We only use training samples from RED140 dataset to construct such a graph and strictly forbid data leakage. The procedure of using training samples for graph construction is commonly adopted in prior works [10, 7, 59, 39, 15, 50]. In the revised version, we will add the statement --- "use training samples from the RED140 dataset."

Q6: Sensitivity analysis:

Thanks for this valid concern. We follow your suggestions and show the model parsing performance with different thresholds in Table R.10. Overall, the best performance is achieved when the threshold is 0.45, as we employ in LGPN. Also, this best performer is comparable to models that use 0.35 and 0.55 as threshold values, showing LGPN's robustness to different thresholds. However, the performance decreases more when the threshold increases to 0.65, which causes the correlation graph to have very sparse connectivities, hindering the learning of hyperparameter dependencies.

The second robustness analysis is Table 7 in the supplementary, which shows that our model parsing performance is generally robust to using different numbers of graph nodes to represent each continuous hyperparameter.

Table R.10 The model performance on different graph thresholds.

Thank the reviewer for recognizing our interesting research topic and easy-to-implement algorithm. In this response, we present clarification of the coordinated attack detection (Q1) and additional empirical results (Q2) that will be added in the revised version.

Q1: Coordinated attack detection clarification:

We follow the procedure defined in the previous work [R3] for the coordinated attacks detection task, which calculates the cosine distance between predicted hyperparameter vectors from two input images (i.e., ${\mathbf{I}}_1$ and ${\mathbf{I}}_2$). If this distance is larger than a certain threshold, we decide two images are from different generative models (GMs); otherwise, they are from the same GM.

More formally, as stated in Sec.3.1 of the main paper, given an image ${\mathbf{I}}$ as the input, predicted outputs from the LGPN are three vectors, e.g., ${\mathbf{y}}^d\in\mathbb{R}^{18}$, ${\mathbf{y}}^c\in\mathbb{R}^{9}$ and ${\mathbf{y}}^l\in\mathbb{R}^{10}$, which represent the predicted discrete architecture hyperparameters, continuous architecture hyperparameters, and loss function, respectively. Then, the final hyperparameter vector (i.e., descriptor) can be a vector ${\mathbf{y}}^t\in\mathbb{R}^{37}$ via concatenating ${\mathbf{y}}^d$, ${\mathbf{y}}^c$, and ${\mathbf{y}}^l$.

Based on this, given two images (i.e., ${\mathbf{I}}_1$ and ${\mathbf{I}}_2$), we use LGPN to retrieve ${\mathbf{y}}^{t1}\in\mathbb{R}^{37}$ and ${\mathbf{y}}^{t2}\in\mathbb{R}^{37}$, respectively. Then, we compute the cosine distance between ${\mathbf{y}}^{t1}\in\mathbb{R}^{37}$ and ${\mathbf{y}}^{t2}\in\mathbb{R}^{37}$. If this distance is larger than a certain threshold, we decide $\mathbf{I}_1$ and $\mathbf{I}_2$ are generated from different generative models (GMs); otherwise, they are from the same GM. This procedure is used by both FEN-PN [R3] and LGPN, and the revised version will include more details.

Q2: Additional model parsing evidence:

We agree that more experimental evidence would solidify our contributions. Due to the paper length, we keep a large number of experimental results in the supplementary. Please refer to our supplementary, which already offers additional quantitative results and visualizations as follows:

• Supplementary Figure 7: Prediction performance for each hyperparameter.
• Supplementary Figure 12: Visualizations of learned correlation matrix similarity.
• Supplementary Table 9: The detailed performance of RED116.
• Supplementary Figure 10: Visualization of model parsing performance on different semantics.
• Supplementary Figure 12: Model parsing performance for different generative models.
• Supplementary Section D: details of the RED140 dataset.
• Supplementary Section E: details of the annotated ground truth.

Moreover, we will add our rebuttal responses in the revised version, which includes:

• The performance on RED116 using different metrics. (Table R.1, R.2, R.3)
• The detailed performance of face and non-face images. (Table R.4)
• The performance when the model takes a different number of images as the input. (Table R.5)
• The performance using GCN refinement with different backbones. (Table R.7)
• The performance with mistaken labeling. (Table R.8)
• The updated image attribution performance. (Table R.9)
• The sensitivity of using different thresholds for the correlation graph. (Table R.10)

We thank the reviewer for acknowledging our practical and novel research topic and the effectiveness of the proposed LGPN. In this response, we offer additional experimental results regarding GTC's effectiveness (Q1), a discussion of mistaken labeling (Q2), and clarification on the hyperparameter hierarchy assignment (Q3).

Q1: Using GCN refinement as model parsing (MP) head:

Thanks! Table.R.7 provides additional experimental results to demonstrate the effectiveness of the proposed GTC.

Table R.7 Performance of different backbones with the GCN refinement block as MP head.

Q2: will the performance drop when some generated images are mistakenly treated as real images in the training set:

This is an interesting advice! We take this mistaken labeling idea and intentionally treat different percentages of generated images as real ones in the training set. We report the performance in Table R.8. Specifically, the model performance is still acceptable when 1% or 5% percent of samples are mistakenly labeled. However, as expected, the performance largely decreases when the LGPN experiences 8% mistaken labeling. This is reasonable as the proposed LGPN identifies the generation trace as the foundation for predicting hyperparameters. Consequently, this mistaken labeling disturbs the learning of the generation trace, largely reducing the overall prediction performance.

Table R.8 Performance metrics for different percentages of mistaken labeling.

Q3: Clarification on hyperparameter hierarchy assignment:

Sorry for the confusion. The hyperparameter hierarchy assignment (the main paper's Figure 4) is a fixed constraint, which uses general knowledge to encourage the model to pool graph nodes that represent hyperparameters with the same attribute. For example, graph nodes for pixel-wise loss functions (e.g., L1 and L2) are merged, and graph nodes for normalization (e.g., Batchnorm and Layernorm) are merged.

When hyperparameter distributions differ largely between training and test samples, our LGPN leverages the GCN refinement block and the powerful GTC as the backbone to achieve a robust and generalizable model parsing ability. This is demonstrated by Table 1 of the main paper, in which we evaluate LGPN on unseen GMs and it shows SoTA model parsing performance. In addition, please refer to Table R.6 from the response to Reviewer nGEg, in which more GMs are used for the evaluation.

Q4: Experiment section clarification:

The supplementary material offers details about the experimental setup and implementations. In the revised version, we will also polish the expression in the experiment section.

We appreciate the reviewer's praise for our method's effectiveness and reasonable algorithm design. Here, we present answers to all concerns raised.

Q1: Difference between ours and two previous works:

We highlight three key differences between our approach and those in references [R1, R2], demonstrating that our proposed model parsing algorithm is more practical and better suited for general image forensic purposes.

Three differences to [R1]:

• [R1]'s algorithm is only evaluated on the MNIST dataset, while our evaluation dataset contains a wider variety of semantics, including hand-written digitals, faces, and generic objects.

• [R1] only focuses on predicting architecture hyperparameters, yet we are capable of predicting both loss functions and architecture hyperparameters.

• [R1]'s metamodel requires n query images to predict hyperparameters, while our LGPN requires only a single image as the input.

Three differences to [R2]:

• Unlike [R2], which uses estimated hyperparameters for generation purposes, we utilize these estimated hyperparameters for image forensic tasks.

• While [R2] directly applies HyperNet [R4] for the hyperparameter prediction, we develope a more sophisticated LGPN with a high generalization ability.

• [R2] focuses on estimating hyperparameters for a single architecture (the generator mentioned in [R2]), yet our approach is evaluated on 140 different Generative Models (GMs), many of which have distinctive architectures.

Q2: Multiple images input:

This is a valuable suggestion! Table R.5 reports results when the LGPN takes multiple images as inputs. Specifically, we denote input images and corresponding learned representations as {I1, I2 ... Ii} and {V1, V2 ... Vi}, respectively. We concatenate {V1, V2 ... Vi} for the final prediction. Table R.5 shows that more input images can possibly slightly improve LGPN's performance: inputting 12 images improves prediction on loss functions and continuous archi. hyperparameters but decreases in predicting discrete archi. hyperparameters. This means naively feeding LGPN with more inputs cannot provide substantial improvements, so a meaningful future research direction can be developing a specific module that effectively processes multiple images, such as meta-models as introduced in [R1] or ensemble learning methods. Secondly, when taking different numbers of images as inputs, our LGPN still maintains decent model parsing performance, indicating our approach's robustness.

Table R.5 Performance of using multiple input images.

Q3: Generalization to the unseen generative model (GM) and undefined hyperparameters:

We agree with you on the importance of generalization ability to unseen GMs and hyperparameters! First, LGPN has effective extensibility and is capable of predicting 37 defined hyperparameters of unseen GMs.

• Table 1 of the main paper reports performance on the RED140 dataset, where all models are evaluated on test sets that only have images from GMs unseen during the training.

• Also, we use Table R.6 to report performances on 10 more unseen GMs (e.g., styleGANv3, controlNet1.1, etc.) that are exclusive to the RED140, showing that both FEN-PN and LGPN can deliver reasonable generalization performances.

Table R.6: Model parsing performances on 10 more unseen GMs besides RED140.

Secondly, both FEN-PN [R3] and our work do not focus on predicting undefined hyperparameters beyond these 37 predictable hyperparameters. This is because the motivation of model parsing is to offer a new understanding to different GMs, thereby benefiting image forensic applications like coordinated attack detection. This understanding purpose is fulfilled by predicting 37 fundamental, predictable hyperparameters defined in [R3].

Of course, predicting undefined hyperparameters, e.g., LeakyReLU, is an important future direction for the model parsing research topic. For example, based on our LGPN, one can add a few new graph nodes for undefined hyperparameters while keeping the original graph nodes—the learned dependency between graph nodes representing ReLU and LeakyReLU should be high.

Lastly, we conduct an experiment (Table R.6.2) by naïve applying the pre-trained FEN-PN and LGPN to (i) classify if the input image is generated by GM with LeakyReLU and (ii) regress the continuous value about the negative slope used in LeakyReLU if available. The poor performance suggests that a trivial extension of existing model parsing methods is insufficient for undefined hyperparameter prediction, and a substantial new research is needed.

Table R.6.2: Performances on predicting LeakyReLU.

Q4: Efficiency discussion:

Although the graph size might increase when adapting to new hyperparameters, our innovative graph pooling design actually largely reduces the graph size via the learned graph pooling mechanism, offering a more computationally efficient solution than prior GCN-based methods [10,7,59]. We implement LGPN with different input graph sizes, which are proportional to the number of hyperparameters. Then, we report the inference speed of one forward propagation as follows, indicating that the time cost is not largely affected by the increasing number of hyperparameters involved.

Thank the authors for the response. The response have solved all my questions. I would like to maintain my score.