LLM-DAMVC: A Large Language Model Assisted Dynamic Agent for Multi-View Clustering

Q1: The motivation appears to overemphasize feature extraction, while the primary objective of Multi-view Clustering (MVC) is arguably the exploitation of cross-view information and correlations. A clarification on how the proposed method balances these two aspects is needed.

A1: Our framework is predicated on the principle that robust feature representation is not a substitute for effective cross-view correlations, but rather a prerequisite for enabling such correlations. We acknowledge that the core of MVC lies in leveraging multi-view information and their inherent correlations. However, without high-quality features, even extensive utilization of multi-view information would yield suboptimal outcomes. To strike this balance, we employ a multi-level optimization strategy. At the foundational level, our model aligns views by minimizing cross-view contrastive loss, ensuring semantic consistency at the sample level. To capture deeper structural similarities, we introduce Frequency-Domain Contrastive Regularization , which aligns the global spectral patterns of features, thereby imposing a more profound form of consistency regularization. Finally, at the decision-making layer, a consistency loss based on KL divergence ensures that the final cluster assignments are consistent across all views. Thus, our approach does not merely focus on feature extraction; instead, it establishes a collaborative hierarchy that fully leverages multi-view information across diverse dimensions (i.e., feature space, frequency domain, and decision space). Therefore, we do not over-rely on feature extraction but rather “treat high-quality features as the cornerstone for multi-dimensional and in-depth cross-view information fusion.

---

Q2: Given that dynamic fusion strategies have been explored in prior MVC research, what are the unique contributions and specific novelties of the proposed dynamic fusion mechanism compared to existing approaches?

A2: While existing schemes explore dynamic fusion, most rely on single metrics for “dynamic weighting” or even adaptive weighting, which fundamentally differs from our approach. Our aggregator leverages a self-attention mechanism to enable context-aware, mutually enhancing feature interactions prior to aggregation. Critically, we pioneer introducing LLMs into the decision loop: rather than performing numerical computations, LLMs execute human-like, logic-based strategic reasoning. This post-LLM integration conducts multi-dimensional state assessment—moving beyond single metrics to holistically evaluate complex states encompassing prediction quality, cluster structure, and resource efficiency. This strategy indicate more flexible capacity when processing complex scenarios (e.g., “Agent A has high confidence but poor cluster balance, while Agent B shows the opposite”) and generate complete strategies featuring primary/backup agents, confidence thresholds, and fusion modes. In summary, compared with existing dynamic fusion strategies, our method is a high-level semantic reasoning-guided fusion strategy.

---

Q3: Could a precise, operational definition of "view-quality" be provided? Furthermore, what concrete experimental evidence can be offered to substantiate the central claim that static fusion strategies are inadequate for handling sample-level variations in view quality?

A3: We consider view quality is not a global attribute but a dynamic, sample-specific, and context-dependent state. Therefore, view quality should also be discussed at the sample level rather than solely at the global view level. Our framework operationalizes this insight by forgoing predefined criteria and instead implementing a real-time, multi-dimensional state assessment mechanism: as detailed in our paper, this mechanism evaluates a metric vector for each view on individual samples, encompassing confidence, entropy, cluster balance, feature quality (computed via intra-class distances), resource efficiency (integrating computation time and memory usage), and consistency with historical predictions—making such multi-dimensional evaluation far more comprehensive than static definitions.

To empirically substantiate our claim regarding the inadequacy of static fusion, we conducted a new "View-Quality Perturbation Experiment" on the CCV dataset. We partitioned the data into two class-based subsets (Group A: classes 0-9; Group B: classes 10-19) and selectively introduced significant noise to View 1 for Group A and View 2 for Group B, simulating drastic sample-level quality variations.

---

Q4: The experimental evaluation is not comprehensive enough in two aspects: insufficient description and small scale of the dataset, and the lack of key SOTA baseline methods.

A4: Regarding Dataset Description and Scale,Detailed statistics are provided below. While our selection includes standard benchmarks, we contend that characterizing them as uniformly "small-scale" overlooks their strategic diversity and complexity. Datasets such as NUS-WIDE-10, CCV, and Handwritten are recognized medium-scale benchmarks posing significant challenges due to their high dimensionality, view heterogeneity, and number of clusters.

We conducted the following comparative experiments using the relevant methods mentioned by the reviewer, including comparisons with t-SVD-MSC and TL-MSC schemes. The results are as follows:

[1]Xie Y, Tao D, Zhang W, et al. On unifying multi-view self-representations for clustering by tensor multi-rank minimization[J]. International Journal of Computer Vision, 2018, 126(11): 1157-1179.

[2]Jia Y, Liu H, Hou J, et al. Multi-view spectral clustering tailored tensor low-rank representation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(12): 4784-4797.

Q5: The manuscript contains several minor typographical and formatting errors that need to be addressed.

A5: We have thoroughly reviewed the entire text, corrected all identified errors, and improved language clarity to ensure the quality of the final version of the article.

Q1: How does your framework mitigate the negative impact of potentially unreliable pseudo-labels generated by the adversarial agent, especially in the early stages of training?

A1: We adopt two strategies to address this issue. First, at the loss-function level, we generate pseudo-labels based on cluster centroid proximity, not direct agent output. We then apply a confidence-based filter (high_conf_mask) to select only the most reliable labels for supervision. The influence of this supervision is further controlled via a curriculum learning scheme, where its loss weight (epoch_factor) gradually increases as training stabilizes. Second, our LLM-Assisted Dynamic Feature Aggregation (LDFA) module provides structural-level control. For Equations (17) and (19), the LLM assesses each agent's output quality and assigns dynamic weights. An adversarial agent producing unreliable results receives a lower weight, thus its negative impact on the final fused representation is actively suppressed.

---

Q2: Have you considered if applying established clustering algorithms like K-Means on the final embedding$\mathcal{Z}$ would yield better results than the current direct assignment method?

A2: The entire learning objective, particularly the clustering-specific loss $L_{clus}$ (Eq. 23), is explicitly formulated to train the network to produce a linearly separable embedding space $\mathcal{Z}$. The model is not merely encouraged to group similar samples, but to directly map their final representations to high-confidence categorical distributions. Applying K-Means would introduce a distance-based clustering objective (Euclidean distance minimization) that is not necessarily aligned with the primary, clustering-oriented loss functions used for representation learning. This objective mismatch could lead to suboptimal results by re-interpreting a space that was already optimized for direct assignment. Furthermore, our approach ensures stability of clustering result by avoiding the initialization sensitivity and additional hyperparameters inherent to iterative algorithms like K-Means.

---

Q3: Could you clarify the specific justification and significance of using heterogeneous (standard and adversarial) agents, as opposed to a single agent design?

A3: Using heterogeneous agents aims to simultaneously optimize for both stability and discriminability. The Standard Agent is dedicated to learning the stable, core structure of clusters via contrastive learning. Concurrently, the Adversarial Agent acts as a targeted regularizer, exploring the decision boundaries by generating adversarial samples (Eq. 8) to enhance inter-cluster separability and robustness. A single agent would struggle to balance these conflicting objectives. The significance of this dual-agent design is that it yields a final representation that is both structurally coherent and highly discriminative.

Q1: What is the unique advantages of using an LLM for dynamic weight assignment compared to traditional attention mechanisms or the lightweight policy network mentioned in the paper?

A1: The unique advantage of the LLM lies in its capacity for semantic reasoning and structured decision-making, which transcends the direct input-to-weight mapping learned by conventional attention or policy networks. As formulated in Equations (18) and (19), the LLM does not just learn a function; it interprets a structured prompt containing semantically meaningful metrics (e.g., prediction confidence, cluster quality). This allows it to perform higher-order reasoning, such as identifying a view as "high-confidence but low-certainty" and dynamically adjusting the fusion strategy accordingly—a capability that simple neural networks struggle to generalize, especially on out-of-distribution or complex view quality combinations. While the policy network provides efficient, fine-grained adjustments, the LLM provides a robust, logic-driven macro-policy. Our ablation study quantitatively confirms that removing this reasoning component leads to a catastrophic performance drop, demonstrating LLM's important contribution.

---

Q2: How does frequency-domain analysis help to improve performance for generic high-dimensional feature vectors that are not from signals or images?

A2: For any feature vector, the Fast Fourier Transform (FFT) provides a principled decomposition into components of varying "frequencies," where we conceptualize low-frequency components as representing the global, slowly-varying structure of the feature manifold, and high-frequency components as capturing local, fine-grained details and noise. Our Frequency-Spectral Contrastive Learning (Eq. 13-15) operates on the magnitude spectrum of these features, which is invariant to translation (i.e., shifts in the feature vector). By encouraging the low-frequency spectra of corresponding samples from different views to be similar, our model learns to align their global structural patterns, effectively acting as a powerful structural regularizer. This forces the model to capture consistent high-level semantics across views while simultaneously reducing sensitivity to high-frequency noise or minor feature perturbations.

Q1: Regarding the novelty of applying LLMs to clustering, and whether the LLM requires training or fine-tuning.

A1: Unlike prior methods using LLMs for static feature extraction, the novelty of our work lie in that we leverage its advanced reasoning to interpret complex inter-view relationships and adapt fusion strategies, without any LLM training or fine-tuning. This core mechanism, combined with our collaborative dual agent architecture and innovative dual domain (spatial+frequency) contrastive learning, is used for multi view clustering.

---

Q2: Regarding the analysis of how different LLMs affect performance.

A2: We test the clustering performace with different LLMs (i.e., DeepSeek-R1(8B) and LLAMA3), as summarized below:

---

Q3: Does the "view quality" evaluation capture view complementarity and noise levels, beyond just prediction certainty?

A3: Our view quality assessment metric focuses on evaluating the downstream impact of view features on the final clustering task, rather than directly quantifying noise or complementarity. Although these two metrics cannot be directly measured, indirect inference remains feasible. For instance, a view severely corrupted by noise struggles to generate clear and compact feature representations, resulting in poorly organized, more dispersed clustering structures with larger intra-cluster distances and consequently lower feature quality scores. This metric indirectly quantifies the harm of noise to the clustering task. Regarding complementarity, highly complementary views facilitate the model in forming clearer and more separable clusters by providing unique discriminative information, corresponding to higher feature quality scores, whereas redundant views yield lower scores.Crucially, this metric is not used in isolation but as part of a holistic system-level assessment by our LLM-guided router. The LLM integrates the metric with other indicators (e.g., prediction confidence, entropy, resource usage) to perform context-aware reasoning. For example, it can recognize that a view with an average performance score but persistent divergence from high-confidence consensus views may provide critical decisive resolving information, thus retaining it to address ambiguous samples. This cooperation between the metric and LLM reasoning is a core strength of our framework.

---

Q4: Why not call the LLM more frequently to capture finer, dynamic changes in the data?

A4: We employ the LLM as a strategic planner that sets the overarching fusion policy based on a stable and holistic view of the data, which does not require batch-level frequency. Frequent LLM calls would introduce prohibitive computational latency and risk destabilizing the training process by overreacting to batch-level noise. To capture finer dynamics without these drawbacks, we employ a lightweight policy network that updates every batch. This creates a highly effective "LLM macro-guidance + network micro-adjustment" framework, where the LLM provides robust strategy while the policy network handles rapid, tactical adaptations, thus achieving both clustering performance and computational efficiency.

---

Q5: A detailed introduction should be given on the application of LLM in clustering, and some formatting errors should be corrected to improve overall clarity.

A5: We will thoroughly revise the manuscript and expand our work on the latest applications of LLM in clustering tasks. In addition, we will carefully proofread the entire paper and correct all formatting and formatting errors.