Socialized Coevolution: Advancing a Better World through Cross-Task Collaboration

Thank you for your thoughtful feedback. We provide detailed responses to the issues. Please refer to the anonymous link (https://anonymous.4open.science/r/Supplementary-materials-for-SC/) for the supplementary tables and proofs.

Q1. Specialization-generalization trade-off

R1. DISC balances specialization and generalization by combining interactive learning with a dynamic master–assistant architecture, enhancing individual capabilities and task-level collaboration.

• Dynamic master–assistant architecture enables balance: For each task, DISC adaptively assigns the model most suited to the task as the master, with others serving as assistants. This role assignment ensures specialization without sacrificing generalization.

• Interactive learning enhances specialization: Classification (CLS) emphasizes image-wise, while detection (DET) attends to patch-wise. Specialization is enhanced through interaction.  

Q2. Connection between theory and empirical results

R2. We further clarify the connection between our theory and DISC through additional experiments (Table 9) and analysis. Sociability Information (SI) exists in both data and models (Definition 3.2), and Theorems 3.5 and 3.7 suggest that enhanced data and model representations enable SC. Therefore, we observed performance under different data and model enhancements, finding that DISC can better leverage SI (Theorem 3.3).

• Theory-guided method: 1) SI (Definition 3.2) is the premise of SC (Theorem 3.3). 2) Data augmentation improves SC (Theorem 3.5). 3) Representation enhancement improves SC (Theorem 3.7). 4) Driven by SI, we enhance the dataset through task-sharing and design DISC with DHC and DSC modules, improving representational capacity and achieving SC.

• Theory-empirical relationship: In machine society, data represents direct experience, and model interaction reflects indirect experience. Theorem 3.5 shows that diverse datasets strengthen the direct experience, while Theorem 3.7 highlights that expert model interaction improves the indirect experience. DISC leverages both through shared datasets and enhances representations via DHC and DSC, enabling SC (Table 9).

Q3. Paper title

R3. If allowed by the conference, we will revise the title to “Socialized Coevolution: Cross-task Collaboration Driven by Dynamic Interactions”, which better reflects the technical contribution.  

Q4. Ablation on progressive interaction and adaptive weighting

R4. Progressive interaction and adaptive weighting allow models to dynamically collaborate per sample without overwhelming the main model. In contrast, direct or static strategies often lead to excessive interaction and degrade learning.

• Progressive interaction: We explored sine- and cosine-based progressive modes under varying thresholds (Fig. 8 in Appendix). Sudden or deep interactions early in training can disrupt the main model, while gradual interaction maintains learning stability.

• Direct integration: We tested non-progressive integration by setting auxiliary weights to 1 (Fig. 3 in paper and Fig. 7 in Appendix). Results show that abrupt inclusion of auxiliary information fails to improve representation and may even hinder learning.

• Static weighting: If an optimal combination existed, only one red patch would appear (Fig. 3 in paper). The multiple red patches motivate dynamic weighting to enhance inter-model interactions. We provide a theoretical analysis showing that dynamic weighting outperforms static weighting [1], with dynamic weight negatively correlated with individual loss and positively correlated with collaborator loss (Eq. 19).

$ &GE(f)\\leq |\\mathcal{M}| \\left( \\mathcal{R}_N(\\mathcal{H}) + \\sqrt{\\frac{\\ln(1/\\Delta)}{2N}} \\right)+\\sum\_{m=1}^{|\\mathcal{M}|} \\hat{err}(f^m) \\ + \\sum\_{m=1}^{|\\mathcal{M}|} \\left[ \\frac{1}{|\\mathcal{M}|} \\underbrace{Cov(\\omega^m, \\ell^m)}\_{\*Negative self-correlation in DSC*} - \\frac{|\\mathcal{M}|-1}{|\\mathcal{M}|} \\sum\_{j \\neq m} \\underbrace{Cov(\\omega^m, \\ell^j)}\_{\*Positive collaborative correlation in DSC*} \\right] $

[1] Provable dynamic fusion for low-quality multimodal data. ICML, 2023.

Q5. Evaluation on diverse and large-scale datasets

R5. We additionally conducted experiments (including standard deviations) on CIFAR10, Food-101, and WIDER FACE (Table 10). Consistent results across multiple datasets show that DISC achieves better performance through dynamic interaction and collaboration.

Q6. Scalability and complexity

R6. With sufficient computational power, DISC can scale to any number of ensembles. For pairwise interactions, the total cost for N models is N(N−1)K, where K represents the per-interaction cost. KD methods are efficient but sacrifice generalization for specialization (Table 11). Compared to MTL, DISC incurs lower costs while better balancing specialization and generalization, facilitating other paradigms and applications.

Thank you for your valuable feedback. We provide detailed responses to the issues. For the supplementary tables and proofs, please refer to the anonymous link (https://anonymous.4open.science/r/Supplementary-materials-for-SC/).

Q1. Connection with social learning

R1. Human and machine societies share a common principle: individuals improve through interaction and collectives through collaboration [1-2]. Both DISC and Social Learning (SL) [3] emphasize acquiring knowledge directly and indirectly, which is empirically validated (Table 9). DISC is inspired by social learning (Eq. 16): its layer-wise hierarchy (Eq. 17), DHC (Eq. 18), and DSC (Eq. 19) correspond to social hierarchy, progressive learning, and cultural learning (Table 7), respectively.

• SL inspires DISC in machine societies. SL involves learning through observation (direct experience) and interaction (indirect experience), while DISC enables Socialized Coevolution (SC) through both data (direct experience) and models (indirect experience), reflecting SL's dual nature.

• Empirical evidence: Experiments (Table 9) show that direct learning from data builds basic ability, while indirect learning from collaborators depends on their ability. DISC combines both to realize SC through SL.

[1] A social path to human-like artificial intelligence. NMI, 2023.

[2] The secret of our success: How culture is driving human evolution, domesticating our species, and making us smarter. Princeton University Press, 2016.

[3] Social learning theory. Prentice Hall, 1977.

Q2. Scalability and generalization across models and tasks

R2. We analyze scalability and generalization from the backbone, training strategy, and task (Table 8), and find that sociability benefits from expert-based models.

• Backbone: ResNet152 outperforms ResNet50 due to its design and stronger representational capacity.

• Training strategy: Among ResNet50 variants, ImageNet pre-training yields the best performance, benefiting from the scale and diversity of the ImageNet dataset.

• Task: Each task corresponds to a different model. Classification (CLS), detection (DET), and segmentation (SEG) focus on image-, patch-, and pixel-wise, respectively, exhibiting sociability.

Q3. Influence of tasks and datasets characteristics on SC

R3. Tasks and data with higher sociability information (SI) better support SC (Theorem 3.3). As in multi-view learning [1], consistency and complementarity among views are essential for effective representation, SC similarly relies on SI (Definition 3.2) among models. Theorem 3.3 proves that SI is a prerequisite for SC and positively correlated with it.

• SI-SC positive correlation: In Definition 3.2, we formalize SI as $\\Phi\_{\\mathcal{M}\_2} = I(X\_{\\mathcal{M}\_2}; Y \\mid X\_{\\mathcal{M}\_1})$ Bayes error rate shows a positive correlation between SI (e.g., tasks and datasets) and SC.

$ \\frac{{H}({Y}|{X}\_{\\mathcal{M}\_{1}})-\\Phi\_{\\mathcal{M}\_{2}}-\log 2}{\log |{Y}|} \\leq {P}^{mul}\_{e\_c} \\leq 1-\\exp (-{H}({Y}\\mid {X}\_{\\mathcal{M}\_{1}})+\\Phi\_{\\mathcal{M}\_{2}}) $

• Data-level SI arises from diverse samples (Theorem 3.5): Structurally diverse datasets, like CIFAR-100 (more classes but low resolution) and VOC07+12 (fewer classes with high resolution and rich scene complexity), offer complementary information, enhancing SI and improving SC.

• Task-level SI derives from multi-wise information (Theorem 3.7): While all tasks aim to learn better representations, they focus on different levels, CLS (image-wise), DET (patch-wise), and SEG (pixel-wise). This complementarity yields SI, which benefits SC.

[1] A survey of multi-view representation learning. IEEE TKDE, 2018.

Q4. Interpretation of Fig.3

R4. Fig. 3 shows performance with fixed weight combinations for features and logits. If an optimal combination existed, only one red patch would appear. The multiple red patches motivate dynamic weighting to enhance inter-model interactions. We provide a theoretical analysis demonstrating that dynamic weighting outperforms static weighting [1]. Specifically, the dynamic weight is negatively correlated with individual loss and positively correlated with collaborator loss (Eq. 19).

$ &GE(f)\\leq |\\mathcal{M}| \\left( \\mathcal{R}_N(\\mathcal{H}) + \\sqrt{\\frac{\\ln(1/\\Delta)}{2N}} \\right)+\\sum\_{m=1}^{|\\mathcal{M}|} \\hat{err}(f^m) \\ + \\sum\_{m=1}^{|\\mathcal{M}|} \\left[ \\frac{1}{|\\mathcal{M}|} \\underbrace{Cov(\\omega^m, \\ell^m)}\_{\*Negative self-correlation in DSC*} - \\frac{|\\mathcal{M}|-1}{|\\mathcal{M}|} \\sum\_{j \\neq m} \\underbrace{Cov(\\omega^m, \\ell^j)}\_{\*Positive collaborative correlation in DSC*} \\right] $

[1] Provable dynamic fusion for low-quality multimodal data. ICML, 2023.

Q5. Formatting issue

R5. Thank you for your suggestions, we will fix it in the final version.

Thank you for your insightful feedback. We provide detailed responses to the issues. Please refer to the link (https://anonymous.4open.science/r/Supplementary-materials-for-SC/) for the Additional tables and proofs.

Q1. MTL setting

R1. The MTL setting is introduced to validate Sociability Information (SI) as the premise of Socialized Coevolution (SC). We include MAD [1] and Pix2SeqV2 [2] as MTL baselines (Table 6), emphasizing the specialization-generalization trade-off in SC.

• SI as the premise of SC: Freezing the backbone reveals common capacity. Classification (CLS) and detection (DET) focus on image- and patch-wise, respectively, and exhibit the potential of SC (Eq. 16) due to their SI (Definition 3.2).

• Characteristics of MTL: MAD and Pix2SeqV2 show that enhancing generalization sacrifices specialization.

• Trade-off in MTL: MAD and Pix2SeqV2 enhance generalization at the cost of specialization, while freezing the backbone preserves specialization but limits generalization.

[1] Masked AutoDecoder is effective multi-task vision generalist. CVPR, 2024.

[2] A unified sequence interface for vision tasks. NeurIPS, 2022.

Q2. Relation to cultural evolution

R2. Human and machine societies share a common principle: individuals improve through interaction and collectives through collaboration [1-2]. Cultural evolution includes Social Hierarchy (SH), Progressive Learning (PL), and Cultural Learning (CL), which motivates the SC (Eq. 16) and DISC (Table 7).

• SH: Rooted in human societies, we use layer-wise hierarchies to enable structured collaboration (Eq. 17).

• PL: Inspired by PL in human education, we use progressive interaction modes to bridge capacity gaps across models (Eq. 18).

• CL: CL mirrors guided learning in human culture, informing our strong-guided communication mechanisms with capability awareness (Eq. 19).

[1] A social path to human-like artificial intelligence. NMI, 2023.

[2] The secret of our success: How culture is driving human evolution, domesticating our species, and making us smarter. Princeton University Press, 2016.

Q3. Differences and similarities from model merging

R3. Model merging and SC share similar knowledge sources, but they differ in premises and goals.

• Differences: 1) Premises. Model merging assumes redundancy in parameters and orthogonality of task vectors, whereas SC is grounded in sociability. 2) Goals. Model merging aims to get a better-merged model, overlooking individual evolution. In contrast, SC improves individual models via interaction and the collective via collaboration, achieving coevolution at both levels.

• Similarities: Both acquire knowledge from other models.

Q4. Connection between theory and methodology & theoretical foundation

R4. We theoretically prove stronger SI leads to better SC (Theorem 3.3). In dynamic machine societies, SI manifests through data and models, enhancing their capacity to facilitate SC (Theorems 3.5 and 3.7). Thus, we propose the dynamically SI-driven DISC and further provide a theory.

• Theory-guided method: 1) Definition 3.2 formalizes SI, and Theorem 3.3 proves SI is essential to SC. 2) Theorems 3.5 and 3.7 show SI exists in both data and models. 3) We propose DISC based on SC (Theorem 3.3), and design DHC and DSC from different tasks (Theorems 3.5 and 3.7), offering insights into SI utilization (Eq. 20).

• Theory based on dynamic collaboration: Dynamic outperforms static [1], with dynamism manifested through progressive interactions and dynamically weighted collaboration. Weights of DSC (Eq. 19) negatively correlate with individual loss and positively correlate with collaborator loss.

$ &GE(f)\\leq |\\mathcal{M}| \\left( \\mathcal{R}_N(\\mathcal{H}) + \\sqrt{\\frac{\\ln(1/\\Delta)}{2N}} \\right)+\\sum\_{m=1}^{|\\mathcal{M}|} \\hat{err}(f^m) \\ + \\sum\_{m=1}^{|\\mathcal{M}|} \\left[ \\frac{1}{|\\mathcal{M}|} \\underbrace{Cov(\\omega^m, \\ell^m)}\_{\*Negative self-correlation in DSC*} - \\frac{|\\mathcal{M}|-1}{|\\mathcal{M}|} \\sum\_{j \\neq m} \\underbrace{Cov(\\omega^m, \\ell^j)}\_{\*Positive collaborative correlation in DSC*} \\right] $

[1] Provable dynamic fusion for low-quality multimodal data. ICML, 2023.

Q5. Training and inference details

R5. For CLS and DET, we set 79 and 147 iterations per epoch, respectively. DSC is active in training and inference. For more details, please see Section B in Appendix.

Q6. Robustness and generalization

R6. We assess robustness and generalization across the backbone, training strategy, and task (Table 8), finding sociability benefits from expert-based models.

• Backbone: ResNet152 outperforms ResNet50 due to its design and better capacity.

• Training strategy: ImageNet pre-training yields the best result, benefiting from the scale of the data.

• Task: CLS, DET, and SEG focus on image-, patch-, and pixel-wise, respectively, exhibiting sociability.