Distributed Collaborative Learning with Representative Knowledge Sharing

Music City Center 

Distributed Collaborative Learning (DCL) addresses critical challenges in privacy-aware machine learning by enabling indirect knowledge transfer across nodes with heterogeneous feature distributions. Unlike conventional federated learning approaches, DCL assumes non-IID data and prediction task distributions that span beyond local training data, requiring selective collaboration to achieve generalization. In this work, we propose a novel Collaborative Transfer Learning (CTL) framework that utilizes representative datasets and adaptive distillation weights to facilitate efficient and privacy-preserving collaboration. By quantifying node similarity via Distributed Energy Coefficients, approximated from Taylor-expanded energy distance, CTL dynamically selects optimal collaborators and refines local models through knowledge distillation on shared representative datasets. These representatives, locally constructed synthetic datasets that encode conditional information, serve as a common ground for knowledge exchange and model comparison. We highlight how Representative Learning enables quantification of model heterogeneity, facilitates transfer under non-IID task distributions, and supports scalable generalization. Simulations demonstrate the benefit of adaptive collaboration, with CTL achieving superior trade-offs between personalization and global coordination. We also discuss a taxonomy of data heterogeneity types, including newly defined model and representation divergences, and illustrate their relevance to node alignment and collaborative efficiency.

Collaborative Transfer Learning

Knowledge distillation

Contrastive Learning

Federated Learning

Representative Learning