Distributed Linear Dimensionality Reduction Assisted by Centralized NN for Classification

We appreciate your insightful feedback and constructive criticism. Here is our response to the points raised:

1. Theoretical Section 3.1: We understand your perspective on the theoretical content of Section 3.1. Our intention was to provide a foundational understanding, but we recognize the need to align more closely with existing discussions on neural network optimality. To address this, we will condense Section 3.1 to ensure our discussion is not redundant.

2. Experimental Settings: Your suggestion to use more established neural network models and widely recognized datasets is well-taken. We plan to extend our experiments to include standard classification datasets like CIFAR10, CIFAR100, and ImageNet, addressing the concern regarding the initial dataset choices and enhancing the relevance and applicability of our findings in future work. It's important to note that our design is inspired by the MLP-Mixer architecture, which has shown state-of-the-art performance on various real-world datasets, rivaling both ResNets and Transformers. This aspect underscores the potential of our method in achieving high-performance results and will be highlighted in our revised experimental setup.

3. Experimental Results: We appreciate your concern regarding the clarity of our experimental approach, particularly in relation to the use of PCA and LDA. In our experiments, when comparing with PCA and LDA, we initially employed these methods to derive W1 to WN, and subsequently retrained the entire neural network with data processed by PCA/LDA. This approach aligns with your suggestion. Our findings indicate that, within this framework, our imbalanced-NN method significantly outperforms PCA/LDA. We acknowledge that this was not sufficiently clear in our initial manuscript, and we will ensure to clarify this methodology and highlight these results in our revised submission.

We appreciate your feedback and the opportunity to clarify the novelty aspects of our proposed method. While at first glance, the approach of data compression via linear projection for transmission and re-projection for training might appear straightforward, the novelty of our work lies in the specific implementation and the unique challenges it addresses in the context of edge computing:

1. Imbalanced Neural Network Approach: Our method goes beyond simple linear projection. It introduces an imbalanced Neural Network (NN)-based approach that is specifically tailored for resource-constrained edge devices. This aspect of our work is not just about compression but also about optimizing the balance between data reduction and classification accuracy, which is a non-trivial challenge in edge computing scenarios.

2. Superior Performance Over Existing Linear Methods: An additional point to highlight is the performance superiority of our approach compared to existing linear methods. Despite its seemingly straightforward nature, our method outperforms all current linear dimensionality reduction techniques in the literature, particularly in the context of similar computational complexity at the device side. This performance edge is a significant contribution of our work, as it demonstrates that even 'simple' methods, when innovatively applied and fine-tuned, can lead to substantial advancements in the field.

3. Adaptability to Device Constraints: Another innovative aspect of our approach is its adaptability to different device capabilities, particularly in scenarios with devices of varying computational powers and data dimensions. Our technique dynamically adjusts to these variations, which is a significant advancement over traditional methods that often assume uniform device capabilities.

4. Integration of New Devices with Minimal Retraining: Another novel contribution is the method's ability to integrate new devices with different data dimensions into an existing network with minimal retraining. This feature significantly reduces the computational overhead and is particularly beneficial in real-world applications where edge networks are dynamic and continually evolving.

Thank you for your feedback. We recognize your concerns regarding the applicability of our method in distributed learning, its practical utility, and the scale of our experiments. We would like to address these points as follows:

1. Practical Applications: The practicality of our approach lies in scenarios where bandwidth and computational resources are at a premium. For instance, in IoT networks or remote sensor setups, where transmitting large amounts of data is impractical or costly, our method can significantly reduce transmission needs while still providing effective classification capabilities. We will clarify and expand on these application scenarios in our manuscript to better highlight the practical utility of our approach.

2. Need for Linear Dimensionality Reduction: While it's true that DNNs perform nonlinear transformations and can yield high performance, they are not always the optimal choice in resource-constrained environments. Linear dimensionality reduction offers a valuable trade-off between complexity and performance, particularly in edge computing scenarios where device capabilities and network bandwidth are limited. Our method aims to optimize this trade-off.

3. Scale of Experiments: We acknowledge your concern regarding the scale of our experiments. Our initial experiments were designed to demonstrate the proof of concept and were hence conducted on a smaller scale for clarity and control. However, we recognize the importance of evaluating our method in more realistic settings involving larger datasets and a higher number of devices. We will extend our experiments to include larger datasets and a more extensive network of devices to better replicate real-world edge computing environments in future work.

We appreciate your detailed review and valuable suggestions. Below, we address each of your points:

1. Use of Larger Datasets: We acknowledge your concern regarding the dataset scale. The initial choice of a smaller dataset was driven by its common use as a benchmark in this domain, allowing for direct comparison with existing methods. However, we understand the need for testing on larger, more complex datasets to better demonstrate the effectiveness of our methodology. We will conducting additional experiments using larger datasets in future work.

2. Importance and Challenges of the Problem Setting: This research is crucial as it addresses the growing need for efficient data processing in low-cost edge devices, which are often constrained by limited computational resources and bandwidth. The challenge lies in performing effective dimensionality reduction without compromising classification accuracy, even when new devices with untrained dimensions are introduced. We will revise Section 2 to better highlight the importance and the challenges of the problem setting.

3. Technical Merits and Contributions: We understand that the technical merits of our proposed method might not have been clearly articulated. Our primary contribution lies in developing an imbalanced NN-based linear dimensionality reduction technique that is adaptable to varying device capabilities and can integrate new devices with minimal retraining. This approach significantly reduces the computational load on edge devices and conserves bandwidth.

4. Datasets and Testbeds/Devices in the Domain: Popular datasets in this domain, besides the ones we used, include CIFAR and ImageNet for image classification. Common testbeds for edge computing scenarios include Raspberry Pi devices. We might investigate them in future research.