Chinese Inertial GAN for Writing Signal Generation and Recognition

For weaknesses 1 and Question 1: The input consists of a 100-dimensional random noise vector and the devised Chinese Glyph Encoding representing the character class, concatenated together to form an input vector. This combined vector passes through a fully connected layer, producing an output of size 256, followed by ReLU activation and batch normalization. The output is then reshaped into a tensor of shape (256, 1), which undergoes a series of 1D transposed convolutional layers. The first transposed convolutional layer uses 512 filters with a kernel size of 7, applying 'same' padding, and is followed by ReLU activation and batch normalization. Subsequent layers reduce the number of filters to 256, 128, 64, and 32, each with a kernel size of 5, maintaining batch normalization and ReLU activation. The final layer consists of 6 filters, corresponding to the six output channels (three for accelerometer and three for gyroscope data).

The input to the discriminator consists of six-channel signals (three for accelerometer data and three for gyroscope data). The signal data undergoes processing through four 1D convolutional layers, which increase in filter count (64, 128, 256, and 512, respectively) and employ kernel sizes of 7 and 5. Each convolutional layer uses 'same' padding and Leaky ReLU activation with an alpha of 0.2, coupled with batch normalization to stabilize the training process. The final convolutional output is flattened and passed through two separate branches. The first branch, designated for real/fake classification, includes a fully connected layer followed by a Sigmoid activation function, yielding a scalar probability that indicates the likelihood of the input being real. The second branch, responsible for class label prediction, also comprises a fully connected layer, followed by a Softmax activation function that outputs a vector of probabilities across all potential classes.

The field of inertial sensors lacks deep learning-based data augmentation methods, and image-based methods are challenging to directly apply to time-series signals, so we chose cGAN as our baseline, which shares the same generator and discriminator as our model. In fact, our CI-GAN only adds three designed modules: CGE, FOT, and SRA. These modules introduce innovative enhancements—CGE provides semantic guidance by encoding the glyph shape of Chinese characters, FOT ensures feature consistency and prevents mode collapse through forced optimal transport, and SRA aligns the semantic relevance between inputs and outputs. These designs result in significant performance improvements, demonstrating the effectiveness and novelty of our approach.

For Question 2: Following your suggestion, we supplemented extensive comparative experiments, including 12 data augmentation methods covering five major categories. As shown in the Table below (Table 1 in the uploaded PDF), the results clearly demonstrate that CI-GAN significantly outperforms all other methods. Unlike in the field of image processing, it is challenging for humans to recognize semantic information in signal waveforms through observation or to judge whether the augmented signals are reasonable. Therefore, data augmentation methods specifically designed for images are not well-suited for sensor signals. In fact, there is a notable lack of deep learning-based data augmentation methods for inertial sensors. Our CI-GAN fills this gap and has been adopted and applied in the industry.

We sincerely hope our response addresses your concerns and we will incorporate all your suggested content into the accepted version.

Thank you once again for your insightful feedback on our paper. We have carefully addressed each of your concerns in our response, particularly regarding the detailed baseline configurations and the comparisons with various data augmentation techniques. We genuinely believe that our work offers valuable advancements in the field, especially given the lack of deep learning-based augmentation methods for inertial sensors.

We would greatly appreciate it if you could take another look at our revisions and explanations. Your feedback is invaluable to us, and we sincerely hope that our paper can meet your expectations and contribute meaningfully to the field.

Thank you for your time and consideration.

For weakness 1: The novelty of the CI-GAN consists of three proposed modules: Chinese Glyph Encoding (CGE), Forced Optimal Transport (FOT), and Semantic Relevance Alignment (SRA). These modules are interdependent, with many structures serving multiple functions. For example, CGE provides semantic guidance for the generator as a condition and imposes constraints on the generated results within the SRA. Similarly, the signal features in SRA also support the FOT module, making the entire architecture compact and sophisticated. Each module also has its innovation. For instance, SRA aligns the relevance between different generation outputs with the relevance between the prompts, significantly reducing hallucinations in the generative model. In June 2024, Nature published an article titled "Detecting Hallucination in Large Language Models Using Semantic Entropy," sharing a similar approach to our SRA. They assess the inconsistency in outputs when the same question is posed multiple times to a large model. Their approach essentially forces the model to give similar outputs for similar prompts. Our SRA goes a step further by ensuring that the relationships between the prompts are consistent with the relationships between the outputs, thereby reducing hallucinations and enhancing the model's practicality and stability.

Compared to the CV and NLP fields, the sensor domain lacks deep learning-based data augmentation methods. Most data augmentation techniques for images are not applicable to time-series signals. Our method is pioneering in the inertial sensor domain and has been adopted by a wearable device manufacturer. We apologize for missing some literature, and we will reference the recommended papers and similar work published in Nature in future versions.

For weakness 2: Unlike general embeddings that capture character meaning, CGE represents character glyph. Inertial sensors capture writing motions containing glyph information, allowing us to construct glyph features of each character during training. To achieve this, we introduce an encoding matrix following the one-hot input and design a Glyph Encoding Regularization (GER) based on Rényi entropy. As GER decreases during training, the Rényi entropy of the glyph encoding matrix increases, leading to two key effects: Each vector's information entropy increases, enabling it to carry more glyph information; The orthogonality between different encodings improves, capturing the differences between glyphs. This design may also benefit other categorical representation tasks by applying a similar regularization term to the category encoding dictionary. Additionally, in CI-GAN, CGE supports the SRA module, helping to alleviate hallucinations in the generative model. Moreover, CGE is also used in FFM to ensure consistency between real signal features, generated signal features, and glyph encoding, where CGE is also directly supervised.

The effectiveness of CGE is primarily due to GER, which is why we provided an ablation study of CGE without showcasing the ablation results of GER separately. Following your suggestion, we removed GER while retaining the encoding matrix, reducing it to a learnable transformation from one-hot encoding without additional guidance. This led to a significant performance decline, as shown in Table 2 (in the uploaded PDF), underscoring the critical importance of the Rényi entropy-based regularization in categorical representation tasks.

For weakness 3: WGAN primarily addresses the overall distribution differences between generated and real samples. In contrast, FOT incorporates a Forced Feature Matching (FFM) mechanism that enhances the realism of the generated signals by aligning their features with those of real samples. Unlike WGAN, FFM imposes an additional constraint, ensuring that the generated samples not only match the real data distribution closely but also maintain consistency in key features. This feature-specific matching is not explicitly achieved in WGAN, which is crucial for signal generation. Unlike images, the quality of signals cannot be readily assessed through visual inspection. Thus, stringent constraints are essential to ensure the reliability of the generated results.

Perceptual loss only constrains the consistency between generated and real signals. Identity loss in ArcFace ensures that the generated face images retain the same identity as the real faces. Differently, FFM proposes triple consistency constraints for generative models: prompt, generated signal features, and true signal features, which not only improves the realism of the generated signals but also ensures their SEMANTIC accuracy. Meanwhile, FFM also supervises the glyph encoding, reflecting the interaction between the proposed three modules. We will supplement these analyses and citations in future versions.

For weakness 4: We recruited two new volunteers, each using their smartphone to write 500 characters in the air, resulting in 1000 new samples. Writing Chinese characters, segmenting, and extracting corresponding signals for each character is extremely labor-intensive. Particularly, the segmentation phase requires optical equipment to precisely mark the start and end times of the signal segments corresponding to each character, which is a highly time-consuming task.

Given the excellent performance of CI-GAN, we used all 1000 newly collected samples for testing without retraining. As shown in Table 3 (in the uploaded PDF), the six classifiers performed even better, whose improvement is likely because these smartphones were newer, and the built-in sensors had not aged much, resulting in higher-quality signals. This demonstrates that the classifiers trained with CI-GAN-generated signals can adapt to sensors of different usage times, providing significant convenience for device manufacturers and leading to CI-GAN adoption and application in the industry.

Finally, we sincerely hope to receive your recognition.

Thank you once again for your thoughtful review and valuable feedback on our paper. We have taken your comments very seriously and have provided detailed explanations and clarifications in our response. We understand that assessing innovation can sometimes involve different perspectives, and we sincerely hope that you will consider our explanations, particularly regarding the aspects of novelty that we have highlighted.

Our research not only introduces some unique design elements but has also shown promising results in practical applications. We genuinely believe that this work can make a meaningful contribution to both the academic community and industry. If there are any further questions or areas that need clarification, we would be more than happy to discuss them in greater detail.

We truly appreciate your time and effort and hope that our work can meet your expectations.

We fully agree with your point: "It is very unlikely that we get more than we give to the system." In fact, what we give to the system is sufficient, as our training data provides multiple writing signals for each Chinese character. In comparison, humans can usually recognize new categories after just one exposure. This suggests that the model already receives enough information; the key is to utilize this information to thoroughly explore and memorize the intrinsic patterns of each Chinese character and then generate reasonable variations. To memorize the patterns of each character, we designed the Chinese Glyph Encoding (CGE) module, which effectively represents the shapes and stroke features of Chinese characters, providing a solid informational foundation for generating new writing signals. To generate reasonable variations, we introduced the Forced Optimal Transport (FOT) and Semantic Relevance Alignment (SRA) mechanisms. FOT addresses common issues in GAN, such as mode collapse and mixing, and establishes a triple constraint involving the prompt, generated signal features, and real signal features, ensuring the generated signals' authenticity and semantic accuracy. The SRA mechanism aligns the relationships of generated signals with the relationships of input Chinese glyph encodings, providing a group-level constraint, which mitigates hallucinations in the generative model, resulting in more realistic and reliable signal samples.

Following your recommendation, we employed five major categories of data augmentation—Time Domain, Frequency Domain, Decomposition, Mixup, and Learning-based strategies—encompassing 12 methods for comparison. All methods generated the same amount of samples (15,000) for training six classifiers, as shown in Table below (Table 1 in the uploaded PDF). Notably, except for our proposed augmentation method, the accuracy of classifiers trained using all other data augmentation methods failed to surpass 50%, whereas our method achieved over 90%. Additionally, due to the lack of deep learning-based augmentation methods in the sensor field, we could only compare our approach with cGAN, which performed worse than many non-deep learning methods, underlining the difficulty of designing deep learning models capable of generating accurate and realistic inertial handwriting signals and highlights the value of our CI-GAN.

In practice, collecting, segmenting, and processing these handwriting signals is a challenging task. We need to isolate and extract the segment corresponding to each character from a continuous signal flow, a process that requires identifying each character's precise start and end points, as shown in Figure 1 in the uploaded PDF. These points are not easily identifiable and often require optical equipment for accurate frame-level segmentation and annotation. We invested significant time and effort to obtain the original 4,500 handwriting signals. Our CI-GAN eliminates these difficulties by providing a straightforward method for generating handwriting signals, thereby saving time and resources.

Thank you for your insightful comment. We hope we have addressed your concerns.

I hope this message finds you well. I wanted to take a moment to sincerely thank you for your thoughtful and detailed review of our submission. Your insights have been invaluable in helping us refine our work, and we have made considerable efforts to address each of your concerns thoroughly.

In our response, we provided a detailed comparison with various data augmentation methods to highlight the significant value of our CI-GAN approach, which introduces generative deep learning models into inertial sensor data augmentation for the first time. Our results, which include extensive testing and ablation studies, show that CI-GAN not only performs significantly better than other methods but also offers a flexible platform that addresses the specific challenges of the sensor signal domain. We genuinely believe that our approach brings innovation to this field, particularly through the designed Chinese Glyph Encoding, Forced Optimal Transport, and Semantic Relevance Alignment, which together form a cohesive and effective system. These elements work in tandem to ensure not just the generation of realistic signals but also their alignment with the semantic content, which is crucial for practical applications.

Thank you once again for your time and effort. We are eagerly looking forward to your response.