Feature-Level Adversarial Attacks and Ranking Disruption for Visible-Infrared Person Re-identification

w1 :Comparison with ReID

As illustrated in Figure 2, the differences and shared characteristics between visible and infrared pedestrian images can be understood. Compared to ReID, VIReID can extract features from two different types of images. While VIReID leverages the complementary information from both modalities, it also introduces additional modality-specific differences, such as the loss of crucial color information. However, the spatial information in both modalities remains unaffected and can mutually aid in capturing pedestrian features. By utilizing the FSAM, VIReID integrates global features from the frequency domain with detailed information from the spatial domain, thereby enhancing feature consistency and improving the representation capability of adversarial features.

w2: What is the total loss function? What is the meaning of ‘$\cal{L_{id}}$’ in Figure 2?

The total loss function can be formulated as follows:

$\cal{L_{id}}$ represents the identity loss. The training process of VIReID is considered an image classification problem, where each identity is a distinct class. During the testing phase, the output from the pooling layer or embedding layer is used as the feature extractor. Given an input image $x{i}$ with label $y{i}$, the probability of $x{i}$ being recognized as class $y{i}$is encoded using the softmax function and denoted as $p(y{i}|x_{i})$. The identity loss is then computed by the cross-entropy

where N represents the number of training samples within each batch.

w3: Visual experiments supporting FSAM

As shown in Figure 3, in the task of VI-ReID, the amplitude component captures the overall brightness and contrast of pedestrian images, reflecting the luminance and color information of the image, while the phase component captures the structural information and details of the pedestrians, including their shape and outline, to help distinguish between different details and features of pedestrians. The combination of these components allows for the effective extraction of global and local features of pedestrians. By focusing further on spatial information characteristics in the phase component, attention to spatial information in the frequency domain is increased, enhancing the ability to express distinguishing features.

Q1: the meanings of features $x_{v1}$ and $x_{i1}$

After applying the FSAM module to the spatial components of the original features, $x_{v1}$ and $x_{i1}$ are combined with the amplitude components of the original features and then transformed back to the original domain using the inverse Fourier transform (IFFT). This process can be described as:

Q2: Attacking more VIReID systems

We understand the reviewer's concern regarding the breadth of attack method validation. In this study, we chose the DDAG[1] and MRCN[2] systems as targets for the attacks. These two methods are representative in the existing VIReID field, employing dual attention mechanisms and modal restoration and compensation mechanisms, respectively, to reduce modality differences, covering different technical approaches. According to Table 4, although their modules provided some resistance, they still achieved very good results. This further demonstrates the broad applicability and effectiveness of our method.

[1] Dynamic dual-attentive aggregation learning for visible-infrared person re-identification. ECCV 2020

[2] MRCN: A Novel Modality Restitution and Compensation Network for Visible-Infrared Person Re-identification. AAAI 2023

Q3: Verify the effectiveness of different modules and their combination

We investigate the effectiveness of different modules and supplement our experiments with the effectiveness of each pair of specific modules. Comparing rows 5-7 with rows 1-3 in Table 5, our proposed modules are effective when used individually and show additive effectiveness when combined. Our experiments validate the superiority of the proposed method.

w1 & limitation: Discussion of the definition of adversarial attacks and what they mean, and how they can be applied in the real world

Adversarial attacks induce misclassification in classifiers by introducing subtle perturbations to inputs. In 2014, Goodfellow et al. [1] demonstrated this using a panda image. Since then, research has focused on identifying model vulnerabilities and improving model robustness. Studies reveal that deep neural networks are susceptible to adversarial examples, where minor input perturbations cause incorrect predictions with high confidence. As deep learning applications expand, the security issues they expose garner more attention.

Technologies like face recognition and person re-identification (ReID) have significant potential in security fields such as criminal investigation, person tracking, and behavior analysis. These technologies support the safe and stable operation of public society. However, their security and reliability are questioned in adversarial environments, limiting their specialized applications. Adversarial attacks offer a new perspective on system security, showing how adversarial examples can evade recognition or impersonate others. For human observers, these examples are often indistinguishable from legitimate ones but cause deep models to err. Evaluating recognition systems' robustness with adversarial attacks identifies system vulnerabilities, encouraging improvements in machine learning model robustness.

Similarly, visible-infrared person re-identification (VIReID) is increasingly used in security systems, necessitating an exploration of its security. Existing attack methods focus on visible image features, neglecting other modalities and cross-modal data distribution variations, potentially reducing their effectiveness in cross-modal image retrieval. This study examines VIReID model security and proposes a universal perturbation attack designed for VIReID.

[1] Explaining and harnessing adversarial examples. Arxiv 2014

[2] Efficient Decision-based Black-box Adversarial Attacks on Face Recognition. CVPR 2019

[3] Transferable, Controllable, and Inconspicuous Adversarial Attacks on Person Re-identification With Deep Mis-Ranking. CVPR 2020

w2: Visualization of antagonistic samples

As shown in the figure1, we can observe the before-and-after comparison of the visualization with the addition of UAP (Universal Adversarial Perturbation). Adversarial attacks involve applying perturbations that are imperceptible to the human eye, causing the model to produce incorrect outputs. The visualization results show no visible difference between adversarial samples and original samples. According to Table 5, although the images appear identical to the human eye after adding noise, they lose some of the original information, leading to decreased recognition performance.

However, our goal is to simulate real-world interference, hoping to enhance the representation capability of adversarial features to handle more complex and diverse scenarios. This approach aims to provide a stronger evaluation capability when assessing the security of VIReID systems, thereby enhancing the social value of our research.

w3: FSAM is not a new thing

While both our method and FDMNet explore feature extraction in the frequency domain, they differ in objectives and model design. FDMNet aims for modal-invariant feature learning via frequency domain decomposition, using the Instance Adaptive Amplitude Filtering (IAF) and Phase Preservation Normalization (PPNorm) modules to enhance modal-invariant components, as we as suppressing modality-specific ones. In contrast, our FSAM module integrates frequency and spatial features for adversarial feature alignment, using Universal Adversarial Perturbations (UAP) to generate adversarial samples. FSAM unifies features by combining the frequency and spatial domains, making visible and infrared image features more consistent.

In summary, FDMNet emphasizes the differences in amplitude components within the frequency domain, striving to enhance the consistency of modal features. Conversely, FSAM focuses on the commonality of phase components in the frequency domain, facilitating the alignment of adversarial features in both frequency and spatial domains.

Q:How could a real-world application employ this method for an attack in a visible-infrared surveillance system? The motivation for the methodology of this paper requires further elaboration.

This study presents a digital attack on VIReID, offering a new perspective on ReID system security. However, it also raises ethical and security concerns about the potential misuse of adversarial attack techniques, which could threaten public safety.

Despite these concerns, adversarial attack research has positive value. It uncovers vulnerabilities in existing systems, encouraging academia and industry to improve the robustness of machine learning models. This research assesses the robustness of VIReID systems and combines adversarial training with proposed attack methods, enhancing system security and benefiting society by promoting a safer technological environment.

Additionally, there is exploration of physical attacks in real-world applications. For example, AGNs[1] can create ordinary-looking glasses that cause facial recognition systems to misidentify individuals. AdvTexture[2] can cover clothing with arbitrary shapes, making individuals wearing such clothing undetectable by human detection systems. Wei et al.[3] propose using insulating materials to create physically feasible infrared patches with learnable shapes and positions, allowing pedestrians to evade infrared detectors.

[1] Adversarial Generative Nets: Neural Network Attacks on State-of-the-Art Face Recognition. Arxiv 2018

[2] Adversarial Texture for Fooling Person Detectors in the Physical World. CVPR 2022

[3] Physically Adversarial Infrared Patches with Learnable Shapes and Locations. CVPR 2023

This rebuttal addresses most of my concerns. I decide to raise my score to weak accept.

w1: The introduction of frequency domain and attention mechanism should be added in the related work.

Frequency domain： In recent years, frequency domain information processing has gained significant attention in deep learning, proving effectiveness for tasks like face recognition and person re-identification. This method analyzes high and low-frequency components to extract useful information by enhancing or suppressing them. For example, PHA[1] boosts high-frequency components for better pedestrian representation. Amplitude and phase components in the frequency domain can also be used to focus on style and color versus spatial information. For instance, SFMNet[2] uses Fourier transforms for face super-resolution, capturing global image information, while Zhang et al.[3] extract modality-invariant features by leveraging amplitude and phase components. Attention Mechanism: Attention mechanisms allow convolutional neural networks to focus on important features while ignoring irrelevant ones. These mechanisms can be categorized into spatial, channel, and other domains. SENET[4] focuses on channel-specific features by compressing spatial dimensions and learning in the channel dimension. The Spatial Transformer Network (STN)[5] captures important regional features by transforming deformed data. CBAM[6] combines spatial and channel attention sequentially to refine image features. For visible-infrared person re-identification, we designed an attention mechanism tailored for the spatial domain.

[1] PHA: Patch-wise High-frequency Augmentation for Transformer-based Person Re-identification. CVPR 2023

[2] Spatial-Frequency Mutual Learning for Face Super-Resolution. CVPR 2023

[3] Frequency Domain Nuances Mining for Visible-Infrared Person Re-identification VIReID. Arxiv 2024

[4] Squeeze-and-Excitation Networks. CVPR 2018

[5] Spatial Transformer Networks. NIPS 2015

[6] Convolutional Block Attention Module. CVPR 2018

w2: The motivation of FSAM

To generate more effective adversarial features, we propose the Frequency-Spatial Attention Module based on the following:

Frequency Domain Feature Decomposition: Visible and infrared images differ significantly in the frequency domain—visible images offer rich texture information, while infrared images contain mostly pixel data. Both share spatial consistency. We use Fast Fourier Transform (FFT) to decompose features into amplitude and phase components, capturing texture and spatial information, respectively. This approach allows targeted processing of each component, optimizing feature extraction.

Application of Spatial Attention: Attention is applied solely to the spatial component. Since the phase component is closely tied to spatial information, the spatial attention module can enhance or suppress specific regions of the feature map. This focus improves the model's ability to highlight important areas for better adversarial feature quality and performance.

Optimization of Cross-Modal Features: Emphasizing spatial information helps the model integrate features from different modalities, enhancing overall comprehension and generalization.

In summary, the Frequency-Spatial Attention Module improves adversarial feature quality and recognition accuracy by combining spatial attention with detailed frequency domain analysis.

w3: Replace it with channel attention

As shown in Table 2, we compared our frequency domain attention module with channel attention mechanisms like SE-Net[4] and ECA-Net[7]. The results reveal that while SE-Net and ECA-Net excel in certain areas, our frequency domain attention module is highly competitive across various metrics. It achieves the best Rank-1 and mAP scores on both datasets. Channel attention mechanisms often focus on global features, which may overlook local details and spatial relationships in cross-modal VIReID tasks, leading to potential information loss and reduced performance.

[7] ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks. CVPR 2020

w4: The effectiveness of FSAM on VIReID

The table3 compares the performance of three VIReID systems before and after the addition of FSAM (Frequency Domain Attention Module). All systems show significant improvements in Rank-1, Rank-10, and mAP after the addition of FSAM, indicating that FSAM plays a positive role in enhancing the performance of VIReID systems. Furthermore, the consistent effects across different datasets and search modes demonstrate its broad applicability and robustness.

Thank you for the author's response. This rebuttal addresses my concerns. I decide to change my initial score to Weak accept.

w1: Limitations of the proposed method**

Although the proposed method is effective in many scenarios, it has certain limitations and conditions under which it may fail. These include:

1. Extreme Imaging Conditions: Our method relies on the alignment of adversarial features across different imaging modalities, such as visible and infrared images. However, significant variations in imaging conditions can impact the effectiveness of the feature alignment. For example, extreme lighting conditions or significant occlusions may degrade the performance of the adversarial attacks, leading to a failure in disrupting the identification process.

2. Feature Consistency: The proposed frequency-spatial attention module is designed to enhance the consistency of features between visible and infrared images. Nevertheless, if the inherent feature differences between the two modalities are too large, the module may not effectively bridge the gap, resulting in reduced attack efficacy.

3. Generalizability: Our approach has been tested primarily on specific datasets and systems, such as SYSU-MM01 and RegDB. While these datasets provide a controlled environment for evaluation, the method's performance in real-world scenarios with diverse and unseen data remains to be fully explored. Factors like variations in camera quality, resolution, and environmental conditions can influence the generalizability of the results.

By acknowledging these limitations, we aim to provide a comprehensive understanding of the scope and applicability of our proposed method, thereby enhancing its utility and reliability in practical deployments.

w2 & Q: Analysis of computational complexity

According to Table 1, we analyzed the computational complexity of several methods.

Flops (Floating Point Operations): This metric measures the number of floating point operations required for a single forward pass of the model, reflecting its computational complexity. The Flops values of all methods in the table are quite similar, ranging from approximately 331.45G to 331.88G, indicating that these methods have comparable computational complexity.

Params (Number of Parameters): This metric measures the total number of trainable parameters in the model, reflecting its scale and complexity. All methods have a parameter count of 23.55M, suggesting that these methods have the same parameter scale, likely due to using the same base model or network architecture.

Conclusion: Despite the similar computational complexity (Flops) and parameter scale (Params) across all methods, there are significant differences in their performance metrics (Rank-1, mAP, mINP). Compared to other methods, our approach has similar computational complexity and parameter count, but performs significantly lower in performance metrics. We did not drastically increase the computational complexity; instead, we achieved superior performance by balancing metrics and computational complexity more effectively.

We have carefully evaluated and optimized the computational complexity of our proposed method to ensure a balance between performance and efficiency. As indicated in Table 1, we have adopted efficient techniques to handle cross-modality data (visible and infrared), reducing the additional computational overhead typically associated with processing different data types. We utilized an optimized network architecture that balances depth and width, ensuring efficient computation without sacrificing performance.

w3: Minor issue: Equation 15 appears to have an error

The corrected formula is: