ADPretrain: Advancing Industrial Anomaly Detection via Anomaly Representation Pretraining

[To W1] (1) The use of "unsupervised anomaly detection" mainly refers to downstream AD tasks, as unsupervised AD is the most basic paradigm in anomaly detection (AD), and our experiments are also based on unsupervised AD methods. This "unsupervised" doesn't mean that the training process is unsupervised.

(2) Combined with (1), we think a good revision would be to use "industrial anomaly detection" in the title and the main text.

(3) The use of "pretraining" has two reasons: 1) We are indeed following the basic paradigm of pretraining: learning on a large-scale dataset, and the learned features can be transferred to downstream datasets and models. However, fine-tuning generally refers to fine-tuning a model on a dataset, and then applying the model to the dataset. We think that pretraining should not limit that must train the whole network. 2) More importantly, we want to emphasize that in the current development stage of AD, pretrained representations specifically learned for AD are of great significance. If it's revised to "pretrained features fine-tuning", the significance of this research direction cannot be well reflected. Current mainstream AD models mostly use pretrained networks to extract features. However, regardless of supervised or self-supervised pretraining on natural images, the pretraining process does not match the goal of AD. From the perspective of the development prospects of AD, it's necessary for AD tasks to have specialized pretrained features instead of continuously using basic pretrained features. We hope to inspire more future works to focus on anomaly representation pretraining.

[To W2] (1) We respectfully argue that you may misunderstand our Angle-Oriented Contrastive Loss. The data augmentation is to create a positive pair for each feature $x_i$. You may think that for a normal/abnormal feature, we use all other normal/abnormal features as positive pairs. But it's not like that (you can look the Eq.(2) again). For $x_i$, we only use $x_{i\prime}$ ($x_{i\prime}$ is the feature from the augmented image, its position in the augmented image is the same as the position of $x_i$ in the original image) as the positive pair. Other features with different labels from $x_i$ are used as negative pairs. This ensures that we only perform contrast between normal and abnormal. This is the same as in classical contrastive learning methods, where for a sample, only the augmented sample is treated as the positive pair, while other samples are treated as negative pairs.

Anomalies are usually different from each other, so abnormal features cannot be mutually used as positive pairs to attract together. For normal, there are also differences among different normal patches, so normal features also should not be mutually used as positive pairs. Thus, for $x_i$, we only use the feature $x_{i\prime}$ as the positive pair. Another reason is that if we use all features with the same labels to $x_i$ as positive pairs, this will cause numerical issues in the loss (i.e., appearing NaN).

In Eq.(2), we only use $x_i$ as the anchor feature, without using $x_{i\prime}$ as the anchor feature. That is to say, there will be no contrast between the augmented image and abnormal data in our loss. You can simply think about the fact that if $x_{i\prime}$ is not used as the anchor feature, there will be no negative pairs formed between $x_{i\prime}$ and abnormal features. Thus, there's no need to worry about the issue you mentioned.

(2) In line 197, we mentioned that the previous work FCDD [23] stated that the pseudo-Huber distance is a more robust distance measure that interpolates from quadratic to linear penalization; we follow to use this distance. Mathematically, when the value $x$ decreases from large to small, the curve of the pseudo-Huber function will change from quadratic to linear, while the curve of the Euclidean function is always quadratic. When $x$ is small, linear function can provide better gradient information compared to quadratic function. The term $e^{d_i}$ is a weighting factor. In line 205 to 209, we provided a good explanation. When $d_i$ is large (i.e., features are far from the hypersphere), $e^{d_i}$ can provide more weighting. When features are closer to the origin (i.e., $d_i \rightarrow -r$), $e^{d_i}$ can provide less weighting. Thus, the loss in Eq.(3) can adaptively assign larger gradients to the features outside the hypersphere for better contraction.

(3) In Fig.2, we provided a clear diagram of the Feature Projector, it's not hard to understand when combined with the diagram. The Feature Projector is actually cross-attention combined with MLP, but we use learnable reference representations as Key and Value.

In line 333 to 340, based on Tab.2(b), we provided a clear explanation. In Tab.2(b), for Cross Attention, we convert normal reference features into residual features and then utilize them as Key and Value. Compared to Self Attention, Cross Attention can perform better. Thus, we think that only embodying normal patterns in Key and Value should be effective. Then, we further adopt learnable Key and Value. Compared to limited normal patterns in Cross Attention, the learnable reference can adaptively learn to represent normal patterns and better cover key normal feature patterns. For downstream datasets, it is fixed to ensure the Feature Projector can properly generate good pretrained features. We attempted to set it as learnable, but the features yielded by the Feature Projector will appear pattern collapse issue.

(4) For all losses, $x$ means residual features. In Sec.3.2 and 3.3, we use $x$ as it is commonly used to represent features. In the revision, we will add a note to avoid confusion.

[To W3] (1) In line 311 to 313, we stated that all ablation studies were based on the VisA dataset. Thus, the test data comes from the VisA dataset.

(2) I respectfully argue that our method is not method-dependent, and you may misunderstand the meaning of embedding-based AD methods in our paper. In the introduction, we explained that we use embedding-based AD methods to refer to AD methods that use pretrained feature networks. So, the embedding-based AD methods in our paper actually cover a wide range. Like the pretrained ResNet-50 on ImageNet, our pretrained features can be applied to any AD methods that use pretrained feature networks. Perhaps this term ''embedding-based'' caused the misunderstanding. In the revision, we will make better modifications for this.

In Tab.1, we utilized 5 AD methods, including PaDiM and PatchCore (feature comparison based), CFLOW (normalizing flow based), GLASS (anomaly synthesis based), and UniAD (reconstruction based), to comprehensively validate the effectiveness of our method. This also demonstrates that our method is not method-dependent. In ablation studies, we evaluate the results based on PaDiM and PatchCore. We also explained in line 269 to 270: these two methods are based on feature comparison without learnable parameters. Thus, the results of these two methods directly depend on the quality of the representation. The better the discriminability between normal and abnormal features, the better the results. Thus, these two methods can directly reflect the quality of pretrained features. For hyperparameter experiments, it's costly for us to evaluate the results on all 5 AD methods, so we think that using PaDiM and PatchCore is the most appropriate.

(3) We think that our contrastive losses are not very sensitive to the three loss hyperparameters. As we found that different loss hyperparameters all can effectively reduce the loss value and converge to a close loss value. This indicates that the optimization effect of the model after loss convergence is similar, so its performance is also close. Compared to the loss hyperparameters, the effects of $N_r$ (see the following (4)) and $K$ are more obvious. In Table 10, the results of Rand(1, 4, 8) and 1 are 94.5/89.3 v.s. 93.8/88.1 under FeatureNorm.

(4) Thanks for your suggestion. We further conducted experiments with $N_r$ set to 32, 64, and 128, and the results are as follows.

It can be seen that the results significantly decrease with $N_r$ set to 32 and 64. This may be because too few learnable references cannot effectively cover key normal feature patterns. We also further conducted experiments with $N_r$ set to 3072 and 4096. When $N_r$ further increases, the results begin to decrease, and an obvious decrease occurs when $N_r$ is 4096. This may be because too many learnable references can lead to difficulties in optimization during training.

[To W4] (1) Due to too many experiments (please see Tab.1, with 6 AD methods, 5 datasets, 4 backbones, and with/without our pretrained features, there is a total of $6 \times 5 \times 4 \times 2 = 240$ experiments), we didn't have enough time and resources to use different random seeds for statistical significance. However, we ensured that all experiments were based on the random seed 42. Thus, we eliminated the influence of randomness, ensuring that all result comparisons are fair. In the revision, we will conduct multiple experiments to report the average values and standard deviation.

We hope the above responses can solve your concerns. If not, we sincerely hope to receive your reply and are very glad to further discuss with you.

Note: We try our best to compress the characters, but we still cannot reply to all questions in this box. We borrow some space from the rebuttal box (lower part) of Reviewer rLun to reply to the remaining questions. We sincerely hope that you can take a look.

[To W1] We think that the reasons for the performance degradation in the non-fixed backbone may be: (1) the basic visual representation capabilities possessed in the pretrained backbone are still valuable and important, and training the whole feature network on AD datasets may cause impairment to the basic visual representation capabilities. (2) The current scale and quality of AD datasets are not enough to support the whole network training of an AD backbone. The use of the word ''pretrain'' has two reasons: (1) We are indeed following the basic paradigm of pretraining: learning on a large-scale dataset, and the learned features can be transferred to downstream datasets and models. (2) More importantly, we want to emphasize that in the current development stage of anomaly detection, pretrained representations specifically learned for anomaly detection are of great significance. If it is revised to “pretrained features finetuning”, the significance of this research direction cannot be well reflected.

[To W2] In the paper spot-the-difference (SPD), the authors propose a data augmentation strategy called SmoothBlend to create slight local differences (or called pseudo anomalies). Then, the authors create pseudo anomalies on natural images from ImageNet to use as negative samples in contrastive learning. The model is pretrained on ImageNet, while our model is pretrained on a real AD dataset, RealIAD. (1) Compared to pseudo anomalies, the negative samples used in our method are more realistic and more in line with the abnormal characteristics in real-world scenarios. (2) In SPD, similar to classical SSL methods (SimCLR, MoCo), contrastive loss is constructed based on global features, while we construct contrastive loss based on more fine-grained local features. Although SPD mentioned that it can promote local sensitivity compared to SimCLR, MoCo, etc., it still may lead to the loss of information about local pseudo anomalies in the global features, as the local augmentation is slight. For fine-grained task like anomaly detection, contrastive learning based on fine-grained local features should be more suitable. (3) SPD was evaluated only on the ResNet-50, PaDiM, and VisA and MVTecAD datasets, while we evaluate our method on more backbones, AD models, and AD datasets.

As the SPD paper was accompanied by the introduction of the VisA dataset, and subsequent works mainly cited this paper due to the VisA dataset, we inadvertently overlooked the method. We greatly appreciate your suggestion and will add a discussion with SPD in the revision.

[To W3] Yes, at that time, we mainly felt that the hypersphere hypothesis in GLASS-h was more compatible with our norm-oriented contrastive loss. In the revision, we will also attempt to conduct experiments based on GLASS-m.

[To W4] Thanks for your suggestion. In the following two tables, we report the results (image-level AUROC/PRO) on carpet, grid, leather, tile, and wood classes from MVTecAD. The first table is based on CLIP-large, and the second table is based on ImageBind.

It can be found that the results on texture classes can also be effectively improved. We are sorry that we cannot provide you with qualitative results directly, as the rebuttal text box does not support uploading images. Previously, NeurIPS supported submitting a one-page PDF file to upload images, but this function was canceled this year. In the revision, we will add the corresponding qualitative results.

[To W5] For the feature map with a downsampling ratio of $s$, we divide the original mask into patches with $s\times s$ size. Then, for each patch, we sum the number of 1s (1 represents abnormal in the ground-truth mask). If the number of 1s is larger than 32, we label this patch as 1. Compared to ''nearest'' downsampling, our downsampling method can preserve more patches labeled as 1. Because the number of abnormal patches is much smaller than that of normal patches, we found that the ''nearest'' downsampling can cause instability in the angle-oriented contrastive loss (i.e., NaN).

[To W6] Yes, the features yielded by the Feature Projector are used to calculate both the Angle-Oriented Contrastive Loss and the Norm-Oriented Contrastive Loss.

[To W7] In line 333 to 340, we provided some explanations to our Learnable Key/Value Attention based on the results in Tab.2(b). In Tab.2(b), for Cross Attention, we convert normal reference features into residual features and then utilize them as Key and Value. Compared to Self Attention, Cross Attention can perform better. Thus, we think that only embodying normal patterns in Key and Value should be effective. Then, we further adopt learnable Key and Value. Compared to limited normal patterns in Cross Attention, the learnable reference should can adaptively learn to represent normal patterns and better cover key normal feature patterns in the residual feature distribution during pretraining. By subtraction, we aim to eliminate the normal representations in the residual feature distribution adaptively learned by the network, further increasing the discrepancy between normal and abnormal residual features.

[To W8] The need for some normal samples is a limitation mainly from residual features. However, we think that collecting a small amount of uncontaminated normal samples is usually feasible and easy for practical use. From the essence of anomalies (defined relative to normal), we think few-shot is more reasonable than zero-shot because at least we need to know what is normal and then detect anomalies. About contamination AD, you can see our response to Weakness 2 of Reviewer rLun, we provide some results and a discussion.

[To W9] We greatly appreciate your suggestion. Residual features are mainly relied on mutually eliminating class-related normal patterns in features through subtraction (you could see the ResAD paper for more explanations), for being class-generalizable. However, if normal images have significant variability, it may cause the few-shot reference samples to be not representative enough (lacking some class-related normal patterns). This may lead to certain class-related normal patterns in the input feature can't be effectively eliminated. Then, the discriminability of residual features under few-shot settings will be compromised.

For practical applications, this issue should be particularly focused and reasonably addressed. Of course, the simplest solution is to increase the number of reference samples. This is feasible, as in practical applications, the number of reference samples is usually not as strict as the 2-shot. However, in practical applications, we expect that the reference samples can fully represent their class, so it’s best to have sufficient differences between the reference samples. Thus, the sample selection strategy cannot be random. A feasible method is to first cluster all available normal samples into different clusters based on a clustering algorithm (e.g., KMeans). Then, based on the number of reference samples, we evenly distribute it to each cluster. When selecting from a cluster, we can prioritize selecting samples closer to the center. Another feasible strategy is to match spatially aligned samples as reference samples for each input sample. This will be helpful for the example you mentioned, as we can match samples with similar rotation directions to the object in the input sample as reference samples. We have provided a corresponding method in Appendix B.

In the revision, we will add the above discussion about the influence of reference samples on residual features and the sample selection strategy to the paper.

[To Q1] Yes, because one characteristic of residual features is that the semantic discriminability in features will be eliminated. However, for anomaly detection, it is okay not to distinguish different types of anomalies, and most AD methods can only detect anomalies and cannot give anomaly types.

[To Q2] We think that our pretrained features can be used for supervised anomaly detection tasks. In supervised AD tasks, the model is more likely to learn good decision boundaries between normal and abnormal based on our pretrained features by effectively utilizing the abnormal samples during training. Because our pretrained features have better discriminability for representing normal and abnormal compared to the original features. Due to time limitation, we currently don't have enough time to provide the corresponding experimental results. In the revision, we will attempt to apply our pretrained features to supervised AD methods.

We hope the above responses can solve your concerns. If not, we sincerely hope to receive your reply and are very glad to further discuss with you.

Dear Reviewer rLun, due to time limitation and to respond to multiple reviewers, we currently don’t have enough time to provide experiment results as comprehensive as in the paper. In the revision, we will supplement the results on other datasets.

[To W1] As the FYD paper does not provide open-source code, we can only compare with the results provided in the paper. In FYD, anomaly detection is achieved based on PaDiM. The paper provides the results (with WideResNet50) on MVTecAD, which are 97.7/98.2 (image-level/pixel-level AUROCs). Based on WideResNet50 and PaDiM, the results of our method are 98.1/98.5. In PANDA and mean-shift contrastive loss (MSC), feature adaptation is based on global features, so only image-level anomaly detection is performed. The two papers provide image-level AUROC on MVTecAD, with values of 86.5 and 87.2. For a more reasonable comparison, we utilized the open-source code of MSC and trained an adapted network (based on WideResNet50) on MVTecAD. Then, we utilized PaDiM for AD modeling, and the obtained results are 96.8/97.0. Thus, compared with PANDA, MSC, and FYD, our method has advantages in providing better features for AD models. As discussed in Related Work, adaptive feature fine-tuning requires specialized fine-tuning on each dataset. By comparison, a more important advantage of anomaly representation pretraining is that pretrained features can be used on multiple downstream AD datasets without the need for fine-tuning on each dataset.

[To W2] Thanks for your suggestion. We further conducted experiments with noisy data following the settings in SoftPatch. Specifically, we added abnormal data from the test set to the training set with a noise ratio of 0.1. We adopted the ''overlap'' setting from the SoftPatch paper, which means the injected abnormal images are also included in the test set. The results are as follows (based on ImageBind, image-level AUROC/PRO).

The results without noisy data are as follows.

It can be seen that the performance of the AD models significantly decreases when there is noisy data. Although the performance of the AD models with our pretrained features also decreases (this is normal, our paper is not intended to tackle the contamination AD task). However, our pretrained features can bring greater performance improvement under noisy data. This indicates that our pretrained features are more robust to noise than the original ImageNet-pretrained features. Moreover, our method is compatible with contamination AD methods, such as SoftPatch. When the training data has noise, we can first utilize the filtering method in SoftPatch to remove the most likely abnormal data, and then train based on the remaining data.

Finally, we note that for the above results, we don't add abnormal data to the normal reference samples. This is because when generating residual features, we only used few-shot and fixed normal samples as reference (i.e., 8 samples). We think that in real-world scenarios, even if training data has noise, it's easy and almost costless to manually collect few-shot normal samples as reference. Thus, we think that it seems not necessary to forcefully introduce abnormal data to the few-shot normal reference samples. In the revision, we will add the above results and discussion and cite the paper you mentioned.

[To W3] Thanks for your suggestion. Based on your suggestion, we further conducted a set of experiments based on EfficientNet-b6 and DINOv2-large, and the results are as follows (image-level AUROC/PRO).

It can be seen that we can also achieve performance improvement on EfficientNet-b6 and DINOv2-large. In the revision, we will supplement the results on other datasets and cite the paper you mentioned.

[To Q1] Please see our response to Weakness 1.

[To Q2] Please see our response to Weakness 2.

[To Q3] Please see our response to Weakness 3.

We hope the above responses can solve your concerns. If not, we sincerely hope to receive your reply and are very glad to further discuss with you.

---

---

---

---

---

---

---

---

---

---

Note: [To Reviewer rLun], you don't need to see the following content. Due to space limitation, we borrow some space to answer some questions of Reviewer oLCH. We sincerely apologize for any inconvenience caused to you.

[To W4] (2) As stated in Abstract, we promise that we will open-source the code, it's not to worry. When submitting the paper, we didn't organize all code well (with sufficient instructions), so we were unable to directly include it in the supplementary materials.

[To Q1] In the Norm-Oriented Contrastive Loss, the basic idea follows one-class-classification (OCC) learning. In OCC learning, normal features are usually optimized to be located inside the unit hypersphere (i.e., when $||x_i|^2_2 = 1$, it locates on the unit hypersphere). In the pseudo-Huber distance $\sqrt{||x_i|^2_2+1}-1$, when $||x_i|^2_2 = 1$, $\sqrt{||x_i|^2_2+1}-1 \approx 0.4$. Setting $r$ to 0.4 is equivalent to the unit hypersphere based on Euclidean distance.

[To Q2] WideResNet50 outputs two layers of features, with dimensions of 512 and 1024. The features generated by CLIP-base/CLIP-large/ImageBind are all 768/1024/1280 dimensions. The feature dimension has no direct relation to detection performance. Detection performance mainly depends on the feature network. It's not that with a larger feature dimension, the detection performance must be better.

[To Q3] In line 274 to 275, we mentioned that "we further propose a simple AD baseline (denoted by FeatureNorm), where we directly utilize feature norms as anomaly scores". For few-shot AD, our FeatureNorm is really simple. Specifically, for an input sample, we extract pretrained residual features from our pretrained model. Then, for a pretrained residual feature $x_i$, the L2 norm $||x_i||_2$ of $x_i$ is used as the anomaly score. In the revision, we will add this more specific description to the paper.

[To L1] We further conducted experiments based on OCSVM and KNN, and the results are as follows (based on ImageBind).

It can be seen that our pretrained features can achieve significant performance improvement, which further indicates that our pretrained features have better discriminability between normal and abnormal. Due to time limitation, we only conducted experiments on two datasets. In the revision, we will supplement experiments on other datasets and try other algorithms.

[To L2] In Tab.2(b), we have conducted ablation studies about MLP (the MLP Projector in Tab.2(b)). Does the ''readout layer'' refer to the network layers that output features? We further conducted experiments, and the results are as follows.

Compared to training the whole network, only training the output layers can significantly improve the results. This may be because fixing most parameters can effectively preserve the basic visual representation capabilities possessed in the pretrained backbone, which are still valuable and helpful. We greatly appreciate your suggestion. In the revision, we will include these ablation studies you mentioned.

Dear Reviewer yUBc, due to time limitation and to respond to multiple reviewers, we currently don’t have enough time to provide experiment results as comprehensive as in the paper. In the revision, we will supplement the results on other datasets.

[To W1] Thanks for your professional review. We use the phrase ''ImageNet-pretrained'' not to mean that the backbones used in experiments are all from ImageNet pretraining, but to refer to visual feature networks pretrained on all sorts of large-scale datasets (not just ImageNet). Because ImageNet is the most well-known pretraining dataset, we thus still use ''ImageNet-pretrained''. We are sorry that this caused misunderstanding and will modify the phrase in the revision. The CLIP and ImageBind models (please see Table 1) used in experiments are based on self-supervised learning (i.e., contrastive language-image pretraining), and are not pretrained on ImageNet. Our experimental comparison way is to separately use the original features and our pretrained AD representations in an AD model (in Table 1, we conducted experiments on 5 AD models), train the AD model, obtain the results, and then compare the corresponding results based on the two features. We keep all other aspects of the model consistent, so the differences in results only stem from the different features. Therefore, the comparison way can directly reflect that our pretrained features are better than the original features when used in AD models.

Based on your suggestion, we further conducted a set of experiments based on DINOv2-large, and the results are as follows.

The results show that we can also achieve performance improvement based on DINOv2-large. Because there are a large number of experiments under one backbone, we currently don't have enough time to run all the backbones you mentioned. In the revision, we will complete the results and validate our pretraining method on other backbones you mentioned. In addition, please see our response to Weakness 3 of Reviewer rLun. We also provide the experiment results based on EfficientNet-b6.

[To W2] We think that in the current development stage of anomaly detection, fundamental representation is more important than architectural novelty. In previous stages, many papers have proposed various architectural novelties, but overlooked that the essence of anomaly detection still can be attributed to the representation quality of features. In fact, architectural novelty is also aimed at better learning how to distinguish between normal and abnormal. If the normal and abnormal features are already highly discriminative (e.g., linearly separable), anomaly detection tasks can be easily accomplished without the need for designing increasingly complex architectural novelties. We also observe that most of the architectural novelties in AD papers are specific to the proposed method and are often hard to be still effective in other methods. However, good representations will be effective in various AD methods.

The current mainstream AD models mostly use pretrained networks to extract features. However, regardless of supervised or self-supervised pretraining on natural images, the pretraining process does not match the goal of anomaly detection. From the perspective of the development prospects of anomaly detection, we think that it's necessary for AD tasks to have specialized pretrained features instead of continuously using basic pretrained features. We think that pretrained representations specifically learned for anomaly detection are of great significance, and our work is only an early exploration. We hope to inspire more future works to focus on anomaly representation pretraining.

[To W3] When our pretrained features are used in downstream AD models, feature extraction and specific AD modeling are still required. Thus, there is no reduction in inference cost. We further conducted an experiment (only 10% of the normal samples from each dataset are used for training), and obtained the following results (based on ImageBind).

The results based on full training data are as follows.

The results show that with less training data, our pretrained features can bring more significant performance improvement, indicating that our pretrained features are beneficial for the sample efficiency (during training) of downstream AD methods.

[To Q1] Please see our response to Weakness 1.

[To Q2] As pretraining is usually based on a large-scale dataset, and currently most AD datasets only have a few thousand samples, we thus choose RealIAD for pretraining due to its data scale. To answer your question, we further performed pretraining on the MVTecAD dataset. The obtained results are as follows (based on ImageBind).

It can be seen that with pretraining on MVTecAD, the results are overall lower than the results obtained based on RealIAD pretraining. Thus, we think that the scale of the dataset will have an impact, a larger scale is more advantageous for pretraining. This also reflects that the AD field still needs larger and higher-quality datasets in the future to better support anomaly representation pretraining.

We are not quite sure about your definition of domain-agnostic. Generally, domain-agnostic means that a trained model can also perform well in other domains or datasets. From this perspective, our pretrained features can achieve better results than the original features when used on 5 downstream datasets and 5 AD methods. This indicates that our pretrained features are domain-agnostic (at least better than the original features) and method-agnostic. If there is any deviation between our understanding of domain-agnostic and yours, we hope to further discuss with you in the subsequent discussion phase.

[To Q3] Do you mean to combine our method with UniAD and train it on the RealIAD dataset? We think that residual features and the Feature Projector could be integrated into UniAD. These two contrastive losses could not be applied to UniAD because they are designed for representation learning, while the learning objective in UniAD is reconstruction. But we think that even if UniAD can perform well after training on RealIAD, it is only a specific AD model and cannot empower other AD models, while the pretrained fundamental representations are more valuable and can be used to enhance downstream AD models.

[To Q4] Please see our response to Weakness 3.

We hope the above responses can solve your concerns. If not, we sincerely hope to receive your reply and are very glad to further discuss with you.