G2PTL: A Pre-trained Model for Delivery Address and its Applications in Logistics System

Dear reviewer, thank you for your valuable suggestions on our work. Below are our detailed responses to your questions:

Questions 1: At inference time, do I need to have graph as input besides the text input for this model? From the model design, it seems that the graphomer is used in the model, thus I would expect edge and node input for the model? If the task is just address normalization without graph input, can we still use this model?

Response：

During the model inference phase, addresses need to be constructed as a graph. For tasks such as address normalization without graph input: Just build a single-node subgraph based on the address (This is a Self-Loop subgraph, edge_type_code = 1000 (Have a self-loop, no delivery route, no AOI co-locate, and no Alias)), and then enter this subgraph into the model.

Because when sampling subgraphs during the training phase, a weighted adjacency matrix is used to describe the relationship between nodes, and the weight value is edge_type_code. In particular, we use self-loop edge to describe the relationship between the node and itself. Its edge_type_code is 1000, which is used to distinguish it from the other 3 types. Therefore, when the downstream task does not have a graph input, you only need to build a single-node graph and add a self-loop edge to this node, which can be used for fine-tuning and inference of a single address. (This part of the code is also in the open source link:https://huggingface.co/Cainiao-AI/G2PTL. All experiments in the paper are tested in this way.)

Questions 2: Do we need to have a correct geographical map before using the proposed model in real life? What if the map is incorrect? How robustness of this model when the graphical information in inference time is noisy?

Response：

1. Whether a geographical map is required depends on your application scene.

   ○ If your application scene is a graph-level task (such as graph-based ETA, path prediction, etc.), we recommend that you input correct geographical map information to the model, which will help the model make more accurate judgments. If you do not have a geographical map, you can construct S2 co-locate edges through the S2 method to replace the AOI co-locate edges (this edge can also represent part of the geographical co-location information, but the method of dividing the earth is different), but the lack of other edges will reduce the performance of the model.

   ○ If your application scene is a single-address level task (such as address normalization, geocoding, etc.), at this time, you only need to construct the address as a single-node graph(The method is in the response to question 1).

2. When creating large-scale heterogeneous graphs, noise is already present in it ：

   ○ AOI co-locate：Limited by the performance of the geocoding service, the data source of AOI co-location edge may be wrong

   ○ Delivery Route: the courier’s delivery route may be wrong (positioning drift, etc.)

   ○ Alias: As shown in Fig1, each node will have multiple address descriptions, and the user will bring in input noise when writing the address.

We believe that these noises will slowly disappear as the amount of training data increases, because these noises are unbiased. At the same time, these noises will enhance the adaptability and robustness of the model. This is also reflected in our actual industrial deployment applications. We have deployed G2PTL into our online logistics system, which processes tens of millions of addresses every day (with the development of logistics business, there are many a new address that the model has never seen before). After the deployment of G2PTL, the accuracy of the address entity recognition online service has been improved by about 9% (84%->93%, the previous online model was BERT), which also reflects the good anti-noise performance of the model.

Questions 3: How large the graph do we need to prepare before using this model for address normalization? Do we need to have a global map or local map? If only local maps are used, is it possible that this method cannot learn the address similarity beyond a local region?

Response：

If you want to use this model for address normalization, you do not need to prepare a map, but you need to use the addresses to build a single-node graph as input (The method is in the response to question 1).

In the inference phase, the model only needs a local map; in the training phase, a global map is needed.

If you use a local map, our method can only learn information such as the spatial relationship of addresses inside the local map, but cannot learn the spatial relationship between addresses outside the local map. For addresses outside the local map, the model can only calculate the similarity of text-level addresses because the model has not learned the spatial information of the corresponding address.

Dear reviewer, thank you for your valuable suggestions on our work. Below are our detailed responses to your questions: Questions 1: Given that a fixed heterogeneous graph has already been predefined, how can the proposed method be scaled to new addresses?

Response：

1. If you are only using the model for inference on new addresses, you only need to build a single-node subgraph based on the new address(This is a Self-Loop subgraph, edge_type_code = 1000 (Have a self-loop, no delivery route, no AOI co-locate, and no Alias)), and then enter this subgraph into the model. Because when sampling subgraphs during the training phase, a weighted adjacency matrix is used to describe the relationship between nodes, and the weight value is edge_type_code. In particular, we use self-loop edge to describe the relationship between the node and itself. Its edge_type_code is 1000, which is used to distinguish it from the other 3 types. Therefore, when a downstream task takes a new address as input, you only need to build a single-node graph and add a self-loop edge to this node, which can be used for fine-tuning and inference of a single address. (This part of the code is also in the open source link:https://huggingface.co/Cainiao-AI/G2PTL. All experiments in the paper are tested in this way.)

2. If you want to add new addresses to the model for fine-tuning. There are 2 ways to extend the model to new addresses:

   ○ If there are a large number of addresses, a new heterogeneous graph can be constructed and trained according to the pre-training method we proposed. (If you cannot obtain edges with domain knowledge such as AOI co-locate and Delivery Route, you can use edges similar to S2 co-locate (refer to ERNIE-GeoL，KDD‘22))

   ○ If there are only a small number of addresses (<=10k), we recommend fine-tuning the model using single-node subgraphs constructed from new addresses.

Questions 2: How do you balance the significance of various tasks within the loss function? Based on the findings from the ablation experiments, it appears that HTC plays a more crucial role in pre-training?

Response：

We have observed that the convergence speeds of different losses are different during the training process. Among them, HTC-Loss converges the fastest because this task is simpler than MLM and Geocoding tasks (HTC only needs to focus on administrative hierarchy relationships among address entities. It only needs to learn all geographical administrative relationships, and the solution space is smaller than the other two tasks). We did not do any special work to adjust the balance between the three loss functions. We just maintained the loss weight setting of 1:1:1. Under this setting, we trained for about 1 week.

Due to training time and cost reasons, we terminated the training. At this time, HTC task has converged, MLM task is close to convergence, and the GC task has not fully converged. Our training loss log is also in the open source link:https://huggingface.co/Cainiao-AI/G2PTL.

We believe that different tasks focus on different knowledge.

In the experiment in Table 1, both the AEP and AET tasks are for a single address, and they should pay more attention to the the relationship of sub-words in the address. Therefore, from the experimental results, the importance of the HTC task will be higher, which is consistent with the goal of HTC task design.

For tasks related to geographical knowledge (absolute position positioning of addresses and relative positions between addresses), the importance of GC will be higher (the ablation experiments of the GC task in Table 1 and the ablation experiments of table 5 in the Appendix both prove this).

The MLM task helps the model handle more input noise (incomplete, partially duplicated, misspelled, etc.) and ensures the model's ability to understand address semantics. The cases in Figures 6 and 7 also prove the effectiveness of the MLM task.

Questions 3: What is the rationale behind choosing to sample from the entire graph as opposed to creating subgraph-level graph models? The former option demands significantly greater computational and storage resources.

Response：

We choose to sample from the entire graph for the following reasons:

1. Our goal is to train a PTM that contains rich delivery knowledge and geographical knowledge and can be applied to geographically related downstream tasks. Therefore, we believe that the training samples of the model should contain a large amount of delivery knowledge and geospatial knowledge. Subgraph-level models may not capture the global structure and relationships present in the entire graph. They may miss out on long-range dependencies and global patterns that can be crucial for certain tasks. Using a subgraph-level graph models will result in the loss of information between nodes, which is detrimental to training a PTM.
2. In our entire graph, the distribution of edge types is nonuniform. Take AOI co-locate as an example: This edge type will only appear between different nodes within the same AOI. It can only describe the co-locate relationship of nodes within the same AOI, and is a strong local pattern. It cannot describe the relationship between node pairs between two AOIs (even if they are adjacent in real geographical space, such as Node 1 and Node 2 in Fig. 1). If we only use AOI co-locate edges to build a sub-graph model, it may only model the local geographical knowledge in our system and miss the global geographical information. Delivery Route edges can supplement it with the geographical information of Cross-AOI, making the edge distribution more consistent with the real world(a closer direct distance of two addresses may take a long time to travel due to geographical barriers.). Details of our heterogeneous graph and the distribution of edges in the heterogeneous graph can be found in the Appendix of the paper.

Dear reviewer, thank you for your valuable suggestions on our work. Below are our detailed responses to your questions: Questions 1: The representation in Figure 1 lacks clarity. The relationship between the left and right sections of the figure is puzzling. For instance, the right side depicts an edge between node 1 and node 2 labeled "001", suggesting "no delivery route, no AOI co-location, and has Alias." However, the left side appears to contradict this, showing a delivery route between node 1 and node 2.

Response：

Sorry, we didn't describe the edge encoding clearly enough. In the binary encoding result of the edge, position 0 is the last bit in the result(rightmost) and position 2 is the first bit(leftmost). Therefore, the edge labeled "001" between node 1 and node 2 in Figure 1 indicates " no AOI co-locate, no alias, only delivery route".

I have read the authors' responses and decided to keep my rating.