Switchable Token-Specific Codebook Quantization For Face Image Compression

Answer 1:

We re-evaluated our method using three different random seeds, and the results are presented in the table below. Due to time and resource constraints, we are only able to report results based on the TiTok-S baseline. As shown in the table, the impact of random seed variation is not significant, which can be attributed to the fact that the core of our method lies in training the codebook, while the encoder and decoder are initialized with pre-trained weights. We plan to update the re-evaluation results for other baselines in future versions.

Table: Switchable Token-Specific Codebook Quantization on different baselines.

Answer 2:

We have reported the FLOPs of the model, the required storage overhead and the inference time per image on a V100 GPU in the table below. As shown, our method does introduce additional storage and inference latency, and this effect becomes more pronounced as the number of codebook groups and the size of the codebook latent space increase. We attribute this to the need to maintain all codebook groups during inference when selecting the nearest codebook based on the minimum error criterion. To address this, we experimented with directly using the routing results during inference, so that the decoder only needs to access the codebook corresponding to the routing index. The benefits of this approach are clearly reflected in the table: for example, with the MaskGIT backbone, storage overhead is reduced by approximately 60% and inference latency by about 40%, while remaining the performance of MeanAcc and IDS.

Table: Inference latency and memory overhead statistics. For ours method, 'NN' and 'CR' refer to Nearest-Neighbor search and Codebook Routing search respectively.

Table: Quantitative comparison between different codebook selection policies during inference.

Answer for limitations 1:

We conducted experiments on the four ethnic groups in the RFW dataset, and the results are shown in the table below. Our findings indicate that the introduction of multiple codebooks does not lead to significant changes in ethnic bias. Although our supplementary materials reveal that the codebook groupings are related to ethnicity, the number of codebook groups is much larger than the number of ethnic categories, so each group is not strongly associated with any single ethnicity. We suggest that future work could explore fine-tuning the decoder for each ethnic group in our stage 3 to better balance recognition performance across different groups. We will include a discussion of these limitations in future versions.

Table: Quantitative comparison between different codebook selection policies during inference.

Answer for limitations 2:

We acknowledge that ultra-low-bpp face compression carries the risk of facilitating large-scale surveillance and potential misuse. To mitigate such concerns, we will provide clear usage guidelines and ethical instructions upon the open-sourcing of our project, and incorporate watermarking into the compressed images to help prevent improper use. We will include a statement addressing these issues in the limitations section of a future version.

Answer for paper writing:

Thank you for pointing out the editorial error; we will correct it in future revisions. Specifically, regarding lines 284-285, our intention was to convey that our method is able to maintain—or even surpass—the baseline performance as the bpp decreases. For example, when bpp = 0.0469, the baseline achieves a MeanAcc of 90.70%, whereas our method achieves a MeanAcc of 92.23% at a lower bpp of 0.0391. We hope this clarification addresses your concerns.

Answer for method 1:

Selecting the most appropriate codebook for the model poses a significant challenge. During training, we experimented with directly using a nearest neighbor strategy for codebook selection. However, we observed that this approach resulted in a notably low codebook utilization rate of only 19.89%, indicating that the multiple codebooks were not being fully exploited. We attribute this to the nearest neighbor strategy’s tendency to favor codebooks that have been more thoroughly optimized, leading to a collapse where multiple codebooks degenerate into a single codebook.

Table: Comparisons of different codebook selection strategies used while training.

To address this issue, we designed a routing strategy based on entropy-regularized optimization during training, which encourages the model to select a more diverse set of codebooks and mitigates codebook overfitting, thereby promoting more comprehensive codebook training. During inference, we adopt the nearest neighbor strategy based on the minimum loss principle, as the codebook with the smallest loss corresponds to the upper bound of our model’s performance. We also experimented with using the routing strategy during inference and found that its performance was comparable to that of the nearest neighbor strategy, while offering a more efficient inference process. Specifically, it significantly reduced both inference latency and storage overhead. For example, under the TiTok-S architecture, the inference latency per image using the routing strategy was 0.0103 seconds, while remaining the MeanAcc of 87.54%, compared to 0.0154 seconds with the nearest neighbor strategy for the MeanAcc of 87.60%.

Table: Quantitative comparison between different codebook selection policies during inference.

Answer for method 2:

We apologize for the oversight—there is an error in Equation 8 in the manuscript. The correct form is $\mathcal{L}\_{\text{ent}} = \sum\_{i=1}^M \bar{g\_{\theta}}(\mathbf{z}\_e) \log \bar{g\_{\theta}}(\mathbf{z}\_e)$, where $\bar{g\_{\theta}}(\mathbf{z}\_e)$ denotes the average probability of the current class across samples in the batch. Equation 8 is designed to encourage the router to select diverse codebook groups for different samples within the same batch, thereby fully exploring and utilizing the expanded codebook latent space. In contrast, Equation 9 measures the entropy of the predicted distribution for each individual sample, aiming to suppress excessive uncertainty in single-sample predictions. Together with Equation 10, these three objectives not only promote comprehensive utilization of the codebook space, but also effectively balance inter-sample diversity with the certainty of individual predictions.

Answer for experiment 1:

You are absolutely right that including image quality metrics is essential for a comprehensive evaluation. In response, we have additionally assessed both 1D and 2D baselines, using three widely adopted metrics: SSIM, PSNR, and LPIPS. The results are summarized in the table below, where IBQ refers to the state-of-the-art 2D baseline published at ICCV 2025. As shown, our method achieves significant improvements in image quality, further demonstrating its effectiveness. Moreover, we observe that the LPIPS metric fluctuates across different baselines. We hypothesize that this is because the LPIPS score is computed using a network pre-trained on ImageNet to assess perceptual similarity between images, while there are substantial differences between facial data and ImageNet data. Therefore, we prefer to use MeanAcc and IDS as more reliable indicators of machine perception for facial compression tasks. We will incorporate these latest results in the revised version of our paper.

Table: Quantitative comparison about image quality with baseline methods.

Answer for experiment 2:

We have provided the detailed recognition accuracy and IDS metrics on five face recognition datasets, as shown in the table below.

Table: Recognition accuracy(%) on five facial datasets.

Table: IDS on five facial datasets.

Regarding your suggestion to compare with face compression methods, we have carefully reviewed the three representative papers you mentioned. Unfortunately, none of these works have released their code, and their experimental settings for the face compression task differ from ours. Specifically, these studies focus on preserving image details in high-resolution data. For example, [1] trains on the high-resolution FFHQ dataset and evaluates image quality on CelebA-HQ, which is not directly comparable to our setting, where the goal is to compress images from face recognition datasets while preserving identity features.

Nevertheless, we attempted to use screenshots of the visual results in Fig.6 from [2] and computed the IDS value between the original images and compressed images to assess the effectiveness in preserving facial identity. As shown in the table, our method demonstrates superior performance in identity preservation. In future versions, we will also include visual comparisons with these face compression methods.

Table: Comparison of IDS between other facial compression methods and ours.

Answer 1:

We have provided a detailed analysis of the required storage overhead and inference latency on a V100 GPU, as shown in the table below. Admittedly, selecting the codebook group based on the minimum error during inference does introduce additional overhead, which is particularly noticeable when using MaskGIT-VQGAN with 256 tokens as the backbone. To address this, we also experimented with directly utilizing the routing results during inference. In this approach, the model only needs to load the codebook group indicated by the routing decision, rather than maintaining all codebook groups on the decoder side. This strategy offers clear advantages, significantly reducing both storage overhead and inference latency. In the future, we also plan to further mitigate this issue through optimization techniques such as model quantization or pruning.

Table: Inference latency and memory overhead statistics. For ours method, 'NN' and 'CR' refer to Nearest-Neighbor search and Codebook Routing search respectively.

Answer 2:

Our method can be readily transferred to images from other domains and is equally applicable to more complex and diverse datasets. For example, we report experimental results on ImageNet-1K, which contains more than 1M images from different categories. As shown in the table below, our approach achieves significant improvements over the original baseline across various metrics. Specifically, the FID score is reduced from 3.91 to 2.27, and the LPIPS score is improved from 0.3889 to 0.3564.

Table: Evaluation of reconstruction performance on the ImageNet-1K 256 × 256.

As discussed in Section A.4 of the supplementary material, we did observe that when increasing the number of routing groups while keeping the size of each codebook fixed, the evaluation metrics first improved and then declined. We attribute this to the limited size of our training dataset CASIA-WebFace, which is insufficient to fully utilize the large codebook space. Additionally, increasing the number of groups raises the routing complexity, resulting in under-trained embeddings within each codebook group. These factors together negatively impact the evaluation metrics. We plan to address this issue in our future work.

Answer for Q1:

We apologize for the oversight—there is an error in Eq. 8 in the manuscript. The correct form is $\mathcal{L}\_{\text{ent}} = \sum\_{i=1}^M \bar{g\_{\theta}}(\mathbf{z}\_e) \log \bar{g\_{\theta}}(\mathbf{z}\_e)$, where $\bar{g\_{\theta}}(\mathbf{z}\_e)$ denotes the average probability of the current codebook group across samples in the batch. This loss is to encourage the router to assign different codebook groups to samples within the same batch, thereby making full use of the expanded codebook latent space.

Answer for Q2:

We appreciate you highlighting this oversight. You are correct—this was an editorial error, and the second $C_1$ should indeed be $C_2$. Regarding the codebook quantization in our method, when both switchable codebook quantization and token-specific codebook quantization are used in combination, the total number of sub-codebooks is indeed $M \times T$. Furthermore, $C_{global}$ refers to the globally shared codebook in previous methods, where all $T$ tokens share a common set of $K$ embeddings. We hope this clarification addresses your concern.

Answer for Q3:

We investigated the utilization of sub-codebooks when assigning separate codebooks to individual tokens. Our findings reveal that these sub-codebooks exhibit no sign of redundancy. Conversely, the token-specific codebook strategy not only further enhances the codebook utilization per token but also mitigates the imbalanced utilization allocation observed when using a single, shared codebook across all tokens, as shown in the table below. For those tokens with low utilization, we believe this may be due to their limited information content, which is related to the performance of the encoder. In the future, it would be reasonable to assign codebooks of varying sizes to different tokens to further reduce redundancy.

Table: Codebook utilization rates (%) per token on LFW dataset.

Answer for Q4:

We have now supplemented the main results in Table 1 by adding the corresponding number of codebook groups ($M$) and the size of individual codebooks ($K$). The updated results are shown below.

It is important to note that the main results presented here solely utilize the token-specific strategy. Consequently, the switchable codebook strategy—specifically designed to reduce the bits per pixel (bpp)—is not included in this main results table. Instead, it can be found in Table 2 within the main body of the paper.

Table: Detailed quantitative comparison with state-of-the-art methods.

Answer for Q5:

When quantizing each token, we employ the nearest neighbor search strategy, not the routing mechanism used at the image level. The reason for this design choice is that, if we were to continue using the routing selection mechanism for each token, it would indeed enhance the diversity of token representations. However, this approach would require storing an additional group index for every token, introducing extra overhead. This contradicts our goal of reducing bpp while increasing the overall codebook capacity. Specifically, assuming that image-level codebook routing is not considered, an image is represented by 256 tokens. If each token-specific codebook is divided into $M$ groups, each containing $K$ entries, then the number of bits required to represent an image is $256 \times (\log M + \log K)$. For example, given an original codebook size of 4096, the required storage is $256 \times \log 4096 = 3072$ bits. However, if we partition the codebook into 256 groups, each of size 256, the storage increases to $256 \times (\log 256 + \log 256) = 4096$ bits.

Answer for Q6:

Yes, we have compared the codebook utilization between nearest neighbor search and our routing selection mechanism, and the results are shown in the table below. When using nearest neighbor search, both codebook utilization and evaluation metrics drop significantly—especially codebook utilization, which falls to only 19.89%. This clearly contradicts our original motivation for introducing multiple codebook groups. We believe this is because, with nearest neighbor search, the model tends to favor codebooks that have been more extensively optimized, leading to redundancy in other codebooks. Therefore, we introduce the routing selection mechanism to encourage the model to make full use of as many codebooks as possible, thereby leveraging the diversity of the codebook latent space.

Table: Comparisons of different codebook selection strategies used while training.

Answer (a):

Thank you for your valuable suggestion. To better assess perceptual quality, we have included visualization examples in the supplementary material. However, due to policy limitations, we regret that we are unable to incorporate additional visual examples at this stage. To further validate the image generation quality of our method, we have conducted quantitative evaluations of image quality metrics, as presented in the table below. We are committed to providing more comprehensive visualizations in future iterations to further demonstrate the efficacy of our approach.

Table: Quantitative comparison about image quality with baseline methods.

Regarding "decreasing the single codebook size might damage the ability to preserve token diversity", in fact, directly reducing the size of a single codebook can indeed weaken its representational capacity. However, our method addresses this issue in two perspectives.

First, we argue that a more diverse or larger codebook feature space is not always optimistic solution; rather, it should be well-matched to the latent space of the encoder outputs. If the codebook covers a very broad space but the tokens cannot effectively utilize it, this will lead to low codebook utilization and wasted resources.

Second, We observed that the original strategy of using a global-shared codebook across all tokens leads to an uneven distribution of codebook utilization. As shown in the table below, different tokens exploit the shared codebook but with varying degrees. To address this, we propose a token-specific codebook strategy, where each token learns its own codebook to maximize the preservation of its visual characteristics. With our approach, the issue of uneven codebook utilization among tokens is significantly alleviated. This demonstrates that our codebook is better able to fit the feature space of the encoder outputs. As a result, the feature information of each token can be more effectively quantized by the current codebook, thereby reducing quantization errors caused by codebook utilization imbalance.

Table: Codebook utilization rates (%) per token on LFW dataset.

Finally, by introducing multiple groups at the image level and employing a routing strategy to assign the most suitable group to each image, our approach effectively increases the overall codebook capacity, even if the size of each individual codebook is reduced. Moreover, each codebook becomes more specialized in capturing the common features of a particular category of images, which helps maintain the diversity of the codebook representation space.

Table: Total codebook utilization rates (%) under five face recognition datasets.

Answer (b):

To maintain the representational capacity of the codebook latent space as individual codebook size decreases, our method employs multiple groups of codebooks and effectively utilizes them. To better evaluate the utilization of these multiple codebook groups, we used the TiTok-S baseline as an example. We tested their codebook utilization under three different bpps, and the results are presented in the table below.

As shown in the table, the overall codebook utilization increases as the number of codebook groups grows. This result supports our hypothesis that, by organizing codebooks into groups, we can use smaller codebooks to capture more representative features. Consequently, this approach allows us to maintain the diversity of the codebook representation space while reducing the bpp during compression.

We have also evaluated the inference latency and storage overhead introduced by our method on a V100 GPU, as summarized in the table below. The table reports the inference latency and required storage for three baselines under different bpp (bits per pixel) settings. As shown, the expansion of the overall codebook size does indeed incur additional storage costs, along with a slight increase in inference latency. Since the number and dimensionality of tokens in the MaskGIT baseline are both significantly higher than those in the TiTok baseline (256 vs. 128 tokens, 256 vs. 12 dimensions), the storage overhead introduced by MaskGIT is substantially greater than that of TiTok. However, we would like to emphasize that in practical applications, our primary concern is the storage and transmission overhead of the images themselves. Our goal is to minimize image storage costs while preserving image quality as much as possible. Under the same bpp, our method achieves better recognition performance and image quality. Additionally, we have also explored approaches to reduce the model storage overhead. We explored a routing-based inference: only the codebook group selected by the router needs to be loaded, rather than searching with the minimum error across all codebooks. With this improvement, both inference latency and storage overhead are substantially alleviated, while the performance remains competitive with the baselines. There are also several engineering approaches to mitigate storage overhead. For example, we suggest performing quantization on the encoding side so that both the features and the codebook are matched within the int8 range. After obtaining the index sequence, the original fp32 codebook can be used on the cloud side for reconstruction and decoding.

Table: Inference latency and memory overhead statistics. For ours method, 'NN' and 'CR' refer to Nearest-Neighbor search and Codebook Routing search respectively.

Table: Quantitative comparison between different codebook selection policies during inference.

Regarding the comparison of training resources, the original paper reports that training TiTok-S requires 32 A100-40G GPUs for 50 hours. In contrast, our method primarily focuses on training the codebook, while the encoder is directly adopted from pretrained weights and the decoder is correspondingly fine-tuned. As a result, our training cost is 8 V100 GPUs for nearly 40 hours. We hope this clarification helps you better understand the differences in training resource requirements.

Answer (c)

Our proposed token-specific strategy can be easily adapted to images from other domains as well. To better evaluate the generalizability of our method, we also conducted experiments on the larger and more diverse ImageNet-1K dataset, with the results presented in the table below. As shown, our approach achieves significant improvements over the original baseline across various metrics. For example, the FID score is reduced from 3.91 to 2.27, and the LPIPS score is improved from 0.3889 to 0.3564. These results demonstrate the necessity and effectiveness of learning a separate codebook for each token.

Table: Evaluation of reconstruction performance on the ImageNet-1K 256 × 256.

As you mentioned, our method relies on the performance of the pre-trained autoencoder baseline, but it can be conveniently integrated into any existing codebook-based image compression framework. Here, we present updated results based on the IBQ baseline published at ICCV 2025 [1], where our approach also demonstrates its effectiveness, as shown in the table below.

Table: Evaluation on state-of-the-art IBQ.

The goal of image compression is to balance the trade-off between compression rate and distortion, aiming to minimize storage costs while preserving as much information from the original image as possible. In practical applications, we recommend deploying a lightweight encoder on edge devices to obtain high-quality compressed representations at lower bpp using our method, which can then be decoded or further processed by a decoder deployed in the cloud. Since our approach remains effective even at extremely low bpp (<0.01), it can significantly reduce the cost of image transmission and storage.

Reference: [1] Shi F, Luo Z, Ge Y, et al. Taming scalable visual tokenizer for autoregressive image generation. ICCV 2025.