Towards Marginal Fairness Sliced Wasserstein Barycenter

We genuinely appreciate the reviewer's time and thoughtful feedback. We would like to elaborate on the discussion as follows:

Q2. Can the authors elaborate on the significance of MFSWB? The description in the current version is somewhat high-level.

A2.Thank you for your question on the impact of the MFSWB. We would like to elaborate on the significance of MFSWB as follow. MFSWB generalizes the notion of a center to the space of probability measures. In particular, the conventional center of a set of points in a vector space corresponds to the center of the smallest circle enclosing all the points in the set. Similarly, MFSWB can be seen as the center of a set of points in the space of probability measures, defined using the SW distance geometry. This generalized notion of a center enables a broader application, such as generalizing k-center clustering to handle probability measures instead of vectors. Furthermore, as demonstrated in the paper, MFSWB can identify the center of 3D shapes (e.g., human brains, 3D objects), which can serve as prototypes for future tasks. For instance, the 3D prototype can be used to generate new 3D shapes via a deformation function. Finally, MFSWB can facilitate learning fair representations across classes by ensuring that the regions of high probability for each class distribution are approximately equal. This fair representation can implicitly enhance fairness in downstream tasks.

Q3. What does the symbol $\sharp$ in Eq.(1) denote?

A3. Thank you for pointing out. The symbol $\sharp$ denotes the pushforward operator. In particular, $\theta\sharp \mu$ denotes the pushfoward measure of $\mu$ through the function $f(x)=\theta^\top x$. We have added an explanation to the revision.

Q4. Can the authors provide a clearer description about the formulation in line 148?

A4In the formula, $[n]$ denotes the set $\{1,2,\ldots,n\}$, $\sigma_1:[n] \to [n]$ is the permutation function such that $x_{\sigma_1(1)} \leq x_{\sigma_1(2)} \leq \ldots \leq x_{\sigma_1(n)}$ or $x_{\sigma_1(1)} \geq x_{\sigma_1(2)} \geq \ldots \geq x_{\sigma_1(n)}$. Similarly, $\sigma_2:[n] \to [n]]$ is the permutation function such that $y_{\sigma_2(1)} \leq y_{\sigma_2(2)} \leq \ldots \leq y_{\sigma_2(n)}$ or $y_{\sigma_2(1)} \geq y_{\sigma_2(2)} \geq \ldots \geq y_{\sigma_2(n)}$, and $\sigma_2^{-1}$ is the argsort operator. The optimal transport map is constructed as $\sigma = \sigma_1 \circ \sigma_2^{-1}$. We added this to the revision of the paper.

Q5. What does the $x_i(y_i)$ in line 144 denote?

A5. In that line, $\{x_1,\ldots,x_n\}$ and $\{y_1,\ldots,y_n\}$ are the supports sets of two empirical measures $\mu =1/n\sum_{i=1}^n \delta_{x_i}$ and $\nu=1/n\sum_{i=1}^n \delta_{y_i}$

Q6. What does the $\pi$ in line 144 denote?

A6. We assume that you are asking for the notion $\phi$ in line 144, $\phi$ is the parameter of the barycenter as defined in line 128. In the fixed support barycenter case, $\phi$ is the weights of supports. In the free support barycenter case, $\phi$ is the supports.

Q7. What modifications to the methodology presented in this paper are necessary to replace SWD with GSWD [1] or ASWD [2]?

[1] Kolouri, Soheil, et al. "Generalized sliced wassertein distances." Advances in neural information processing systems 32 (2019).

[2] Chen, Xionglie, Yongxin Yang, and Yunpeng Li. "Augmented sliced Wassertein distances." arXiv preprint arXiv:2006.08812 (2020).

A7. Thank you for your insightful questions and your references. We added these references to the revision. GSWD and ASWD utilize different types of projections compared to SW. In particular, GSWD uses non-linear projections such as circular, polynomial, and so on, while ASWD utilizes an injective non-linear augmentation function followed by a linear projection. Replacing SWD by GSWD and ASWD results in different marginal fairness barycenter problems. This change leads to a geometric change in the barycenter, which is worth a careful investigation in our opinion. Moreover, the notion of a marginal fairness barycenter can adapt to any metric, such as Wasserstein, SWD, GSWD, and ASWD. With each choice of metric, an implied geometry is used to seek the barycenter. Since the marginal fairness barycenter is the main focus of the paper, we leave the investigation with other metrics to future works.

Please feel free to ask if you still have any other questions.

Q9. USWB first appears in Figure 1 and used in several places. But it is not defined.

A9. Thank you for pointing out, USWB stands for Uniform Sliced Wasserstein Barycenter, which sets marginal weights in Sliced Wasserstein Barycenter problem to be uniform. We added the explanation to the revision.

$\min_{\mu}\text{USWB}(\mu; \mu_{1:K}); \quad \text{USWB}(\mu;\mu_{1:K})= \frac{1}{K} \sum_{k=1}^K \text{SW}_p^p (\mu,\mu_k).$

Q10. I can understand the F-metric is significantly improved with the proposed solution. But for W-metric why it is also improved. Are there feasible explanation or insights? Can you provide an explanation or discussion on why the W-metric improves along with the F-metric? This would be valuable to better understand the proposed method.

A10. Thank you for your insightful question. It is worth noting that MFSWB has higher W-metric than USWB in the Gaussian cases in Section 4.1. For other applications, MFSWB does achieve lower USWB. The reason for this phenomenon is due to the chosen optimization procedure and the different landscapes of MFSWB and USWB. Therefore, the obtained barycenters might be only local optimums. At those optimums, MFSWB is better than USWB in both F-metric and W-metric. Nevertheless, we project that MFSWB would have lower F-metric than USWB and have higher W-metric than USWB at the global optimum. We would like to refer to the simple simulation of Gaussians in Section 4.1 as the ideal example. We will leave the investigation about optimization to future works.

Q11. What does USWB stand for in the paper?

A11. Thank you for your question, as mentioned in question 9, USWB stands for Uniform Sliced Wasserstein Barycenter i.e., the Sliced Wasserstein Barycenter problem where all marginals have equal weights. We added the explanation to the revision.

Q12. Can the method extended to settings with more than two images or point cloud?

A12. Thank you for your questions. The barycenter problem is defined with an arbitrary number of marginals i.e., $K>1$. Similarly, our proposed MFSWB is defined with $K>1$ marginals e.g., $K=2,3,...$. In Section 4.1., we consider $K=4$ marginals of Gaussians. In Section 4.4, we consider $10$ marginals where each marginal is a class distribution. The main reason we used two images and two point clouds in Section 4.2 and 4.3 is that averaging more than two images and point clouds might lead to unnatural color palettes and 3D shapes for human's perception due to the heterogeneity of all marginals. For example, an average of many 3D shapes of car can be just a 3D ball. Therefore, it requires more efforts to select marginals to obtain a visually interpretable barycenter as a 3D shape prototype. Since the paper focuses on the concept of MFSWB, we leave the careful investigation on the effect of varying the number of marginals to future works.

Please feel free to ask if you still have any other questions.

We sincerely appreciate the reviewer’s time and insightful comments. We would like to further develop the discussion as follows:

Q8. Most of the experiments conducted are simple or on simple dataset, such as Gaussian, Simple Color Hamonization of two images or point cloud of only two shapes, AE on simple MNIST dataset. I am wondering the scability and real application of the proposed method. For example, if more than two (e.g. >5) images or point clouds are adopted, what might be the results? Or are there more realistic potential appplications of the proposed method? If related work or potential applications are discussed, it would be better. For example, CelebA dataset results or Fairness generation [R1].

[R1] Fair generative modeling via weak supervision.

A8. Thank you for the reference. We have added the reference to the paper. We decided not to use the CelebA dataset because converting its labels to the desired format requires additional effort. Due to time constraints, we plan to leave experiments with the CelebA dataset for future work. Instead, we opted for the STL10 dataset, which shares the same image dimensions as CelebA. To further investigate the scalability of the proposed method, we have recently experiment with CIFAR10 ($d = 32 \times 32 \times 3$) and STL10 ($d = 64 \times 64 \times 3$) for the Sliced Autoencoder experiments. Below are the tables summarizing CIFAR10 and STL10 experiments. Overall, compared to the baselines, the proposed methods achieve greater fairness notion and bring the generated images closer to the dataset distribution in both latent and image spaces, though this comes at the expense of reduced image reconstruction quality. It is worth noting that we use FID score to measure F-metric and W-metric on images space of CIFAR10 and STL10. We added the details of these experiments into the revised version of our paper (Table 5 and Table 6 in page 28).

Table 1: Comparison of methods with $\kappa_2 =0.5$ on CIFAR10 after 500 epochs.

Table 2: Comparison of methods with $\kappa_2 =0.5$ on STL10 after 500 epochs.

The authors' rebuttal comprehensively addressed my questions and concerns, with additional experimental results.
I would like to increase my score from 6 to 8.

We would like to thank the reviewer for the time and constructive feedback. We would like to extend the discussion as follow.

Q1. Did you have a typo in line 506? Should $P_{f(X_k)}(x)$ be equal to $\frac{1}{M}\sum_{i=1}^M \delta_{f(X_{k,i})}$?

A1. Thank you very much for pointing out the typo. It should be $\frac{1}{M}\sum_{i=1}^M \delta_{f_\phi(X_{k,i})}$. We fixed all related typos in the revision.

Please feel free to ask if you still have any other questions.