Automatic Outlier Rectification via Optimal Transport

A framework for regression tasks with outliers (see robust statistics) is introduced, based on an optimal transport cost function. A novel concave cost function directly facilitates the rectification of outliers. Mathematical proofs are given and evaluation is performed on two simpler and one real world task with state-of-the-art results.

The paper presents a new framework for outlier detection using optimal transport with a concave cost function. Unlike traditional methods that separate outlier detection and estimation, this approach integrates both into a single optimization process, improving efficiency. The concave cost function helps accurately identify outliers. The method outperforms traditional approaches in simulations and empirical analyses for tasks like mean estimation, least absolute regression, and fitting option implied volatility surfaces.

The paper presents a novel method for outlier rectification in robust statistics. In particular, taking inspiration from distributionally robust optimization (DRO) methods, this paper proposes to extend the formulation to the field of robust statistics. To this end, a rectification set is constructed using optimal transport distances with concave cost functions. The authors have proven theoretically that the reformulation is eventually equivalent to an adaptive quantile (regression) estimator with the quantile controlled by the budget parameter for mean estimation and least absolute regression.

Experiments are conducted to demonstrate the effectiveness of the proposed method on various tasks, including mean estimation, least absolute regression, and option implied volatility surface estimation.

This paper is, essentially, about robust model fitting. In this paper, it is proposed to optimize an optimal transport problem. As loss function, the authors choose concave functions ||z-z‘||^r with r \in (0,1). The idea behind choosing this shape of functions is that it allows to move outliers farther due to decreasing increases of transportation costs. The authors provide a proof that outliers can be identified easily by solving a linear program. Alternatively, they propose to use the quick select algorithm. Experimental results of the proposed algorithm are presented on option volatility prediction, a problem in finance.

A new approach for robust estimation is proposed based inspired by optimal transport. The general approach is introduced and then new estimators are derived for three particular cases : mean estimation, linear fitting and surface regression. The estimators are compared numerically with standard estimators on synthetic data for the first two cases and on financial datasets for surface regression. Many details, proofs and explanation are provided in the appendix which is 25 pages long.

We sincerely thank all reviewers for their positive feedback and appreciate the effort to review our work. We respond below to individual points.

fFC9

• Material, neural nets. While there is significant material in our work, we note that other papers at NeurIPS have large appendices as well. As our paper introduces a new estimation framework and theoretical results, we didn't have space to explore links to neural nets in depth. We are investigating it in future work.

i67o

• Flow of information. We respectfully disagree that “there is no flow of information… between outlier removal… and the gradient step.” Our min-min estimator is an alternating minimization procedure, where each step informs the next. The gradient step depends on outlier rectification, and information is carried over via parameter updates across iterations. This approach ensures more information flow between the substeps than a two-step method.

• Chin et al. We appreciate this addition. While both papers address contaminated data, there are fundamental differences in terms of goals (hypothesis generation vs predictive modeling), the conceptual problem, theory, and application domains. We view this work as complementary and will certainly discuss and cite it in our camera-ready version’s related work.

• Figures. We’ve included many robustness figures in the appendix showing performance for many delta and r values. Since knot points and lambda depend on these params, we refer to Figures 6, 8, 11, and 12 for a comprehensive view. If the reviewer has a specific case, we would be happy to add it.

CDCT

• Designs, Proposition 1. We appreciate the feedback on how we present how e.g. the transport cost function affects the algorithm. Specific suggestions would help us address this point. Pages 4-5 demonstrate how concave vs. convex cost functions affect the algorithm. Appendix F provides further explanation. Sensitivity analyses throughout illustrate additional details. We hope these sections suffice but welcome more feedback. For Prop 1, we have given sources for the proof in the text following. The reviewer can find the proof in the cited work Blanchet and Murthy 2019, Theorem 1 and Remark 1.

• Baselines. We value the feedback and opportunity to clarify. Our estimation approach can be applied to any regression algorithm, and we demonstrate this through diverse methods in sections 5.2.1 & 5.2.2, including a SOTA deep learning estimator. We disagree that the deep learning estimator lacks robustness, as it incorporates different regularizations that impart robustness. Our method aims to robustify regression estimators, and we believe we show this. We’ll move results for 5.1 to the main text, as suggested.

• Limitations. We respectfully disagree that no limitations are addressed. App. D.2 and D.3 cover limitations of our procedure. In our camera-ready version, we’ll add a summary of these to the main text and reference these appendices.

iniy

• Relationships. A full comparison is beyond scope, but we give an overview here and will include a version of this in our camera-ready version. M-estimators are a broad class of diverse extremum estimators containing robust and non-robust methods. Robust ones traditionally use specific loss functions to reduce outlier sensitivity. We believe our approach is more fundamental as it builds on top of a given general loss. The statistician has full control of the outlier modeling task (via specification of optimal transport theory using a concave transport cost) and the statistical task (via the chosen loss); also, our method gives the optimal rectifier (i.e. transporter). Our approach differs conceptually and demonstrates superior performance over the M-estimators in our experiments. We are open to additional M-estimators if suggested. RANSAC handles outliers by iteratively selecting random data subsets, fitting models, and evaluating models on inliers to reach a best model. In contrast, our method optimally selects the set of points for rectification using a single optimization procedure. RANSAC only identifies inliers, while our method does this and rectifies outliers. We believe our approach is more direct, potentially more computationally efficient. It's based on optimal transport theory rather than heuristic random sampling.

qw3a

• Concerns. We chose 10 to represent "fat finger" mistakes, such as decimal misplacements in finance, which have historically caused significant losses. We recognize the reviewer’s concern wrt proprietary code. Our method is used by industry partners with IP restrictions common in finance, but we've disclosed the algorithm, hyperparameters, and other details needed for replication. The deep learning data and baseline code are on GitHub.

• Questions. Q1: We use robust methods for mean estimation, e.g. the median and trimmed mean estimators. Notably, the trimmed mean uses the true corruption rate, a strong advantage. Despite this, our method, which doesn’t know any true parameters of the DGP, still outperforms this estimator. Q2: Our experiments for mean estimation and LAD regression are intentionally straightforward to make our novel framework easily understandable. Despite this, robust estimators like the trimmed mean and Huber perform poorly, indicating the difficulty of these problems and the improvement our estimator offers. Wrt deep learning, the choice of a multiplier of 10 is justified by the high variability in options prices, covering a wide range due to inherent price differences in options chains.

All

• Details/typos. References and misspellings are fixed. References to Problem 3 have been corrected to Problem 4. The delta in Problem 4 is in the ball R(P’_n) in Eq. 5 (page 4 line 152). “Budget” has been added to Figs 2 and 3.

We hope the reviewers are satisfied with our responses. If our responses have adequately addressed concerns, then we politely request an increased score to reflect this change, if appropriate. Thank you!

Hello, it has been about two days since we've posted our rebuttal, and we wanted to gently remind the reviewers to view it. We politely request that the reviewers either provide further feedback, or, if they agree that our response has addressed their concerns, to please increase their score. Thank you!

I find that the joint approach to solving fitting as well as outlier estimation proposed has potential to influence the robust statistics field. The paper's experimental validation could be further strengthened by studying broader class of outlier distributions. I am satisfied with the authors rebuttal and consider this to be a valuable paper to be accepted.