SketchFill: Sketch-Guided Code Generation for Imputing Derived Missing Values

• In particular, there is no study done to assess each of the components discussed in Fig 3. It would be good to understand how each of the components as they are built add to improved performance of the overall system.

• The lit. review seems to compare to some prior imputation approaches but evaluation is done withKNN as a baseline. It would be good to compare to other statistical approaches like mean, conditional mean, Unconditional mean imputation, Conditional mean imputation (Buck's method), Stochastic regression imputation and Maximum likelihood-based imputation. A table showing the characteristics of each techniques, their advantages and disadvantages would be useful to the end reader.

• The paper could be strengthened by sharing the python code generated for their different dataset. It is safe to assume that the python code is not generating the formulae for the different datasets especially BMI and Bajaj ?

• The evaluation is performed on datasets that are fairly small and not reflecting real world challenges. The largest dataset has total tuples of ~3.6K and columns of size 13. What is the computational complexity of the approach ? There also several datasets available on the kaggle framework where these approaches could be benchmarked in a thorough manner.

• The presentation of results were confusing; It was not clear how accuracy was computed ? Was it 1-MAE ? Additional insights from the Table (results) were missing.

• The paper can be strengthened by adding key details about the LLMs themselves in particular architectures, use of ICL, fine tuning and others ..

• The sketch based solution was template-driven for this solution; it was not clear how it would scale for any new large dataset with missing values.

1. The related work section can be improved by expanding the discussion of LLM-based methods in greater detail. While the current section discusses CoT prompting, code generation, and MetaGPT, it lacks an analysis of how these approaches address the specific challenges of DMVI. For instance, discussing scenarios where CoT prompting outperforms code generation, or vice versa, in terms of handling imputation tasks would offer more clarity on the relative strengths and weaknesses of each method. This could also include the types of missing values and data structures each method is most suited for. The section would also benefit from addressing any challenges that LLM-based methods still face in this domain, such as token length restrictions or susceptibility to producing hallucinated outputs in complex imputation tasks.

2. The novelty of the proposed method is somewhat outlined in sections 1-2, but the differentiation from existing methods could be made clearer. While the paper introduces SketchFill as a sketch-guided approach for DMVI using a combination of Meta-Sketch and Domain-Sketch for generating imputation formulas, the uniqueness of this approach could be made clearer by contrasting it more explicitly with the limitations of existing LLM-based methods. For example, the authors could highlight that SketchFill offers a structured way to guide LLMs through multi-step reasoning processes by explicitly incorporating domain-specific knowledge, whereas existing methods struggle with tasks requiring complex derivation formulas or reasoning across multiple data points.

3. The paper lacks analysis of the Meta-Sketch's impact on performance - it's unclear how different Meta-Sketch designs affect SketchFill's ability to discover correct formulas. An ablation study comparing different Meta-Sketch structures, or experiments showing how variations in Meta-Sketch detail influence imputation accuracy, would help to understand which sketch components are needed.

4. The paper lacks analysis of SketchFill's performance in scenarios where a dataset contains both derived and non-derived missing values. The current evaluation focuses exclusively on formula-based missing values. The authors could extend their experiments to show how SketchFill compares with traditional imputation methods for non-derived values, or demonstrate how their Meta-Sketch system could be adapted to detect and handle different types of missing values within the same dataset.

• In my view, the proposed method in this paper is overly focused on engineering aspects, and I struggle to identify a clear scientific contribution.
• The problem addressed in this paper does not align with the formulation of missing data. The authors focus on predicting values in a single target column using other columns as predictors, which essentially frames this as a very very very standard supervised learning task rather than a missing data problem. Calling this as a missing data issue solely because some values are absent can be misleading.
• The framework proposed in this paper essentially mirrors the standard procedures of supervised learning. Several steps in the Meta-Sketch, like S1 and S2, are very standard preprossing procedures, while in S3, common regressors such as linear models or neural networks are typically employed to predict the target variable, just as done here. S4 involves predicting outcomes, which aligns directly with conventional practices, and finally, validation is conducted in steps S5 and S6 to assess model performance. Overall, this approach closely resembles a standard supervised learning task.
• The core of this technique lies in code generation, which is used to approximate an unknown function. Typically, optimizers are applied to refine the model; however, this paper relies entirely on the LLM, including the "reflector" component, to optimize this function. There is no guarantee that this approach will yield an improved solution, as relying on the LLM as a definitive guide for optimization is werid and raises concerns.
• The effectiveness of the proposed method comes from its reliance on well-known criteria, such as BMI. For instance, when prompted to calculate BMI, the LLM can readily provide the equation based on the input variables. This effectively reduces the problem to a retrieval task. However, this raises the question of how the method would perform when tasked with approximating a more complex function, such as $y = 238 \cdot x^2 \cdot 29 \cdot z + \exp(k) \cdot 22.33455454$. Would this approach still be viable in such cases?
• Perhaps conferences focused on LLM, like ACL, would be better place for this paper, as they are likely to have reviewers better equipped to evaluate its contributions to LLM applications. However, from the perspective of a ML researcher, I still struggle to identify any scientific contribution in this work.

The proposed methodology does not fundamentally address or propose a new principle for table data generation, nor does it solve its intrinsic challenges, which means the learning of distributions from data. Instead, the approach is primarily based on Conditional Mean Imputation using masking techniques, with a strategy that validates the effectiveness of this method before extending it to other approaches. The proposed method is closer to an engineering ensemble approach of previously developed data synthesis methods.

Therefore, the observed superior performance likely results from the method’s ability to effectively impute missing values when there are physical constraints among column data. However, this situation is likely to be limited in practical tabular data applications, as missing values in data with such constraints are typically expected to arise only in specific cases, such as code errors, in most databases.