LDINet: Latent Decomposition and Interpolation for Single Image FMO Deblatting

Thank you for the valuable comments. The following are our responses to the comments.

Q1: why the weights in the formula at the bottom of page 4 (which has no formula label) are set this way and why they serve the purpose of "fully leverage the information from both the two neighboring parts.

A1: The weights in this formula are set in such way to guarantee that $Q_{\tau}$ is identity to $P_{\tau}$ when $\tau = t$ or $\tau=t’$. In the case that $\tau \in (t, t’)$, a larger weight is given to the transformed result from the closer latent part and a smaller one to the other transformed result. Since the target latent frame $Q_{\tau}$ is a combination of two transformed results from two neighboring latent parts, it would fully leverage the information from both sides.

Q2: The training cost of the proposed method compared to DeFMO.

A2: For the training dataset used in Section 3, it costs about 1.5 days and 2.5 days for the training of our method (60 epochs) and DeFMO (50 epochs) respectively on 8 Nvidia A5000 GPUs.

Q3: The motivation for choosing the direction with a smaller relative error rate when determining the trajectory direction.

A3: For the direction selection, we actually follow DeFMO to leverage such a strategy for determining the appropriate trajectory direction. The difference is that our method only calculates the relative error rate on the masks. Since in the generated results, only the masked part of the appearance is what we need, it can be considered that the reconstruction of the mask needs to be focused on.

Q4: The contributions are somehow incremental. The encoder-decoder framework and most of the training losses mentioned in this article have already been proposed in DeFMO.

A4: Although our method is built with a similar encoder-decoder framework and loss functions, we want to emphasize that the novelty of our method mainly lies in the decomposition-interpolation paradigm, which allows naturally introducing the required time indexes via interpolation operation. And the well-designed AffNet helps perform feature alignment between different frames during the interpolation. Further, the splitting scheme of the scalar-like and gradient-like feature maps could further improve the alignment effect as the processing way of scalar and gradient fields under the affine coordinate transformations is different. This helps generate a more accurate latent frame at the target time index for decoding. These designs can well distinguish our method from DeFMO. We will add such explanations in the revision.

Q5: More ablation studies and experiments for the DIB module.

A5: We have provided an ablation study to compare the performance between settings of linear interpolation and affine interpolation in the DIB module. Please refer to the common response part. In addition, we also provide the visualization of their output images in the supplemental material for better understanding.

Q6: Some doubts about the randomness on the performance of the proposed method.

A6: Please refer to the common responses part.

Thank you for the valuable comments. The following are our responses to the comments.

Q1: Are AffNet and separating gradient fields really necessary?

A1: Please refer to the common responses part.

Q2: It would be helpful to add SfB results if sub-frame silhouettes are used from the proposed method as input.

A2: We have provided the results for SfB with silhouettes generated by LDINet as follows:

Since time is limited, we only report the 8-step results with two prototypes and set the learning rate to 0.03 for stable optimization in SfB. And the std is provided in parentheses for 3 times experiments. We will add this comparison into the revision.

Q3: An ablation where the feature maps are simply interpolated by linear interpolation.

A3: Please refer to the common response part.

Q4: On page 4, it says that $P’’(x)$ is equal to the gradient of the scalar field $S(x)$, but the scalar field was already defined as $P’(x)$. Is $P’(x) = S(x)$?

A4: $P’(x)$ is not $S(x)$. Due to the separating of scalar-like and gradient-like feature maps, $P’’(x)$ and $P’(x)$ are generated from different channels of the same latent part. Thus, though $S(x)$ is the scalar field to produce $P’’(x)$, there actually exists differences between $P’(x)$ and $S(x)$.

Q5: Loss changes w.r.t. DeFMO.

A5: Please refer to the common responses part.

Q6: For the Equation 6, (1) about the sum term that denotes the number of pixels within the FMO blur, and (2) why is this normalization different for the first (without $>0$) and the second term (with $>0$)?

A6: Since $I$ denotes the blur image which is the average of a series of sharp images at the given time indexes within the exposure time, the formula reviewer provided could count the same pixel multiple times because of its appearance in masks of different time indexes. In our formula, we use a pixel-wise indicator function $[\cdot >0]$ to set each pixel that appears in any mask to 1 which avoids the multiple-counting.

On the other hand, $>0$ is part of the pixel-wise indicator function $[\cdot >0]$. The first normalization term is without the function because the mask $M_\tau$ is a binary map whose value is either 0 or 1 for different pixels and thus it could be directly summed. While the second normalization needs the function to convert the summation over time indexes $\sum_{\tau} M_{\tau}$ to a binary map to further count the size of the blurred area.

Q7: The Arch. (bi-branched) ablation in Table 2 and the dependence of pretraining.

A7: In the decoder, we separate the estimation of the appearance and the mask with two different branches as in Figure 1. In particular, this separation happens after the first two Resblocks of the decoder. Otherwise, they are estimated with one shared branch setup. We will add more explanations to the revision. In addition, for the dependence of pretraining, we provide the following ablation study to show its advantage:

Since time is limited, we only report the one-run results for w/o pretraining here and we will further complement the results.

Q8: About the randomness of the experimental results.

A8: Please refer to the common responses part.

Q9: About the trend change in Table 3.

A9: For “In Table 3, it is seen that the score goes down if the number of parts is set to 20.”, this is expected as the final results would rely more on the quality of the direct output of the encoder when the number of parts increases and the affine transformation would not produce effects as the space for interpolation is gradually reduced.

Q10: About the typos and some expressions.

A10: Thanks a lot, and we will correct the typos and revise our paper more carefully. For the description of "Conventional methods mostly recover a clear image at the median of the motion", such expression is only to illustrate the common problem setup in the existing literatures. We will revise accordingly to avoid confusion.

Thank you for the valuable comments. The following are our responses to the comments.

Q1: Why we need to further decompose the convolution as a linear combination of summation and series of directional derivatives.

A1: The main reason is to help improve the alignment effect by the affine transformation. In particular, by decomposing the convolution as a linear combination of summation and series of directional derivatives, the resulting feature maps are correspondingly split into scalar-like and gradient-like feature maps. Further, the scalar-like and gradient-like feature maps have different processing ways under the affine coordinate transformation. Thus, different ways of transformation operations are enforced on these two kinds of feature maps to fully exploit their particular characteristics, and thus better alignment effect could be achieved, which helps improve the embedding quality for any time index and obtain better overall performance.

Q2: About the affine transformation. The paper mainly describes what has been implemented to approximate the affine transformation via prediction of two affine transformations. Would this be redundant as one transformation should be the inverse of the other?

A2: We predict the two affine transformations mainly for the consideration of the numerical stability of the model. Empirically, using the inverse operation directly has the risk of numerical instability, especially in the first several epochs in the training. Such a design of modeling the transformation from both directions also provides some flexibility when either one happens to be not precise.

Q3: The Equation 10 is not correct.

A3: Thanks for pointing this out. We will correct it by using the Frobenius Norm for computing the relative distance between two matrices.

Q4: There is no guarantee that the predicted affine transformation is invertible.

A4: For the invertibility, based on the experimental statistics, the id loss in Equation 10 is less than 1e-4 during the training process, which empirically shows that the predicted affine transformation approximates to be invertible. We will add such analysis in the revision.

Q5: All the equations should be written properly.

A5: Thanks for pointing them out. We will revise the Equations 5, 6, 7, and 8 accordingly.

Q6: About the qualitative results.

A6: In the supplemental material, we have provided the compared results for the recovery of the appearance of the FMO in the form of videos.

Thank you for the valuable comments. The following are our responses to the comments.

Q1: The novelty is limited.

A1: Please refer to the common responses part.

Q2: Fail to provide evidence of why the proposed method works.

A2: Thanks for your suggestion. We would reorganize the paper and provide more required explanations for the important components of the proposed method as is pointed out. To further validate the effectiveness of our work, we would add more ablation/experimental studies. For example, (1) We would compare with the results by linear interpolation to show the benefits of the affine transformation in our interpolation method, (2) We would provide an ablation setting to show the importance of the pre-training, and (3) We would provide more qualitative results to show the benefits of the decomposition-interpolation paradigm.

Q3: The presentation of this paper is too poor to be fully understood. (1) More details about the difference between the proposed network and the prior work DeFMO. (2) More details about the implementations for scalar-like and gradient-like feature maps. (3) More details for the process from eq. 3 to eq. 5.

A3: We will revise the paper accordingly.

(1) The main difference between our method and DeFMO lies in the way of constructing latent embedding for different time indexes. DeFMO concatenates the target time index with shared latent feature maps as the corresponding embedding for the subsequent decoding. Instead, our method first decomposes the feature maps in latent space into several latent parts and then generates the required corresponding embedding for any time index via an interpolation method with affine transformation. This helps greatly improve the flexibility of generating diverse outputs for different time indexes. Further, in the decoder part, DeFMO generates the mask and appearance of the object with only one branch, while our method generates them with two separate branches as they have different value distributions. More details about the difference will be added in the revision.

(2) For the scalar-like and gradient-like feature maps, such design mainly considers the characteristics of convolution operation under the affine coordinate transformations to improve the interpolation quality. In particular, on the one hand, the convolution operation could be seen as a linear combination of summation and series of directional derivatives, where the directional derivatives are linear projections of the gradient fields. On the other hand, the processing way of scalar and gradient fields under the affine coordinate transformations is different. Thus, it is very beneficial to represent the feature maps output by the convolution operation as a combination of scalar-like and gradient-like feature maps. Such a well-designed splitting scheme helps improve the alignment effect by the affine transformation. In practice, our model learns the scalar-like and gradient-like feature maps with a pre-training stage. In the pre-training, we first obtain the latent parts for the original input image. Then we split the channels of latent parts into the scalar-like and gradient-like categories where each scalar-like feature map has a single channel and each gradient-like feature map has two channels as the two components of the gradient field, which are processed by an estimated affine transformation by AffNet. Further, the resulting feature maps are forced to be consistent with the corresponding ones for a distorted input version which is produced by applying a pre-set affine transformation to the FMO blur contained in the original input image. Meanwhile, in this process, the estimated affine transformation is also encouraged to be similar to the pre-set one. The related ablation study is provided in Table 4 to validate the importance of generating two kinds of feature maps instead of only one kind. More details will be added in the revision accordingly.

(3) In particular, Equation 3 and Equation 4 formulate the supposed transformations for the scalar-like and gradient-like feature maps, respectively. In Equation at the bottom of page 4, a function $\Phi$ is first defined to split the input feature map into the scalar-like and gradient-like feature maps, which are then processed with the transformations in Equation 3 and 4 correspondingly. The two respective results are concatenated as the output. Therefore, the Equation 3 and 4 provide the important component for the operation of the Equation at the bottom of page 4. For Figure 2, since it is defined that the latent feature map $P_t$ is the concatenation of the scalar part $P_t’$ and the gradient part $P_t’’$, only the process of the Equation at the bottom of page 4 is explicitly presented in Figure 2(a), where the function $\Phi$ takes as input ($P_t$, $A_{\tau\rightarrow t}$) and ($P_{t’}$, $A_{\tau\rightarrow t’}$) for generating $Q_{\tau}$. More explanations will be added in the revision.

Dear reviewers, Could you please read the authors' responses and give your feed back? The period of Author-Reviewer Discussions is Fri, Nov 10 – Wed, Nov 22. Many thanks, AC