We thank the reviewer for the feedback. We respond to the concerns and questions below:

[Q1] (...) how do the prediction risk and the estimation risk behave in Figure 1 and 2 respectively as $n$ and $p$ increase? In other words can we infer any pattern from the results shown in Theorem 3.4 and 3.5?

• Please see [General Response] and Fig S1 in the attached PDF. We test a wide range of $(n,p)$ pairs $((100,200),(100,400),(100,1k),(500,1k),(500,2k),(500,5k),(500,50k),(500,100k),(10k,150k))$ where $1k=1000$.
• Fig S1 (first three rows) shows that, even if the values of $n$ and $p$ change, the results are almost identical if $\gamma=p/n$ remains the same.
• This is an interesting result and easily predictable since $\gamma$ affects the values of $\mathbb{E}_X[\text{Tr}((X^\top X)^+\Sigma)]$ and $\mathbb{E}_X[\text{Tr}((X^\top X)^+)]$. For example, if $\Sigma=I$, then $\mathbb{E}_X[\text{Tr}((X^\top X)^+\Sigma)]=\mathbb{E}_X[\text{Tr}((X^\top X)^+)]\rightarrow\frac{1}{\gamma-1}$ in the limit of $n,p\rightarrow\infty,p/n\rightarrow\gamma$. So for a pair of sufficient large $n$ and $p$, only $\gamma$ determines the level sets. For a discussion of anisotropic $\Sigma$, please refer to Remark 4.9 below Cor 4.8 (Line 273-283).

[Q2] (...) Why the dependent $x_i$'s are not considered?

• We can allow for dependent $x_i$'s. The remark we made such that "each $x_i$ has a positive definite covariance matrix and is independent of each other" is just a sufficient condition for "Assumption 4.1. rank($X$) $= n$ almost everywhere". It is possible to satisfy Assumption 4.1 even if $x_i$'s are dependent and more importantly, without the rank assumption, the numerator in the equation below Line 243 becomes $p-\text{rank}(X)$, which makes the RHS $=r_\Sigma^2\frac{p-\text{rank}(X)}{p}$.

[Q3] (...) Are the authors assuming $\beta$ to be random. What does this mean?

• Yes, we are assuming that $\beta$ to be random. We make the random $\beta$ assumption (Assumption 4.2) in order to obtain an exact closed-form finite-sample expression for the prediction risk in Corollary 4.3. This type of assumption has been used before in the literature [e.g., 23, 20, 7] after the influential work by Dobriban and Wager (2018, Annals of Statistics). Although it may be less natural than the fixed $\beta$ assumption, it is helpful to obtain a clean insight into the problem.

[W1, W3, Q4] There are some notational discrepancies and theoretical inconsistencies. Some notations such as $a(X)$ and $b$ (...) have been clarified later in the appendix. (...) What is $S$ in equation (4)? (...)

• Thank you for pointing that out. We moved the (full) proof of Thm 3.4 in which all $S=\Sigma^{1/2},a(X)=\lambda((X^\top X)^+ \Sigma),b=\lambda(\Omega)$ are defined to Appendix because of the page limit. This may cause some notational discrepancies. We will move the definitions to the main part and revise our manuscript accordingly.

[Q5] In equation (5) if $p\gg n$, does this mean the bias go away?

• No, the bias does not go away. If $p\gg n$, i.e. $\gamma\gg 1$, then the bias is $[\text{Bias}(\hat\beta\mid X)]^2=r^2\_\Sigma\frac{p-n}{p}\approx r^2_\Sigma>0$ which is 1 in Fig 4 (Left).

[Q6] In corollary 4.8, I thought the double asymptotics on $n$ and $p$ have already been used. Then what is the limit wrt $n$ mean in the second term on the right hand side?

• The second term you refer to is from the expected variance. As shown in our main Thm 3.4, we decompose the expected variance into two parts (i) $\mathbb{E}\_X[\text{Tr}((X^\top X)^+\Sigma)]$ and (ii) $\frac{\text{Tr}(\Omega)}{n}$, i.e., $\mathbb{E}_X[\text{Var}(\hat\beta\mid X)]=\text{(i)}\times\text{(ii)}$ Here, we apply the double asymptotics on $n$ and $p$ to each part (the limit of a product is the product of the limits), i.e., $\lim\_{n,p\rightarrow \infty, p/n\rightarrow \gamma} \text{(i)}=s^*$ and $\lim\_{n,p\rightarrow \infty, p/n\rightarrow \gamma} \text{(ii)}=\lim\_{n\rightarrow\infty}\frac{\text{Tr}(\Omega)}{n}$ since (ii) does not depend on $p$ (and $\gamma$).

[W2] Some remarks following theorem 3.4 and 3.5 where the design matrix has a known distribution say Gaussian would have been useful examples to get insight on the results proved in the theorems.

• This is a great suggestion. The design matrix plays a role in $\mathbb{E}_X[\text{Tr}((X^\top X)^+\Sigma)]$ (Thm 3.4) and in $\mathbb{E}_X[\text{Tr}(\Lambda^+)]/p$ (Thm 3.5).
• First, in Thm 3.5, $\mathbb{E}_X[\text{Tr}(\Lambda^+)]/p$ in the limit is $\Theta(\frac{1}{\gamma-1})$ which decreases to $0$ as $\gamma\rightarrow\infty$ and increases to $\infty$ as $\gamma\searrow 1$. This is because $\mathbb{E}_X[\text{Tr}(\Lambda^+)]/p\rightarrow s^\ast$ in the limit (Thm 4.7). Thus, for a sufficiently large $n$ and $p$, we have $\mathbb{E}\_X[\text{Tr}(\Lambda^+)]/p\approx s^\ast$. Fig 4 (Right) empirically validates this approximation for not very large $n$ and $p$ ($n=50$ and $p\in [50,5000]$). And $s^*=\Theta(\frac{1}{\gamma-1})$ (cf. $s^*\_{\text{iso}}=\frac{1}{\gamma-1}$). This approximation depends on the degree of anisotropy of $\Sigma$ as discussed in eq (7).
• Second, it is not straightforward for Thm 3.4. Thus, to get some insights, we set $n=1$ and then $\mathbb{E}_X[\text{Tr}((X^\top X)^+\Sigma)]=\mathbb{E}_x[\text{Tr}((xx^\top)^+\Sigma)]=\mathbb{E}_x\Big[\frac{x^\top\Sigma x}{\\|x\\|^4}\Big].$ The numerator $x^\top\Sigma x$ has an expectation of $\text{Tr}(\Sigma^2)=\Theta(p)$, and the denominator $\\|x\\|^4$ ($x\sim\mathcal{N}(0,\Sigma)$) has an expectation of $2\text{Tr}(\Sigma^2)+\text{Tr}(\Sigma)^2=\Theta(p^2)$ which increases faster than that of the numerator as $p\rightarrow\infty$. Furthermore, if $\Sigma=I$ and $p>2$, then $\mathbb{E}\_x[1/\\|x\\|^2]=1/(p-2)\rightarrow 0$ as $p\rightarrow\infty$. Fig 4 (Left) empirically validates that the variance is small for a large $\gamma$.

We thank the reviewer for the feedback. We respond to the question below:

On large-scale validation

• Please see our top-level comment [General Response] and Fig S1 in the PDF attached to it.
• We additionally tested a wide range of $(n,p)$ pairs including $(500,5\text{k})(500,50\text{k}),(500,100\text{k}),(10\text{k},150\text{k})$ where $1\text{k}=1000$.
• Fig S1 (last row) shows that our theory ($y$-axis) matches the expected variance ($x$-axis) for a high-dimensional $x_i\in\mathbb{R}^p$ for $p=50\text{k},100\text{k},150\text{k}$. Note that CIFAR-10 and ImageNet have $p\approx 3\text{k}$ and $p\approx 150\text{k}$ dimensional data, respectively.
• Fig S1 (first three rows) shows that, even if the values of $n$ and $p$ change, the results are almost identical if the ratio $\gamma=p/n$ remains the same.
• This is an interesting result and easily predictable since $\gamma$ affects the values of $\mathbb{E}_X[\text{Tr}((X^\top X)^+\Sigma)]$ and $\mathbb{E}_X[\text{Tr}((X^\top X)^+)]$. For example, if $\Sigma=I$, then $\mathbb{E}_X[\text{Tr}((X^\top X)^+\Sigma)]=\mathbb{E}_X[\text{Tr}((X^\top X)^+)]\rightarrow \frac{1}{\gamma-1}$ in the limit of $n,p\rightarrow\infty,p/n\rightarrow\gamma$. So for a pair of sufficient large $n$ and $p$, only the ratio $\gamma$ determines the level sets. For a discussion of anisotropic $\Sigma$, please refer to Remark 4.9 below Corollary 4.8 (Line 273-283).

We thank the reviewer for the feedback and comments. We also believe that it is important to understand the real-world regression challenges with even more intricate patterns of error dependence. Even though it is extremely difficult to fully grasp the complexity of real-world problems, the paper aims to provide a relatively general theoretical analysis by relaxing some restrictive assumptions. We again emphasize our contributions: we relax the previous assumptions as follows:

We respond to the concerns and questions below:

On the assumption "$\varepsilon$ is independent of $X$"

• This is a common assumption in most existing studies.
• We can further relax the independence assumption. Specifically, $\Omega(X) :=\mathbb{E}[\varepsilon\varepsilon^\top \mid X]$ may depend on $X$. Then, the variance is $\text{Var}_\Sigma(\hat\beta\mid X)=\text{Tr}(X^\dagger\Omega(X)X^{\dagger\top}\Sigma)=a(X)^\top\Gamma(X)b(X)$ where $a(X):=\lambda((X^\top X)^\dagger\Sigma)$, $b(X):=\lambda(\Omega(X))$, and $\lambda(A)$ is a vector with its $i$-th element $\lambda_i(A)$ as the $i$-th largest eigenvalue of $A$.
• Therefore, with a weaker assumption "$\lambda(\Omega(X))=\lambda(\Omega(OX))$ for any orthogonal matrix $O$", we can still obtain a similar conclusion: $\mathbb{E}\_X[\text{Var}_\Sigma(\hat\beta\mid X)]=\mathbb{E}\_X[a(X)^\top\Gamma(X)b(X)]\overset{\text{Lemma} 3.3}{=}\mathbb{E}\_X[\mathbb{E}\_{O}[a(OX)^\top\Gamma(OX)b(OX)]]=\mathbb{E}\_X[a(X)^\top\mathbb{E}\_{O}[\Gamma(OX)]b(X)]$ $=\mathbb{E}\_X\Big[a(X)^\top\frac1n Jb(X)\Big]=\mathbb{E}\_X\Big[\frac1n\sum\_{i,j}a\_i(X) b\_j(X)\Big]=\mathbb{E}\_X\Big[\frac1n\text{Tr}((X^\top X)^\dagger\Sigma)\text{Tr}(\Omega(X))\Big].$ Here, $J$ is the all-ones matrix (see the proof of Theorem 3.4 for the details).
• Even without this assumption, using the matrix inequality $\Omega(X) \preceq \Omega^\ast := \sup_X ||\Omega(X)|| I_n$, we can obtain an inequality $\mathbb{E}_X[\text{Var}(\hat\beta\mid X)]\leq \frac{\text{Tr}(\Omega^\ast)}{n}\mathbb{E}_X[\text{Tr}((X^\top X)^{\dagger}\Sigma)].$

On the left-spherical symmetry assumption

• We believe that the left-spherical symmetry assumption is restrictive but not overly restrictive because it can be strictly weaker than the usual assumption $x_i\sim_{iid}\mathcal{N}(0,\Sigma)$. For example, $x_i$'s can be i.i.d. features from a mixture of centered Gaussian distributions.

On the assumption $\text{rank}(X)=n$ almost everywhere

• If $x_i$ is independent of each other and has a positive definite covariance matrix (e.g., $x_i\sim_{iid}\mathcal{N}(0,\Sigma)$ and $\Sigma\succ 0$), then $\text{rank}(X)$ is $n$ almost everywhere. Thus, we believe that the rank $n$ assumption is not overly restrictive.
• Moreover, this assumption is only made for the convenience of our asymptotic analysis. Even without the assumption we can obtain a similar result with $\text{rank}(X)$ instead of $n$.
• Without the rank $n$ assumption, the numerator in the equation below Line 243 becomes $p-\text{rank}(X)$, which makes the RHS $=r_\Sigma^2\frac{p-\text{rank}(X)}{p}$.

We thank the reviewer for the feedback. We respond to the concerns below:

On the technical contributions

• We have a concise proof. A compact and special technique made it possible.

• The main technical difficulty is that we generally cannot directly factor out $\Omega$ from $\text{Tr}(X^\dagger\Omega X^{\dagger\top}\Sigma)$.

• In the isotropic error case $\Omega=\omega^2 I_n$, we can easily obtain ($\omega^2$ out of $\text{Tr}$) $\text{Var}_\Sigma(\hat\beta\mid X)=\text{Tr}(X^\dagger\Omega X^{\dagger\top}\Sigma)=\omega^2\text{Tr}(X^\dagger X^{\dagger\top}\Sigma)=\omega^2\text{Tr}((X^\top X)^{\dagger}\Sigma).$

• Under general error assumption (e.g., anisotropic error), however, this is not feasible. We would like to emphasize that, to address this technical difficulty, we compute the "expected" variance ($\mathbb{E}\_X[\cdot ]$ over $X$) to apply Lemma 3.3 which is technically novel: $\mathbb{E}\_X[\text{Var}_\Sigma(\hat\beta\mid X)]=\mathbb{E}\_X[\text{Tr}(X^\dagger\Omega X^{\dagger\top}\Sigma)]\overset{\text{Thm} 3.4}{=}\frac{\text{Tr}(\Omega)}{n}\mathbb{E}\_X[\text{Tr}((X^\top X)^{\dagger}\Sigma)].$

• We again emphasize our contributions: we relax the previous assumptions as follows:

I am happy with the comments author made in their rebuttal. I think they have largely addressed most of the queries of the reviewers. I have increased my score to 6.

One issue which I am still not comfortable with is the random $\beta$ assumption in assumption 4.2. I agree to the literature Dorriban and Wager (AoS, 2018) authors suggested but what I am uncomfortable with is the fact that we are doing OLS to get $\hat{\beta}$ but to obtain bias of the prediction risk we are making the assumption as if $\beta$ has a prior distribution.

Is it not possible to make some other assumption on $\beta$ rather than the distributional assumption and work in the fixed $\beta$ setting?

Many thanks to Reviewer pxrE for raising the score to 6 and providing further comments. In principle, it would be possible to work in the fixed $\beta$ setting. For example, if we focus on the prediction risk, equation (4) in the paper (the displayed equation between lines 236 and 237) is still valid without the random $\beta$ assumption. Then, we can interpret the bias result similarly with $S\beta (S\beta)^\top$, where $S = \Sigma^{1/2}$, instead of $r_\Sigma^2$ defined in Assumption 4.2. However, when we consider asymptotic results by letting $n$ and $p$ converge to infinity, we need to come up with a suitable assumption such that $S\beta (S\beta)^\top$ could be treated fixed or we need to find a suitable limit. To simplify our discussion and provide a clean result, we followed Dorriban and Wager (AoS, 2018) that provides a heuristic approach for an average-case analysis of dense parameters. We hope that you will find our short-cut approach more acceptable.

Dear Reviewers,

Thank you for the valuable comments during the discussion period. We would like to summarize our contributions again:

The majority of the previous analyses have been limited to an unrealistic regression error structure, assuming iid errors with common variance. We explore more general regression error assumptions, highlighting the benefits of overparameterization in a more realistic setting that allows for clustered or serial dependence. Our findings suggest that the benefits of overparameterization can extend to time series, panel and grouped data.

Notably, we establish that the estimation difficulties associated with the variance components can be summarized through $\text{Tr}(\Omega)$ and thus they are not affected by the degrees of dependence across observations. $\mathbb{E}\_X[\text{Var}_\Sigma(\hat\beta\mid X)]=\mathbb{E}\_X[\text{Tr}(X^\dagger\Omega X^{\dagger\top}\Sigma)]\overset{\text{Thm} 3.4}{=}\frac{\text{Tr}(\Omega)}{n}\mathbb{E}\_X[\text{Tr}((X^\top X)^{\dagger}\Sigma)].$

Thanks again.