Properties of Least Squared Estimator

#econometrics #economics

Oh, Hyunzi. (email: wisdom302@naver.com)
Korea University, Graduate School of Economics.
2024 Spring, instructed by prof. Kim, Dukpa.

Main References

Kim, Dukpa. (2024). "Econometric Analysis" (2024 Spring) ECON 518, Department of Economics, Korea University.
Davidson and MacKinnon. (2021). "Econometric Theory and Methods", Oxford University Press, New York.

Fitted Values and Residuals

From the last chapter Geometry of Least Squares Estimator > Orthogonal Projections, we have the result of

.
.
Remark (sum of residual).

If includes a constant term, then we have

Proof.Since , we have Then, since , we have Therefore, . □

. where the last inequality holds since

Remark (residual and error).

The length of estimated residual is always smaller than the true error. This holds since is estimated to minimize the length of the residual vector.
. where the third equality holds since .

Remark (Pythogorean law).

The relationship shows the Pythagorean law since the projected image is orthogonal to the estimated residual vector.
Pasted image 20240515224311.png

If there is a constant term as a regressor, .
- From the result 2), we have thus .
- Then, we have thus the first row equals to

Remark (average fitted value equals to actual mean).

Since we have , this leads to therefore, .

Goodness of Fit

Assume that there is a constant in the regression and let then we have where the last equation holds since . Here, we denote each term as where in vector term, Each term refers to:

Total Sum of Squares (TSS): squared deviations between the sample
Explained Sum of Squares (ESS): squared deviations between the estimated
Residual Sum of Squares (RSS): squared deviations of residuals from .

Definition (R-squared).

A measures the goodness of fit for a regression, by explaining the proportion of the total variation explained by the regression.

We can represent in various forms.

Scalar form:
Norm form:
Matrix form: where

Remark (uncentered R-squared).

The uncentered is defined as

Remark (range of R-squared).

We have since from the matrix form, is projecting onto the subspace of the range space of .

Remark (monotone increase of R-squared by

Note that the range of gets smaller as the number of regressors increases. Therefore, the term always gets smaller as the increases, which leads to an increase in .

By ^4b6116 Remark 6 (monotone increase of R-squared by ), has unappealing feature since we can always increase by adding irrelevant regressors. Thus, we have additional method called adjusted .

Definition (adjusted R-squared).

The adjusted R-squared is defined as where is the number of regressors, and is the sample size.

On Asymptotic Results in Basic Linear Model, we will show that increases when the new variable has the absolute value of statistic greater than 1, and decreases when it is less then 1.

Reparameterization

Consider two linear regressions: where for some non-singular matrix .

Since we have the relationship of as is invertible.

As we have already shown in Geometry of Least Squares Estimator > Theorem 15 (linear transformations of regressor), this indicates that the OLS estimate is robust to the change in unit of measurement.

Frisch-Waugh-Lovell Theorem

Consider a linear model with regressors: where is the least squares estimate.

Theorem (Frish-Waugh-Lovell Theorem).

Let be a full rank matrix of exogenous regressors, be a matrix of exogenous regressors. And let a vector of endogenous variables.
Note let be the OLS estimates of obtained from and be the OLS estimates of obtained from the regression with Then, we have

The theorem implies that the OLS estimates from multiple regressors and equals to the 2-stage regressions where the first stage is and , and the second stage estimator is unexplained part of by explained by the unexplained part of by . This means that captures the pure effect of that cannot be explained by .

Proof.Let , and . Then the first regression is and for , , and , then the second regression is WTS First, from thus which leads to Pre-multiplying on the equation 2), Combining the equations 1) and 3) together, since is idempotent and symmetric, which completes the proof. □

Implications of FWL Theorem

The following remarks are implied by the ^5c5540 Theorem 8 (Frish-Waugh-Lovell Theorem).

Remark (omitted variable problem).

If some relevant regressor is not controlled, then the OLS estimator is biased by the amount of indirect effect of the omitted variable to the controlled variable. which means, where

Remark (demeaned variables).

If the both regressor and regressed are demeaned, then the regression without a constant variable is equals to the regression with a constant. i.e. for is equivalent to

Proof.Note that by Geometry of Least Squares Estimator > Proposition 16 (constant term averaging), we have where .
From by applying ^5c5540 Theorem 8 (Frish-Waugh-Lovell Theorem), the estimates is equivalent to where and . □