Normal Distribution Theory

#econometrics #economics

Oh, Hyunzi. (email: wisdom302@naver.com)
Korea University, Graduate School of Economics.
2024 Spring, instructed by prof. Kim, Dukpa.

Main References

Kim, Dukpa. (2024). "Econometric Analysis" (2024 Spring) ECON 518, Department of Economics, Korea University.
Davidson and MacKinnon. (2021). "Econometric Theory and Methods", Oxford University Press, New York.

Multivariate Normal Distribution

Definition (multivariate normal distribution).

An n-dimensional random vector is said to have (multivariate) Normal distribution with parameters and , and denoted by if its probability density is given by where and is positive definite matrix.

Lemma (linear transformation of normal distribution).

Suppose that a random vector has a multivariate normal distribution where . Then, we have

See Statistical Proof > ^2567e3 Statistical Proof > Theorem 24 (linear transformation of normal distribution) for the detailed proof.

Independent Multivariate Normal

Lemma (independent random variable).

The random variables are said to be independent if and only if where is joint probability density of and is marginal probability density of .

See Statistical Proof > ^8ee059 Statistical Proof > Definition 8 (statistical independence) for the rigorous definition.

Lemma (independent in multivariate normal).

Let and . Then and are independent if and only if .

Proof.Remark that since , we have We first prove the only if part, and then prove the if part.

() Assume and are independent. Then by Statistical Proof > ^6adf1f Statistical Proof > Theorem 21 (covariance under independence), we have () Assume . Then, since the variance-covariance matrix is symmetric by we have .
Since is diagonal matrix, we have also, Therefore, we have Therefore, by ^5b702c Lemma 3 (independent random variable), and are independent. □

Lemma (independent between matrices normal).

Let -dimensional random vector and let and be nonrandom matrices. Then, and are independent if and only if .

Proof.Note that and since , we have Additionally, the joint distribution is Using these remarks, we first prove only if part, and then prove the if part.

() Assume and are independent. Then by Statistical Proof > ^6adf1f Statistical Proof > Theorem 21 (covariance under independence), we have meaning that

() Assume that . Then Therefore, we have Then, by ^93d2d0 Lemma 4 (independent in multivariate normal), and are independent. □

Lemma (independent between symmetric and idempotent matrices).

Let -dimensional random vector and let and are symmetric and idempotent matrices. Then, the following results hold:

and are independent if .
The linear function and are independent if .

Proof.First, assume . Then by the symmetricity of and , we have Thus by ^234d84 Lemma 5 (independent between matrices normal), and are independent.

As and are idempotent, we have and Therefore, and are independent.

Secondly, assume . Since is symmetric, we have Thus by ^234d84 Lemma 5 (independent between matrices normal), and are independent. Then, as is idempotent and symmetric, we have Therefore, and are independent. □

Relationships to Chi-Squared Distribution

Definition (chi-squared distribution).

Let be independent random variable following a standard normal distribution respectively: Then, the sum of their squares follows a chi-squared distribution with degrees of freedom: Note that the probability density function of is

Remark (moments of chi-squared).

For the chi-squared distribution the first and second moments of is

Lemma (multivariate normal and chi-squared distribution).

Let -dimensional random vector . Then

Proof.Let . Since is a variance-covariance matrix, it is symmetric and positive semi-definite. Thus by Introductory Linear Algebra > ^7398fb Introductory Linear Algebra > Proposition 5 (decomposition of a positive semidefinite matrix), there exists some matrix such that Additionally, if , then is positive definite, which means it is invertible and we have Thus we have i.e. follows standard normal distribution. Furthermore, by denoting and , we have Now, using ^df04ad Definition 7 (chi-squared distribution), we now show that can be expressed as a sum of squared standard normal. As hols for all , we have which completes the proof. □

Corollary (quadratic form of projection matrix follows chi-square).

Let be a projection matrix with rank , and let be an -vector random variable such that . Then we have

Proposition (normal and chi-squared distribution).

Let be independent random variables following a normal distribution respectively as Now define the sample mean and unbiased sample variance as Then, the following holds:

Relationships to Student's T-Distribution

Definition (t-distribution).

Let and be independent random variables following a standard normal distribution and a chi-squared distribution with -degrees of freedom: Then, the ratio of to the square root of , divided by the respective degrees of freedom, is said to be -distributed with degrees of freedom :

Proposition (normal and t-distribution).

Let be independent random variables following a normal distribution respectively as Now define the sample mean and unbiased sample variance as Then, the following holds:

Proof.First, by ^46ff8a Proposition 11 (normal and chi-squared distribution), we have And by ^40fcf1 Lemma 2 (linear transformation of normal distribution), we have Thus, we have and by ^4115bb Definition 12 (t-distribution), we have which completes the proof. □

Proposition (t and normal distribution).

Let be a random variable following student's T distribution. Then, we have which are alternatively shown in Econometric Analysis/Asymptotics > Corollary 11 (CLT of student-t distribution).

Relationships to F-Distribution

Definition (f-distribution).

Let and be independent random variables following a chi-squared distribution with and degrees of freedom, receptively: Then, the F-distribution is defined as where .

Proposition (f and chi-squared distribution).

By the law of large numbers, we can express -distribution in terms of distribution:

Proposition (f and t distribution).

From ^4115bb Definition 12 (t-distribution), we have thus .