Introductory Linear Algebra

#math #linear-algebra

Geometry of Vector Space

Orthogonality

Definition (orthogonal and orthonormal vectors).

Consider a set of vectors . We say is orthogonal if And is orthonormal if

Definition (orthogonal complement).

Let be a linear subspace of . Then the orthogonal complement of is

Eigenvalue and Eigenvector

Definition (eigenvectors and eigenvalues).

A nonzero vector is an eigenvector of if where the scalar is called eigenvalues, and is called as an eigenvector of that corresponds to .

Proposition (properties of eigenvalue).

Let be matrix which is diagonalizable, where are the eigenvalues. Then the followings hold:

  1. .
  2. .
  3. Each eigenvalue of an idempotent matrix is either zero or unity.
  4. implies if is invertible.
  5. Each eigenvalue of an orthogonal matrix is either or .
  6. Each eigenvalue of a positive semidefinite matrix is non-negative.
  7. There exists at least one eigenvalue of if the matrix is singular.
Proposition (decomposition of a positive semidefinite matrix).

Let be a symmetric and positive semidefinite. Then the matrix can be decomposed into where .

Proof.Assume be a symmetric and positive semidefinite matrix.

Since is symmetric, it is diagonalizable with real eigenvalues. Therefore, we can decompose into where the columns of are the eigenvectors of and is a diagonal matrix with -th entry is an eigenvalue of that corresponds to the eigenvector of -th column of . i.e. Also, as is positive semidefinite, all eigenvalues are non-negative. Therefore, we have Therefore, we have This completes the proof. □

Theorem (decomposition of symmetric and idempotent matrix).

Let be a symmetric and idempotent with rank . Then the matrix can be decomposed into where and .

Proof.Since , it can be diagonalized as where is orthogonal matrix with each columns are eigenvectors and is a diagonal matrix where each -th elements are eigenvalues that corresponds to -th column of . Also, as , we assume that the nonzero diagonal elements of are at the top left.

As is idempotent, remark that any eigenvalue of is or , since must hold. Thus, the top left elements of are .

Therefore, as is symmetric and idempotent, where is matrix obtained by keeping the first columns of . Note that the fourth equality holds as are orthogonal, and the fifth equality holds since the the elements of are either or .

Furthermore, we have since is orthogonal. □

Linear Transformation

Range and Null Space

Definition (range or column space).

The range space or column space of a matrix of dimension is,

Remark (rank).

The rank of a matrix of dimension is

Definition (null space).

The null space of a matrix of dimension is

Lemma (rank-nullity theorem).

Let the be called as nullity. Then we have

Check Dimension Theory on Linear Algebra textbook for more detailed proof.

Lemma (null space is orthogonal complement of row space).

For a matrix of dimension, we have

Proof.We show that the both must be a subset to another one.
() Let , then by Definition 9 (null space), we have WTS: , which equals to by Definition 7 (range or column space) and Definition 2 (orthogonal complement), Since , we have

() Let . Then by the definition, Since this relationship holds for every , let . Then we have which means . □

Symmetric and Idempotent

Definition (symmetric and idempotent).

Let be a matrix.

  • is symmetric if .
  • is idempotent if .
Remark (rank of symmetric idempotent).

Let be a symmetric and idempotent matrix. then

Orthogonal Projection

Definition (projection).

A linear mapping is a projection mapping onto if i.e., .

Definition (orthogonal projection).

A linear mapping is an orthogonal projection mapping onto if is a projection mapping and i.e.,

Theorem (orthogonal projection is symmetric and idempotent).

A matrix is an orthogonal projection mapping onto its own column space if and only if is both symmetric and idempotent.

Proof.We first prove the only if part, then we prove if part.
() Let be a orthogonal projection on .

  1. since is projection, by Definition 14 (projection), we have
    • as , we have s.t. .
    • Therefore, thus , i.e. is idempotent.
  2. since is orthogonal projection, by Definition 15 (orthogonal projection), we have

() Let be both symmetric and idempotent.

  1. First we show is projection on .
    1. notice that by the Definition 7 (range or column space).
    2. Let . then by Definition 7 (range or column space), thus by the idempotent of , Therefore,
  2. Next we show is orthogonal projection.
    • Let . Then since this holds for all , by letting , we have where the first equality holds since is symmetric.

This completes the proof. □

Theorem (eigenvalue decomposition of orthogonal projection).

Let be an orthogonal projection matrix onto an -dimensional linear subspace of . Then there exists an orthogonal matrix such that

The Theorem 17 (eigenvalue decomposition of orthogonal projection) is a direct result from Theorem 6 (decomposition of symmetric and idempotent matrix).

Inner Products and Norms

Interactive Graph
Table Of Contents
Introductory Linear Algebra
Geometry of Vector Space
Orthogonality
Eigenvalue and Eigenvector
Linear Transformation
Range and Null Space
Symmetric and Idempotent
Orthogonal Projection
Inner Products and Norms