## Definition

Let $$A$$ be an $$n\times n$$ matrix. $$A$$ is said to be symmetric if $$A = A^\mathsf{T}$$.

### Examples

$$\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$$, $$\begin{bmatrix} \pi & 1 \\ 1 & \sqrt{2} \end{bmatrix}$$, $$\begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 5 \\ 3 & 5 &6 \end{bmatrix}$$

Symmetric matrices are found in many applications such as control theory, statistical analyses, and optimization.

## Eigenvalues of real symmetric matrices

Real symmetric matrices have only real eigenvalues. We will establish the $$2\times 2$$ case here. Proving the general case requires a bit of ingenuity.

Let $$A$$ be a $$2\times 2$$ matrix with real entries. Then $$A = \begin{bmatrix} a & b\\ b & c\end{bmatrix}$$ for some real numbers $$a,b,c$$. The eigenvalues of $$A$$ are all values of $$\lambda$$ satisfying $\left|\begin{array}{cc} a - \lambda & b \\ b & c - \lambda \end{array}\right | = 0.$ Expanding the left-hand-side, we get $\lambda^2 -(a+c)\lambda + ac - b^2 = 0.$ The left-hand side is a quadratic in $$\lambda$$ with discriminant $$(a+c)^2 - 4ac + 4b^2 = (a-c)^2 + 4b^2$$ which is a sum of two squares of real numbers and is therefore nonnegative for all real values $$a,b,c$$. Hence, all roots of the quadratic are real and so all eigenvalues of $$A$$ are real.

## Orthogonal matrix

Real symmetric matrices not only have real eigenvalues, they are always diagonalizable. In fact, more can be said about the diagonalization.

We say that $$U \in \mathbb{R}^{n\times n}$$ is orthogonal if $$U^\mathsf{T}U = UU^\mathsf{T} = I_n$$. In other words, $$U$$ is orthogonal if $$U^{-1} = U^\mathsf{T}$$.

If we denote column $$j$$ of $$U$$ by $$u_j$$, then the $$(i,j)$$-entry of $$U^\mathsf{T}U$$ is given by $$u_i\cdot u_j$$. Since $$U^\mathsf{T}U = I$$, we must have $$u_j\cdot u_j = 1$$ for all $$j = 1,\ldots n$$ and $$u_i\cdot u_j = 0$$ for all $$i\neq j$$. Therefore, the columns of $$U$$ are pairwise orthogonal and each column has norm 1. We say that the columns of $$U$$ are orthonormal. A vector in $$\mathbb{R}^n$$ having norm 1 is called a unit vector.

### Examples

The identity matrix is trivially orthogonal. Here are two nontrivial orthogonal matrices: $$\displaystyle\frac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}$$, $$\displaystyle\frac{1}{9}\begin{bmatrix} -7 & 4 & 4 \\ 4 & -1 & 8 \\ 4 & 8 & -1 \end{bmatrix}$$

## Orthogonal diagonalization

A real square matrix $$A$$ is orthogonally diagonalizable if there exist an orthogonal matrix $$U$$ and a diagonal matrix $$D$$ such that $$A = UDU^\mathsf{T}$$. Orthogonalization is used quite extensively in certain statistical analyses.

An orthogonally diagonalizable matrix is necessarily symmetric. Indeed, $$( UDU^\mathsf{T})^\mathsf{T} = (U^\mathsf{T})^\mathsf{T}D^\mathsf{T}U^\mathsf{T} = UDU^\mathsf{T}$$ since the transpose of a diagonal matrix is the matrix itself.

The amazing thing is that the converse is also true: Every real symmetric matrix is orthogonally diagonalizable. The proof of this is a bit tricky. However, for the case when all the eigenvalues are distinct, there is a rather straightforward proof which we now give.

First, we claim that if $$A$$ is a real symmetric matrix and $$u$$ and $$v$$ are eigenvectors of $$A$$ with distinct eigenvalues $$\lambda$$ and $$\gamma$$, respectively, then $$u^\mathsf{T} v = 0$$. To see this, observe that $$\lambda u^\mathsf{T} v = (\lambda u)^\mathsf{T} v = (Au)^\mathsf{T} v = u^\mathsf{T} A^\mathsf{T} v u^\mathsf{T} A v = \gamma u^\mathsf{T} v$$. Hence, if $$u^\mathsf{T} v\neq 0$$, then $$\lambda = \gamma$$, contradicting that they are distinct. This proves the claim.

Now, let $$A\in\mathbb{R}^{n\times n}$$ be symmmetric with distinct eigenvalues $$\lambda_1,\ldots,\lambda_n$$. Then every eigenspace is spanned by a single vector; say $$u_i$$ for the eigenvalue $$\lambda_i$$, $$i = 1,\ldots, n$$. We may assume that $$u_i \cdot u_i =1$$ for $$i = 1,\ldots,n$$. If not, simply replace $$u_i$$ with $$\frac{1}{\|u_i\|}u_i$$. This step is called normalization.

Let $$U$$ be an $$n\times n$$ matrix whose $$i$$th column is given by $$u_i$$. Let $$D$$ be the diagonal matrix with $$\lambda_i$$ as the $$i$$th diagonal entry. Then, $$A = UDU^{-1}$$.

To complete the proof, it suffices to show that $$U^\mathsf{T} = U^{-1}$$. First, note that the $$i$$th diagonal entry of $$U^\mathsf{T}U$$ is $$u_i^\mathsf{T}u_i = u_i \cdot u_i = 1$$. Hence, all entries in the diagonal of $$U^\mathsf{T}U$$ are 1.

Now, the $$(i,j)$$-entry of $$U^\mathsf{T}U$$, where $$i \neq j$$, is given by $$u_i^\mathsf{T}u_j$$. As $$u_i$$ and $$u_j$$ are eigenvectors with different eigenvalues, we see that this $$u_i^\mathsf{T}u_j = 0$$.

Thus, $$U^\mathsf{T}U = I_n$$. Since $$U$$ is a square matrix, we have $$U^\mathsf{T} = U^{-1}$$.

The above proof shows that in the case when the eigenvalues are distinct, one can find an orthogonal diagonalization by first diagonalizing the matrix in the usual way, obtaining a diagonal matrix $$D$$ and an invertible matrix $$P$$ such that $$A = PDP^{-1}$$. Then normalizing each column of $$P$$ to form the matrix $$U$$, we will have $$A = U D U^\mathsf{T}$$.

To see a proof of the general case, click here.

## Exercises

Give an orthogonal diagonalization of $$A = \begin{bmatrix} 3 & -2 \\ -2 & 3\end{bmatrix}$$.