Linear Algebra: III Square and Inverse Matrices

Square and Inverse Matrices

An $n$ by $n$ square matrix has the same number of rows and columns:

$$ A = \begin{bmatrix} a_{11} & \cdots & a_{1n} \\ \vdots & & \vdots \\ a_{n1} & \cdots & a_{nn} \end{bmatrix} $$

If $A$ is an $n$ by $n$ square matrix and $\vec{x} \in \mathbb{R}^n$, then the product $A\vec{x}$ is also a vector in $\mathbb{R}^n$, defining a linear transformation $T: \mathbb{R}^n \to \mathbb{R}^n$.

The identity matrix $I$ is an $n$ by $n$ square matrix with 1s on the main diagonal and 0s elsewhere:

$$ I = \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & & 0 \\ \vdots & & \ddots & \vdots \\ 0 & 0 & \cdots & 1 \end{bmatrix} $$

It satisfies:

$$ AI = IA = A $$

The inverse of a square matrix $A$, denoted $A^{-1}$, is a matrix such that:

$$ A A^{-1} = A^{-1} A = I $$

Which may not exist. If $A^{-1}$ exists, then $A$ is called invertible. If $A$ and $B$ are invertible $n$ by $n$ square matrices, then:

$$ (AB)^{-1} = B^{-1} A^{-1} $$

Also, the inverse of a transpose is the transpose of the inverse since $A A^{-1} = I$ so $(A^{-1})^\top A^\top = I$ thus:

$$ (A^\top)^{-1} = (A^{-1})^\top $$

Consider the system:

$$A\vec{x} = \vec{b}$$

If $A$ is invertible, then multiplying both sides by $A^{-1}$,

$$ A^{-1}A\vec{x} = A^{-1}\vec{b} \quad \;\Rightarrow\; \quad \vec{x} = A^{-1}\vec{b} $$

If \( A \) is invertible, then its columns are linearly independent. This is because if a linear combination of the columns of $A$ equals the zero vector

$$A\vec{x} = \vec{0}$$

Then, multiplying both sides by $A^{-1}$,

$$ A^{-1}A\vec{x} = A^{-1}\vec{0} \quad \;\Rightarrow\; \quad \vec{x} = \vec{0}
$$

Thus, the only solution is the trivial solution, so the columns of \( A \) are linearly independent.

If the columns of \( A \) are linearly independent, then the rows of \( A \) are linearly independent. Assume the columns of \( A \) are linearly independent. Then the rows of \( A^\top \) are the columns of \( A \), so they are linearly independent. Now suppose:

$$A^\top \vec{x} = \vec{0}$$

Taking the transpose of both sides and multiplying by $A^{-1}$,

$$\vec{x}^\top A = \vec{0}^\top$$
$$ \vec{x}^T= \vec{0}^T $$

Thus, the only solution is the trivial solution, so the rows of \( A \) are linearly independent.

Let $\vec{a}$ and $\vec{b}$ be $n$ dimensional column vectors. Then the outer product is an $n$ by $n$ square matrix defined as:

$$ \vec{a}\vec{b}^\top = \begin{bmatrix} a_1 b_1 & a_1 b_2 & \cdots & a_1 b_n \\ a_2 b_1 & a_2 b_2 & \cdots & a_2 b_n \\ \vdots & \vdots & \ddots & \vdots \\ a_n b_1 & a_n b_2 & \cdots & a_n b_n \end{bmatrix} $$

Where $\vec{a}\vec{b}^\top = \vec{b}\vec{a}^\top$.

The vector projection of $\vec{a}$ onto $\vec{b}$ can be rewritten as:

$$ \mathrm{Proj}_{\vec{b}} \ \vec{a} = \frac{\vec{b}^\top \vec{a}}{\vec{b}^\top \vec{b}} \vec{b} = \frac{\vec{b}\vec{b}^\top}{\vec{b}^\top \vec{b}} \vec{a} $$

Where $\frac{\vec{b}\vec{b}^\top}{\vec{b}^\top \vec{b}}$ is called the projection matrix onto $\vec{b}$.

If $\vec{b}$ is a unit vector (written $\hat{b}$) so that $\vec{b}^\top \vec{b} = \|\vec{b}\|^2 = 1$, then the projection matrix is $\hat{b}\hat{b}^\top$.

There are three types of elementary square matrices besides the identity corresponding to three linear row operations.

First, a swapping elementary matrix $(E^{\text{swap}})_{pq}A$ swaps row $p$ and $q$ of $A$. It is equivalent to the identity matrix with row $p$ and $q$ swapped. For $3$ by $3$ square matrices:

$$ (E^{\text{swap}})_{12} = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix}$$
$$ (E^{\text{swap}})_{23} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{bmatrix} $$
$$ (E^{\text{swap}})_{13} = \begin{bmatrix} 0 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 0 \end{bmatrix} $$

Where

$$ (E^{\text{swap}})^{-1}_{pq} = (E^{\text{swap}})_{pq} $$

Second, a scaling elementary matrix $(E^{\text{scale}})^k_p A$ scales row $p$ of $A$ by $k$. It is equivalent to the identity matrix with the $1$ in row and column $p$ changed to $k$. For $3$ by $3$ square matrices:

$$ (E^{\text{scale}})^k_{1} = \begin{bmatrix} k & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$
$$ (E^{\text{scale}})^k_{2} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & k & 0 \\ 0 & 0 & 1 \end{bmatrix} $$
$$ (E^{\text{scale}})^k_{3} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & k \end{bmatrix} $$

Where

$$ ((E^{\text{scale}})^k_p)^{-1} = (E^{\text{scale}})^{1/k}_p $$

Finally, a shearing elementary matrix $(E^{\text{shear}})^k_{pq} A$ scales row $p$ of $A$ by $k$ and adds this to row $q$ of $A$. It is equivalent to the identity matrix with the $0$ in row $q$ and column $p$ changed to $k$. For $3$ by $3$ square matrices:

$$ (E^{\text{shear}})^k_{12} = \begin{bmatrix} 1 & 0 & 0 \\ k & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$
$$ (E^{\text{shear}})^k_{23} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & k & 1 \end{bmatrix} $$
$$ (E^{\text{shear}})^k_{13} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ k & 0 & 1 \end{bmatrix} $$

Where

$$ ((E^{\text{shear}})^k_{pq})^{-1} = (E^{\text{shear}})^{-k}_{pq} $$

A permutation matrix is a product of swapping elementary matrices.

Let $A$ be an invertible matrix. Suppose that there exist at most $n^2$ elementary matrices $E_1, \ldots, E_{n^2}$ for each entry in $A$ and a permutation matrix $P$ such that:

$$ E_{n^2} \cdots E_1 P A = I $$

Then,

$$ A^{-1} = E_{n^2} \cdots E_1 P $$
$$ A = (E_{n^2}\cdots E_1P)^{-1} = P^{-1}E_1^{-1} \cdots E_{n^2}^{-1} $$
$$ P A = E_1^{-1} \cdots E_{n^2}^{-1} $$

There are infinitely many ways to write both $A$ and $A^{-1}$ as a product of elementary matrices. The standard process for finding $A^{-1}$ is called Gauss–Jordan elimination.

Multiply $A$ by a permutation matrix so that the diagonal entries are not zero. Start by scaling the first scalar in the first column (the first pivot) to $1$ and then shearing the remaining $n-1$ scalars in the first column to $0$. Continue this process until scaling the $n$th scalar (the $n$th pivot) in the $n$th column to $1$ and then shearing the remaining $n-1$ scalars in the $n$th column to $0$.