Mathematics Blog

Gram–Schmidt and QR Decomposition Suppose that $\{ \vec{v}_1, \ldots, \vec{v}_n \}$ is a set of linearly independent column vectors in $\mathbb{R}^n$, and let $A$ be an $n$ by $n$ square matrix: $$ A = [\vec{v}_1 \ \cdots \ \vec{v}_n] $$ Then the Gram--Schmidt process produces an orthonormal basis $$ \{ \hat{q}_1, \ldots, \hat{q}_n \} $$ which can be written as the $n$ by $n$ orthonormal matrix: $$ Q = [\hat{q}_1 \ \cdots \ \hat{q}_n] $$ Let $\hat{v}_1$ be a unit vector. Then the vector $$ (I - \hat{v}_1 \hat{v}_1^\top)\hat{v}_2 $$ Is orthogonal to $\hat{v}_1$ since $$ \hat{v}_1^\top (I - \hat{v}_1 \hat{v}_1^\top)\hat{v}_2 = \hat{v}_1^\top\hat{v}_2 - (\hat{v}_1^\top\hat{v}_1)\hat{v}_1^\top\hat{v}_2 = \hat{v}_1^\top\hat{v}_2 - \hat{v}_1^\top\hat{v}_2 = 0 $$ Using this idea, we construct an orthonormal basis: $$ \hat{q}_1 = \frac{\vec{v}_1}{\|\vec{v}_1\|} $$ $$ \vec{q}_2 = (I - \hat{q}_1 \hat{q}_1^\top)\vec{v}_2, \qquad \hat{q}_2...

Orthonormal Matrices A square $n$ by $n$ matrix $Q$ is called orthonormal if: $$ QQ^\top = Q^\top Q = I $$ $$ Q^{-1} = Q^\top $$ The columns $\vec{q}_1, \ldots, \vec{q}_n$ of $Q$ (and the rows) form an orthonormal basis of $\mathbb{R}^n$: $$ \vec{q}_i^\top \vec{q}_j = 0 \text{ if } i \neq j, \quad \|\vec{q}_i\| = 1 $$ An orthonormal matrix preserves inner products / dot products: $$ (Q\vec{x})^\top(Q\vec{y}) = \vec{x}^\top Q^\top Q \vec{y} = \vec{x}^\top I \vec{y} = \vec{x}^\top \vec{y} $$ An orthonormal matrix also preserves magnitude: $$ \|Q\vec{x}\|^2 = (Q\vec{x})^\top Q\vec{x} = \vec{x}^\top Q^\top Q \vec{x} = \vec{x}^\top I \vec{x} = \vec{x}^\top \vec{x} = \|\vec{x}\|^2 $$ So an orthonormal matrix preserves angles between vectors: $$ \cos{\theta} = \frac{(Q\vec{x})^\top(Q\vec{y})}{\|Q\vec{x}\|\|Q\vec{y}\|} = \frac{\vec{x}^\top \vec{y}}{\|\vec{x}\|\|\vec{y}\|} $$ There are two special types of orthogonal matrices. First, a...

Triangular Matrices and LU Decomposition A square matrix $A$ is called upper triangular if: $$ a_{ij} = 0 \quad \text{for all } i > j $$ A square matrix $A$ is called lower triangular if: $$ a_{ij} = 0 \quad \text{for all } i Upper and lower triangular $n \times n$ square matrices both contain at least $$ \frac{n(n-1)}{2} $$ Zeros and $$ \frac{n(n+1)}{2} $$ Other entries. The identity matrix $I$ is both upper triangular and lower triangular. Scaling elementary matrices and their products are also both upper and lower triangular. Shearing elementary matrices are either upper triangular or lower triangular but never both. Swapping elementary matrices are neither upper nor lower triangular. If $A$ and $B$ are upper triangular $n \times n$ square matrices, then $AB$ is also upper triangular. Indeed, for $i > j$: $$ (ab)_{ij} = \sum_{k=1}^n a_{ik} b_{kj} $$ If $i > k$, then $a_{ik} = 0$. If $k \ge i$, then $k > j...

Cross Product and Triple Scalar Product Consider two $2$ dimensional vectors: $$ \vec{u}= \begin{bmatrix} a \\ c \end{bmatrix}, \quad \vec{v}= \begin{bmatrix} b \\ d \end{bmatrix} $$ The area of the parallelogram formed by these vectors is: $$ \text{Area} = \|\vec{u}\| \, \|\vec{v}\| \sin \theta $$ where $\|\vec{u}\| = \sqrt{a^2 + c^2}$, $\|\vec{v}\| = \sqrt{b^2 + d^2}$, and $\theta$ is the angle between these vectors. Recall: $$ \vec{u}^\top \vec{v} = \|\vec{u}\| \, \|\vec{v}\| \cos \theta $$ The dot product is: $$ \vec{u}^\top\vec{v} = ab + cd $$ Therefore: $$ \cos \theta = \frac{ab + cd}{\sqrt{a^2 + c^2}\sqrt{b^2 + d^2}} $$ Since $\sin^2 \theta = 1 - \cos^2 \theta$, then: $$ \sin^2 \theta = 1 - \frac{(ab + cd)^2}{(a^2 + c^2)(b^2 + d^2)} = \frac{(a^2 + c^2)(b^2 + d^2)-(ab + cd)^2}{(a^2 + c^2)(b^2 + d^2)} $$ Thus: $$ \text{Area}^2 = (a^2 + c^2)(b^2 + d^2)\sin^2 \theta = (a^2 + c^2)(b^2 + d^2) - (ab + cd)^2 $$ $$ = ...

Determinants The determinant of a square matrix $A$, denoted $\det(A)$, describes how the linear transformation $T: \mathbb{R}^n \to \mathbb{R}^n$, $$ T(\vec{x}) = A\vec{x} $$ scales $n$ dimensional area / volume. If $|\det(A)| > 1$, areas / volumes are expanded. If $|\det(A)| If $\det(A) = 0$, areas / volumes collapse to lower dimension. The sign of $\det(A)$ indicates whether the transformation preserves or reverses orientation. Scaling volume, the determinant of a matrix product is the product of determinants: $$ \det(AB) = \det(A)\det(B) $$ Also, a scalar matrix product scales all $n$ rows / columns by the scalar so: $$ \det(cA) = c^n \det(A) $$ If a matrix has a row or column with zeros, then $\det(A) = 0$, since the $n$ dimensional volume collapses to zero under a transformation that spans at most an $(n-1)$ dimensional space. If any two rows or columns of $A$ are linearly dependent, then $\det(A) = 0$, since the $n$ dime...

Square and Inverse Matrices An $n$ by $n$ square matrix has the same number of rows and columns: $$ A = \begin{bmatrix} a_{11} & \cdots & a_{1n} \\ \vdots & & \vdots \\ a_{n1} & \cdots & a_{nn} \end{bmatrix} $$ If $A$ is an $n$ by $n$ square matrix and $\vec{x} \in \mathbb{R}^n$, then the product $A\vec{x}$ is also a vector in $\mathbb{R}^n$, defining a linear transformation $T: \mathbb{R}^n \to \mathbb{R}^n$. The identity matrix $I$ is an $n$ by $n$ square matrix with 1s on the main diagonal and 0s elsewhere: $$ I = \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & & 0 \\ \vdots & & \ddots & \vdots \\ 0 & 0 & \cdots & 1 \end{bmatrix} $$ It satisfies: $$ AI = IA = A $$ The inverse of a square matrix $A$, denoted $A^{-1}$, is a matrix such that: $$ A A^{-1} = A^{-1} A = I $$ Which may not exist. If $A^{-1}$ exists, then $A$ is called invertible . If $A$ and $B$ are inve...

Matrices An $m$ by $n$ matrix is an ordered collection of $n$ $m$th dimension column vectors as well as $m$ $n$ dimensional row vectors written as: $$ A= \begin{bmatrix} a_{11} & \cdots & a_{1n} \\ \vdots & & \vdots \\ a_{m1} & \cdots & a_{mn} \end{bmatrix} = \begin{bmatrix} \vec{c}_1 & \cdots & \vec{c}_n \end{bmatrix} \quad \text{where} \quad \vec{c}_j = \begin{bmatrix} a_{1j} \\ \vdots \\ a_{mj} \end{bmatrix} $$ $$ A= \begin{bmatrix} a_{11} & \cdots & a_{1n} \\ \vdots & & \vdots \\ a_{m1} & \cdots & a_{mn} \end{bmatrix} = \begin{bmatrix} \vec{r}_1^\top \\ \vdots \\ \vec{r}_m^\top \end{bmatrix} \quad \text{where} \quad \vec{r}_i^\top = \begin{bmatrix} a_{i1} & \cdots & a_{in} \end{bmatrix} $$ A column vector is a $n$ by $1$ matrix, a row vector is a $1$ by $n$ matrix, and a scalar is a $1$ by $1$ matrix. If $A$ is a matrix with $m$ rows and $n$ columns, then its transpose $A^\top$ is a matrix with ...

Scalars and Vectors A scalar is defined as a real number $c \in \mathbb{R}$ with 1 dimensional length $|c|$ and either positive or negative direction. A vector is an ordered collection of scalars with dimension $n \geq 1$: Written as a column vector $$\vec{a} =\begin{bmatrix} a_1 \\ \vdots \\ a_n \end{bmatrix}\in \mathbb{R}^n $$ Written as row vector $$\vec{a}^\top =\begin{bmatrix} a_1 & \ldots & a_n \end{bmatrix}\in \mathbb{R}^n$$ Where $\top$ is the transpose operator that exchanges column vectors with row vectors. Note $(\vec{a}^\top)^\top = \vec{a}$. With magnitude $$\|\vec{a}\| = \|\vec{a}^\top\|=\sqrt{a_1^2+\ldots+a_n^2} \geq 0$$ And direction (with the exception of $\vec{0}$ where $\|\vec{0}\|=0$ and $\vec{0}$ has no direction). Row vectors can be treated as transposes of column vectors, and the same operations apply via this identification. Vector addition for column vectors is defined as $$\vec{a} +\vec{b} = \...