$\mathbf{x}$: vector of unknowns

$\mathbf{b}$: constant vector

Solution: If $\mathbf{A}$ is invertible, $\mathbf{x} = \mathbf{A}^{-1}\mathbf{b}$.

Geometric Interpretation: Intersection of hyperplanes.

ML Relevance: Solving for parameters in linear regression, optimization problems.

Definition: A scalar value calculated from the elements of a square matrix.
- For $2 \times 2$: $\det(\mathbf{A}) = \begin{vmatrix} a & b \\ c & d \end{vmatrix} = ad - bc$.
- For $3 \times 3$: $\det(\mathbf{A}) = a(ei-fh) - b(di-fg) + c(dh-eg)$.
Properties:
- If $\det(\mathbf{A}) = 0$, $\mathbf{A}$ is singular (not invertible).
- $|\det(\mathbf{A})|$ represents the scaling factor of volume/area by the transformation $\mathbf{A}$.
- $\det(\mathbf{A}\mathbf{B}) = \det(\mathbf{A})\det(\mathbf{B})$.
- $\det(\mathbf{A}^T) = \det(\mathbf{A})$.
ML Relevance: Checking matrix invertibility, understanding data variance, Principal Component Analysis (PCA).

Definition: For a square matrix $\mathbf{A}$, an eigenvector $\mathbf{v}$ is a non-zero vector that, when $\mathbf{A}$ is applied to it, only changes by a scalar factor $\lambda$ (the eigenvalue).
$\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$
Finding them: $(\mathbf{A} - \lambda\mathbf{I})\mathbf{v} = \mathbf{0}$. For non-trivial solutions, $\det(\mathbf{A} - \lambda\mathbf{I}) = 0$ (characteristic equation).
Properties:
- Eigenvectors corresponding to distinct eigenvalues are linearly independent.
- Symmetric matrices have real eigenvalues and orthogonal eigenvectors.
ML Relevance: PCA (Principal Component Analysis) for dimensionality reduction, understanding variance directions in data, spectral clustering, Markov chains.

Vector Space: A set of vectors closed under vector addition and scalar multiplication.
Subspace: A subset of a vector space that is itself a vector space.
Span: The set of all possible linear combinations of a set of vectors.
Linear Independence: A set of vectors $\{\mathbf{v}_1, \dots, \mathbf{v}_k\}$ is linearly independent if $c_1\mathbf{v}_1 + \dots + c_k\mathbf{v}_k = \mathbf{0}$ implies $c_1 = \dots = c_k = 0$.
Basis: A set of linearly independent vectors that span the entire vector space. The number of vectors in a basis is the dimension of the space.
Rank of a Matrix: The dimension of the column space (or row space). Max number of linearly independent columns (or rows).
- For an $m \times n$ matrix $\mathbf{A}$, $\text{rank}(\mathbf{A}) \le \min(m, n)$.
- If $\text{rank}(\mathbf{A}) = n$, columns are linearly independent.
Null Space (Kernel): The set of all vectors $\mathbf{x}$ such that $\mathbf{A}\mathbf{x} = \mathbf{0}$.
ML Relevance: Basis vectors can be components (PCA), understanding redundancy in data features, feature engineering.

Definition: Breaking down a matrix into a product of simpler matrices.
Eigen-decomposition: $\mathbf{A} = \mathbf{P}\mathbf{\Lambda}\mathbf{P}^{-1}$
- $\mathbf{P}$: matrix whose columns are eigenvectors