5. Determinants

The determinant is a scalar-valued function of a square matrix that captures essential properties of the linear transformation represented by that matrix. It provides a single number that summarizes the “scaling” and “orientation” behavior of the transformation.

History

Surprisingly, determinants were used before matrices. Historically, they emerged from the need to solve systems of linear equations:

• Independent Discovery (Late 17th Century): The concept was developed independently by Seki Kōwa in Japan (1683) and Gottfried Wilhelm Leibniz in Germany (1693). Seki Kōwa used them for elimination theory, while Leibniz used them to solve systems of linear equations using indices.
• Formalization (18th - 19th Century): Augustin-Louis Cauchy (1812) gave the modern name “determinant” and proved the multiplication theorem ($det(AB)=det(A)det(B)$).
• The Jacobian: Carl Jacobi (1841) introduced the “functional determinant” (the Jacobian), which bridged the gap between algebra and multivariable calculus.

Initially, determinants were just a tool for calculating solutions (like Cramer’s Rule), but they eventually became central to understanding the geometry of linear transformations.

Definition and Calculation

A determinant is only defined for square matrices ($n\times n$).

2x2 Matrix

For a matrix $A=\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$: $\text{det}(A)=|A|=ad-bc$

3x3 Matrix (Rule of Sarrus)

For a matrix $A=\left[\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right]$: $\text{det}(A)=a(ei-fh)-b(di-fg)+c(dh-eg)$

Laplace Expansion (General)

The determinant of an $n\times n$ matrix can be calculated by expanding along any row or column using minors and cofactors: $\text{det}(A)={\sum }_{j=1}^n{a}_{ij}(-1{)}^{i+j}\text{det}(M_{ij})$ where $M_{ij}$ is the $(n-1)\times (n-1)$ submatrix formed by deleting the $i$-th row and $j$-th column.

Properties of Determinants

1. Identity Matrix: $\text{det}(I)=1$.
2. Transposition: $\text{det}(A^T)=\text{det}(A)$.
3. Multiplication: $\text{det}(AB)=\text{det}(A)\cdot \text{det}(B)$.
4. Inversion: $\text{det}(A^{-1})=\frac{1}{\text{det}(A)}$ (if $A$ is invertible).
5. Row/Column Operations:
   • Swapping two rows/columns multiplies the determinant by $-1$.
   • Multiplying a row/column by a scalar $k$ multiplies the determinant by $k$.
   • Adding a multiple of one row to another does not change the determinant.
6. Singularity: $\text{det}(A)=0$ if and only if $A$ is singular (not invertible). This happens if rows or columns are linearly dependent.

Geometric Interpretation

The determinant measures how much a linear transformation scales “volume” in space. This interpretation is often the “aha!” moment for students because it makes the abstract algebra feel tangible.

Scaling Factor

• 2D: The absolute value of the determinant $|\text{det}(A)|$ is the area of the parallelogram formed by the column vectors of $A$.
• 3D: The absolute value $|\text{det}(A)|$ is the volume of the parallelepiped formed by the column vectors.
• Higher Dimensions: It represents the $n$-dimensional volume (hypervolume).

Orientation

• Positive Determinant: The transformation preserves orientation (right-hand rule stays the same).
• Negative Determinant: The transformation reverses orientation (it involves a “reflection”).
• Zero Determinant: The transformation “collapses” the space into a lower dimension (e.g., a 3D volume squashed into a 2D plane).

Where the Geometric Interpretation is Crucial:

1. Change of Variables (Calculus): When switching from Cartesian to Polar or Spherical coordinates, the Jacobian determinant tells us how to scale the “area element” ($dA$) or “volume element” ($dV$). Without it, we would get the wrong answer when integrating.
2. Computer Graphics: To check if a transformation (like scaling) has flipped a shape “inside out” (reflection), developers check the sign of the determinant.
3. Data Variance: In PCA, the determinant of the covariance matrix represents the “total variance” or the “volume” of the data cloud.
4. Physics: Calculating the cross product or triple product involves determinants, which represent the area or volume of geometric shapes in space.

Applications

1. Matrix Invertibility

In machine learning algorithms (like Linear Regression), we often need to invert matrices ($A^{-1}$). A matrix is invertible only if $\text{det}(A)\ne 0$. A determinant close to zero suggests “near-singularity,” which can lead to numerical instability.

2. Multivariate Gaussian Distributions

In probabilistic modeling, the probability density function of a multivariate normal distribution depends on the determinant of the covariance matrix $\Sigma$: $f(\mathbf{x})=\frac{1}{\sqrt{(2\pi {)}^k|\Sigma |}}\exp \left(-\frac{1}{2}(\mathbf{x}-\mu {)}^T{\Sigma }^{-1}(\mathbf{x}-\mu )\right)$ Here, $|\Sigma |$ represents the “volume” of the spread of the data. A small determinant indicates that the data is concentrated along a lower-dimensional subspace (high correlation between features).

3. Change of Variables (The Jacobian)

When transforming variables in multi-dimensional calculus (e.g., in Normalizing Flows or probability density transformations), the Jacobian determinant is used to account for how the “volume” elements change: $dxdy=|\text{det}(J)|dudv$

4. Calculating Eigenvalues

Determinants are fundamental for finding the eigenvalues of a matrix. The eigenvalues $\lambda$ are the roots of the characteristic equation: $\text{det}(A-\lambda I)=0$

5. Numerical Stability and Redundancy

• Multicollinearity: If the determinant of the feature correlation matrix is near zero, it indicates high multicollinearity, meaning some features are redundant.
• Condition Number: While not the same, the determinant is related to the “health” of a matrix in numerical computations.

---

Related Notes

• 2. Linear Algebra
• Principal Component Analysis (PCA)
• Eigenvalues and Eigenvectors
• Singular Value Decomposition