9. Vector Calculus

Vector calculus extends standard calculus to multidimensional spaces. In machine learning and deep learning, most operations involve vectors, matrices, or tensors. Understanding vector-vector derivatives is crucial for algorithms like backpropagation.

Vector-Vector Derivative (The Jacobian)

When we have a vector-valued function $\mathbf{f}(\mathbf{x})$ that maps an $n$-dimensional vector $\mathbf{x}\in {\mathbb{R}}^n$ to an $m$-dimensional vector $\mathbf{f}\in {\mathbb{R}}^m$, the derivative of $\mathbf{f}$ with respect to $\mathbf{x}$ is a matrix of partial derivatives called the Jacobian matrix.

Let $\mathbf{f}(\mathbf{x})=\left[\begin{array}{c}{f}_1(\mathbf{x})\\ f_2(\mathbf{x})\\ ⋮\\ f_m(\mathbf{x})\end{array}\right]$ and $\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]$.

The Jacobian matrix $J$ of $\mathbf{f}$ with respect to $\mathbf{x}$ is an $m\times n$ matrix defined as:

$$\begin{array}{r}J=\frac{\partial \mathbf{f}}{\partial \mathbf{x}}=\left[\begin{array}{cccc}\frac{\partial f_1}{\partial x_1}& \frac{\partial f_1}{\partial x_2}& \dots & \frac{\partial f_1}{\partial x_n}\\ \frac{\partial f_2}{\partial x_1}& \frac{\partial f_2}{\partial x_2}& \dots & \frac{\partial f_2}{\partial x_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial f_m}{\partial x_1}& \frac{\partial f_m}{\partial x_2}& \dots & \frac{\partial f_m}{\partial x_n}\end{array}\right]\end{array}$$

In numerator layout (the most common convention in ML), the $i,j$-th entry of the Jacobian is: $J_{i,j}=\frac{\partial f_i}{\partial x_j}$

• Row $i$ represents the gradient of the scalar output $f_i$ with respect to the input vector $\mathbf{x}$.
• Column $j$ represents how all output components change when the input component $x_j$ is perturbed.

Common Vector Derivatives

Here are some of the most common vector derivatives encountered in machine learning:

1. Linear transformation (Matrix-Vector multiplication): If $\mathbf{f}(\mathbf{x})=A\mathbf{x}$, where $A$ is an $m\times n$ matrix (independent of $\mathbf{x}$), then: $\frac{\partial }{\partial \mathbf{x}}(A\mathbf{x})=A$

2. Dot Product: If $f(\mathbf{x})={\mathbf{w}}^T\mathbf{x}$, where $\mathbf{w}$ is a constant vector, the output is a scalar, so the Jacobian is a $1\times n$ matrix (a row vector): $\frac{\partial }{\partial \mathbf{x}}({\mathbf{w}}^T\mathbf{x})={\mathbf{w}}^T$

3. Quadratic Form: If $f(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$, where $A$ is an $n\times n$ matrix: $\frac{\partial }{\partial \mathbf{x}}({\mathbf{x}}^{T}A\mathbf{x})={\mathbf{x}}^T(A+A^T)$ If $A$ is symmetric, this simplifies to $2{\mathbf{x}}^{T}A$.

4. Pointwise / Element-wise Functions: If $\mathbf{f}(\mathbf{x})$ applies a scalar function $g$ element-wise to $\mathbf{x}$ (e.g., an activation function like ReLU or Sigmoid), then $f_i(\mathbf{x})=g(x_i)$. Because $f_i$ only depends on $x_i$, all off-diagonal partial derivatives $\frac{\partial f_i}{\partial x_j}$ for $i\ne j$ are zero. The Jacobian is a diagonal matrix:

\begin{aligned} \frac{\partial \mathbf{f}}{\partial \mathbf{x}} = \text{diag}(g’(\mathbf{x})) = \begin{bmatrix} g’(x_1) & 0 & \dots & 0 \ 0 & g’(x_2) & \dots & 0 \ \vdots & \vdots & \ddots & \vdots \ 0 & 0 & \dots & g’(x_n) \end{bmatrix} \end{aligned}

## Chain Rule for Vectors The multivariate chain rule is the foundation of **Backpropagation** in neural networks. If we have nested vector functions $\mathbf{y} = \mathbf{g}(\mathbf{x})$ and $\mathbf{z} = \mathbf{f}(\mathbf{y})$, then the composition is $\mathbf{z} = \mathbf{f}(\mathbf{g}(\mathbf{x}))$. The Jacobian of $\mathbf{z}$ with respect to $\mathbf{x}$ is the matrix product of their individual Jacobians: $$ \frac{\partial \mathbf{z}}{\partial \mathbf{x}} = \frac{\partial \mathbf{z}}{\partial \mathbf{y}} \frac{\partial \mathbf{y}}{\partial \mathbf{x}} $$ If $\mathbf{x} \in \mathbb{R}^n$, $\mathbf{y} \in \mathbb{R}^m$, and $\mathbf{z} \in \mathbb{R}^p$: - $\frac{\partial \mathbf{z}}{\partial \mathbf{y}}$ is a $p \times m$ Jacobian matrix. - $\frac{\partial \mathbf{y}}{\partial \mathbf{x}}$ is an $m \times n$ Jacobian matrix. - The product is a $p \times n$ matrix, which correctly matches the dimension of $\frac{\partial \mathbf{z}}{\partial \mathbf{x}}$. ### Application in Backpropagation In deep learning, the final output $L$ (the loss) is a scalar. During backpropagation, we compute the gradient of the loss with respect to a layer's weights or inputs. Let $L = h(\mathbf{y})$ and $\mathbf{y} = \mathbf{f}(\mathbf{x})$. By the chain rule: $$ \frac{\partial L}{\partial \mathbf{x}} = \frac{\partial L}{\partial \mathbf{y}} \frac{\partial \mathbf{y}}{\partial \mathbf{x}} $$ Here, $\frac{\partial L}{\partial \mathbf{y}}$ is a $1 \times m$ row vector (the gradient from the layer above), and $\frac{\partial \mathbf{y}}{\partial \mathbf{x}}$ is the $m \times n$ Jacobian of the current layer. The result is a $1 \times n$ row vector, which is the gradient to pass down to the next layer.