c:I[2972,["954","static/chunks/d3ac728e-d737a8481cd749b9.js","972","static/chunks/972-532bec7ef71338f7.js","333","static/chunks/app/posts/%5Bslug%5D/page-2bd090bff81d6806.js"],""] d:I[3796,["954","static/chunks/d3ac728e-d737a8481cd749b9.js","972","static/chunks/972-532bec7ef71338f7.js","333","static/chunks/app/posts/%5Bslug%5D/page-2bd090bff81d6806.js"],"default"] f:I[4565,["954","static/chunks/d3ac728e-d737a8481cd749b9.js","972","static/chunks/972-532bec7ef71338f7.js","333","static/chunks/app/posts/%5Bslug%5D/page-2bd090bff81d6806.js"],"default"] e:T735,

Understanding Gradient Descent

Gradient descent is the backbone of modern machine learning. Let's explore the mathematics that makes AI learn.

The Cost Function

When training a model, we minimize a cost function $J(\theta)$ where $\theta$ represents our model parameters.

For linear regression, the cost function is:

$$ J(\theta) = \frac{1}{2m} \sum_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)})^2 $$

Where $m$ is the number of training examples and $h_\theta(x)$ is our hypothesis function.

The Gradient

The gradient tells us the direction of steepest ascent. We compute the partial derivative:

$$ \frac{\partial J}{\partial \theta_j} = \frac{1}{m} \sum_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)}) \cdot x_j^{(i)} $$

The Update Rule

We update parameters by moving opposite to the gradient:

$$ \theta_j := \theta_j - \alpha \frac{\partial J}{\partial \theta_j} $$

Here $\alpha$ is the learning rate — too small and learning is slow, too large and we might overshoot.

Neural Network Backpropagation

For neural networks, we use the chain rule. Given layers $l$ and $l+1$:

$$ \delta^{(l)} = ((\Theta^{(l)})^T \delta^{(l+1)}) \circ g'(z^{(l)}) $$

The gradient for weights becomes:

$$ \frac{\partial J}{\partial \Theta_{ij}^{(l)}} = a_j^{(l)} \delta_i^{(l+1)} $$

Inline Math Examples

The sigmoid activation $\sigma(z) =

📬 Never Miss an Update