Backpropagation by Hand

Calculate the same gradients with the chain rule and PyTorch autograd, then apply one gradient-descent update. The network has two inputs, one ReLU hidden unit, and one linear output. It is deliberately small, not itself a deep network.

The constructed delivery example matches Lecture 28: distance 1 km, 2 stops, and target time 4 minutes. These are teaching values, not measured delivery data. One improved prediction does not demonstrate generalization.

Run the cells in order with PyTorch installed. This small CPU calculation uses float64 (64-bit floating point) to make numerical gradient comparisons precise; many ordinary training examples instead use float32.

Inputs and Target

X has one row per example and one column per feature. Here its shape is (1, 2). The target y has shape (1, 1): one example, one numeric response. The learning rate is the update’s step size.

import torch

DTYPE = torch.float64
X = torch.tensor([[1.0, 2.0]], dtype=DTYPE)
y = torch.tensor([[4.0]], dtype=DTYPE)
learning_rate = 0.01
print("PyTorch:", torch.__version__)
print("Input:", X.tolist(), "shape:", tuple(X.shape))
print("Target:", y.tolist(), "shape:", tuple(y.shape))
PyTorch: 2.11.0
Input: [[1.0, 2.0]] shape: (1, 2)
Target: [[4.0]] shape: (1, 1)

Initialize Parameters You Can Check

The function creates fresh leaf tensors: tensors whose gradients PyTorch can store directly in .grad. requires_grad=True requests derivative tracking.

Tensor Meaning Shape Initial values
w_hidden Input weights w_1,w_2 (2, 1) 0.5, 0.25
b_hidden Hidden bias b_{\mathrm h} (1,) 0
w_output Output weight v (1, 1) 2
b_output Output bias b_{\mathrm o} (1,) 0

These are four tensors containing five scalar parameters. The dictionary groups them by name. This explicit matrix implementation stores weights as (inputs, units) for X @ W; nn.Linear.weight uses the transposed (outputs, inputs) layout.

def make_parameters(hidden_bias=0.0):
    return {
        "w_hidden": torch.tensor(
            [[0.5], [0.25]], dtype=DTYPE, requires_grad=True),
        "b_hidden": torch.tensor(
            [hidden_bias], dtype=DTYPE, requires_grad=True),
        "w_output": torch.tensor(
            [[2.0]], dtype=DTYPE, requires_grad=True),
        "b_output": torch.tensor(
            [0.0], dtype=DTYPE, requires_grad=True),
    }

params = make_parameters()
for name, parameter in params.items():
    print(name, tuple(parameter.shape), parameter.detach().tolist())
w_hidden (2, 1) [[0.5], [0.25]]
b_hidden (1,) [0.0]
w_output (1, 1) [[2.0]]
b_output (1,) [0.0]

Define the Forward Calculation

The hidden unit computes z=x_1w_1+x_2w_2+b_{\mathrm h} and h=\operatorname{ReLU}(z). The rectified linear unit (ReLU) keeps positive values and replaces negative values with zero. The output is \hat y=hv+b_{\mathrm o}.

@ is matrix multiplication. Adding a one-dimensional bias broadcasts it over examples. The function returns the preactivation (the weighted sum before ReLU), hidden value, and prediction so that we can inspect each stage.

def forward(inputs, parameters):
    z_hidden = inputs @ parameters["w_hidden"]
    z_hidden = z_hidden + parameters["b_hidden"]
    hidden = torch.relu(z_hidden)
    prediction = hidden @ parameters["w_output"]
    prediction = prediction + parameters["b_output"]
    return z_hidden, hidden, prediction

Inspect the Intermediate Values

Before any update, z=0.5\cdot1+0.25\cdot2=1, h=1, and \hat y=2 minutes. The target remains 4 minutes. Computing this forward pass changes no parameter.

z_hidden, hidden, prediction = forward(X, params)
print("Hidden weighted sum:", z_hidden.item())
print("Hidden ReLU output:", hidden.item())
print("Prediction before update:", prediction.item())
Hidden weighted sum: 1.0
Hidden ReLU output: 1.0
Prediction before update: 2.0

Define Mean Squared Error

Mean squared error (MSE) averages squared prediction errors. We require matching prediction and target shapes to avoid accidental broadcasting. For this one-example, one-output case, L=(\hat y-y)^2=(2-4)^2=4.

There is no factor of one half, matching the slides. For m averaged scalar output entries, \partial L/\partial\hat y_i=2(\hat y_i-y_i)/m. prediction.numel() counts those entries.

def mse_loss(prediction, target):
    if prediction.shape != target.shape:
        raise ValueError("Prediction and target shapes must match")
    return (prediction - target).square().mean()

loss_before = mse_loss(prediction, y).item()
print("Loss before update:", loss_before)
Loss before update: 4.0

Ask Autograd for Derivatives

Autograd differentiates the recorded operations. zero_grad(set_to_none=True) clears previous gradients; a fresh forward pass constructs a new graph; backward() computes derivatives. It does not update the parameters.

Stochastic gradient descent (SGD) is the optimizer used later for the update. We clone the automatic gradients so that clearing .grad later will not erase our comparison values. detach() removes gradient history; clone() makes an independent copy.

optimizer = torch.optim.SGD(params.values(), lr=learning_rate)
optimizer.zero_grad(set_to_none=True)
before = {name: p.detach().clone() for name, p in params.items()}
_, _, prediction = forward(X, params)
loss = mse_loss(prediction, y)
loss.backward()
automatic = {name: p.grad.detach().clone()
             for name, p in params.items()}
unchanged = all(torch.equal(before[name], p)
                for name, p in params.items())
assert unchanged
print("Parameters unchanged after backward:", unchanged)
Parameters unchanged after backward: True

Differentiate the Same Operations by Hand

The chain rule multiplies derivatives along a path and adds contributions when paths merge. For this example, the output derivative is g_{\mathrm o}=2(\hat y-y)=-4. The output-weight derivative is hg_{\mathrm o}=-4, and the output-bias derivative is g_{\mathrm o}=-4.

The hidden preactivation derivative is g_{\mathrm h}=g_{\mathrm o}v\,\mathbb{1}[z>0]=-8. Here \mathbb{1}[z>0] is 1 when the condition is true and 0 otherwise. Thus the two input-weight derivatives are -8 and -16, and the hidden-bias derivative is -8.

The function uses matrix products and sums over examples, rather than relying on scalar-only multiplication. Summing bias derivatives with sum(dim=0) gives the same shape as each bias parameter. At zero ReLU has no unique mathematical derivative; PyTorch uses zero. no_grad() prevents this manual check from building a second gradient graph.

def manual_gradients(inputs, target, parameters):
    with torch.no_grad():
        z, hidden, prediction = forward(inputs, parameters)
        if prediction.shape != target.shape:
            raise ValueError("Prediction and target shapes must match")
        d_prediction = 2 * (prediction - target)
        d_prediction = d_prediction / prediction.numel()
        d_hidden = d_prediction @ parameters["w_output"].T
        relu_slope = (z > 0).to(d_hidden.dtype)
        d_z = d_hidden * relu_slope
        return {
            "w_hidden": inputs.T @ d_z,
            "b_hidden": d_z.sum(dim=0),
            "w_output": hidden.T @ d_prediction,
            "b_output": d_prediction.sum(dim=0),
        }

Compare Values and Shapes

A shape mismatch must fail even if broadcasting could make two tensors look numerically close. First check exact shapes, then compare values within floating-point tolerances using torch.testing.assert_close. A failed assertion stops execution instead of silently accepting a wrong derivative.

Expected gradients are w_hidden = [[-8], [-16]], b_hidden = [-8], w_output = [[-4]], and b_output = [-4].

manual = manual_gradients(X, y, params)
for name, parameter in params.items():
    assert manual[name].shape == parameter.shape
    assert automatic[name].shape == parameter.shape
    torch.testing.assert_close(
        manual[name], automatic[name], rtol=1e-9, atol=1e-10)
    print(name, "shape:", tuple(parameter.shape))
    print("  autograd:", automatic[name].tolist())
    print("  manual:  ", manual[name].tolist())
print("All gradient values and shapes match.")
w_hidden shape: (2, 1)
  autograd: [[-8.0], [-16.0]]
  manual:   [[-8.0], [-16.0]]
b_hidden shape: (1,)
  autograd: [-8.0]
  manual:   [-8.0]
w_output shape: (1, 1)
  autograd: [[-4.0]]
  manual:   [[-4.0]]
b_output shape: (1,)
  autograd: [-4.0]
  manual:   [-4.0]
All gradient values and shapes match.

Apply One Update, Then Recompute

For each parameter \theta, gradient descent uses \theta_{\mathrm{new}}=\theta-\eta\,\partial L/\partial\theta, where \eta=0.01. All gradients above use the old parameter values. optimizer.step() applies those automatic gradients; the manual gradients are only a check, not a second update.

The new values are w_1=0.58, w_2=0.41, b_{\mathrm h}=0.08, v=2.04, and b_{\mathrm o}=0.04. A fresh forward pass gives z=h=1.48, \hat y=3.0592 minutes, and loss 0.88510464. inference_mode() avoids recording derivatives during this measurement.

optimizer.step()
optimizer.zero_grad(set_to_none=True)
for name, parameter in params.items():
    print(name, parameter.detach())
with torch.inference_mode():
    new_z, new_hidden, new_prediction = forward(X, params)
    loss_after = mse_loss(new_prediction, y).item()
print(f"Hidden value after update: {new_hidden.item():.4f}")
print(f"Prediction after update: {new_prediction.item():.4f}")
print(f"Loss before: {loss_before:.8f}")
print(f"Loss after:  {loss_after:.8f}")
assert loss_after < loss_before
w_hidden tensor([[0.5800],
        [0.4100]], dtype=torch.float64)
b_hidden tensor([0.0800], dtype=torch.float64)
w_output tensor([[2.0400]], dtype=torch.float64)
b_output tensor([0.0400], dtype=torch.float64)
Hidden value after update: 1.4800
Prediction after update: 3.0592
Loss before: 4.00000000
Loss after:  0.88510464

A Separate Case: An Inactive ReLU

Start from the original parameters, changing only the hidden bias to -2. Now z=-1, h=0, and \hat y=0. The hidden unit passes no gradient on this input. Both input weights, the hidden bias, and the output weight have zero gradients; the output bias still has gradient 2(0-4)=-8.

This fresh parameter dictionary is independent of the one just updated. A unit inactive on one input is not necessarily inactive on all possible inputs.

inactive = make_parameters(hidden_bias=-2.0)
inactive_z, inactive_h, inactive_prediction = forward(X, inactive)
inactive_loss = mse_loss(inactive_prediction, y)
inactive_loss.backward()
print("Preactivation:", inactive_z.item())
print("Hidden output:", inactive_h.item())
print("Prediction:", inactive_prediction.item())
for name, parameter in inactive.items():
    print(name, "gradient:", parameter.grad.tolist())
Preactivation: -1.0
Hidden output: 0.0
Prediction: 0.0
w_hidden gradient: [[0.0], [0.0]]
b_hidden gradient: [0.0]
w_output gradient: [[0.0]]
b_output gradient: [-8.0]

Further Experiment: Learning Rate

A correct gradient does not guarantee that an arbitrarily large step will reduce the loss. This helper starts from the same original parameters for every rate, performs one update, and returns the loss before and after. It does not continue training from a previous rate’s result.

def one_step(rate):
    local_params = make_parameters()
    local_optimizer = torch.optim.SGD(local_params.values(), lr=rate)
    local_optimizer.zero_grad(set_to_none=True)
    _, _, local_prediction = forward(X, local_params)
    local_loss = mse_loss(local_prediction, y)
    initial_loss = local_loss.item()
    local_loss.backward()
    local_optimizer.step()
    with torch.inference_mode():
        _, _, updated_prediction = forward(X, local_params)
        updated_loss = mse_loss(updated_prediction, y).item()
    return initial_loss, updated_loss

Compare the Rates

All rows start at loss 4. The rate 0.01 helps here, while 0.1 overshoots and increases loss. These results describe one constructed update, not a generally optimal learning rate.

for rate in (0.001, 0.01, 0.1):
    initial, updated = one_step(rate)
    print(f"lr={rate:g}: {initial:.6f} -> {updated:.6f}")
lr=0.001: 4.000000 -> 3.594088
lr=0.01: 4.000000 -> 0.885105
lr=0.1: 4.000000 -> 106.502400

Check Your Understanding

  1. Which call computes gradients, which updates parameters, and which clears gradients?
  2. Why do the manual bias derivatives use a sum, and why must their shapes match?
  3. Why can the output bias still learn when the hidden ReLU is inactive?
  4. Why divide the MSE derivative by the number of prediction elements?
Answers and discussion
  1. The operations are separate. loss.backward() accumulates derivatives in .grad. optimizer.step() changes the parameters. optimizer.zero_grad(set_to_none=True) clears stored gradients. no_grad() and inference_mode() control derivative recording; neither updates or clears parameters by itself.
  2. A shared bias contributes to every example. Its derivative adds contributions across the batch. sum(dim=0) removes the example axis and returns the bias parameter’s exact shape. A shape check prevents broadcasting from hiding a wrong matrix or batch dimension.
  3. There is a direct path from output bias to prediction. Its local derivative is 1, regardless of the hidden activation. In the inactive case its gradient is -8. The output-weight gradient is zero because its input is zero, and the hidden-parameter gradients are blocked by ReLU’s zero slope at z=-1.
  4. The derivative preserves the average. MSE averages over m scalar output entries, so each derivative is 2(\hat y_i-y_i)/m. Omitting the divisor differentiates a sum instead and changes the update’s scale. Here m=1.

Takeaway: backpropagation computes derivatives; an optimizer uses them. Lower loss on one example is not proof of performance on new examples.

References: Official PyTorch Autograd Tutorial, MSELoss, and SGD.