当前位置：首页 > news >正文

神经网络中标量求导和向量求导

news 2025/8/22 9:19:59

0. 引出问题

在神经网络反向传播过程中 loss = [loss₁,loss₂, loss₃]，为什么 ∂loss/∂w

∂loss₁/∂w 
∂loss₂/∂w
∂loss₃/∂w

∂loss₁/∂w 和 loss 维度一样都是三位向量，[∂loss₁/∂w, ∂loss₂/∂w, ∂loss₃/∂w] 就变成3*3的矩阵
如下所示：

import torchw = torch.tensor([1.0, 2.0,3.0], requires_grad=True)
loss = w * 3  
print("loss: \n", loss)loss_m = []for i, val in enumerate(loss):w.grad = None  # 清零val.backward(retain_graph=True)print(f"∂loss{i+1}/∂w = {w.grad}")loss_m.append(w.grad.clone())print("loss_m: \n", torch.stack(loss_m))

输出结果：

loss: tensor([3., 6., 9.], grad_fn=<MulBackward0>)∂loss1/∂w = tensor([3., 0., 0.])
∂loss2/∂w = tensor([0., 3., 0.])
∂loss3/∂w = tensor([0., 0., 3.])loss_m: tensor([[3., 0., 0.],[0., 3., 0.],[0., 0., 3.]])

loss: tensor([3., 6., 9.]) 为向量，对w求导时为矩阵
但是 w.grad 必须是标量或张量，不能是向量矩阵

1. 标量求导

import torchw = torch.tensor([1.0, 2.0,3.0], requires_grad=True)
loss = w * 3  
print("loss: \n", loss)loss_m = []
# 方法1：分别计算
for i, val in enumerate(loss):w.grad = None  # 清零val.backward(retain_graph=True)print(f"∂loss{i+1}/∂w = {w.grad}")loss_m.append(w.grad.clone())print("loss_m: \n", torch.stack(loss_m))grads = torch.autograd.grad(loss.sum(), w,retain_graph=True)
print("grads: \n", grads)  grads1 = torch.autograd.grad(loss.mean(), w)[0]
print("grads1: \n", grads1)

输出；

loss: tensor([3., 6., 9.], grad_fn=<MulBackward0>)
∂loss1/∂w = tensor([3., 0., 0.])
∂loss2/∂w = tensor([0., 3., 0.])
∂loss3/∂w = tensor([0., 0., 3.])
loss_m: tensor([[3., 0., 0.],[0., 3., 0.],[0., 0., 3.]])
grads: (tensor([3., 3., 3.]),)
grads1: tensor([1., 1., 1.])

同样的例子：

import torch# 3个样本的真实数据
x = torch.tensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], requires_grad=True)
y_true = torch.tensor([1.0, 2.0, 3.0])# 线性模型：y = w₁x₁ + w₂x₂
w = torch.tensor([0.5, 0.5], requires_grad=True)
predictions = (x @ w)  # [1.5, 3.5, 5.5]
print("预测值:", predictions)
# 计算每个样本的梯度
individual_grads = []
for i in range(3):loss = (predictions[i] - y_true[i])**2loss.backward(retain_graph=True)individual_grads.append(w.grad.clone())w.grad.zero_()print("样本1梯度:", individual_grads[0]) 
print("样本2梯度:", individual_grads[1])  
print("样本3梯度:", individual_grads[2])  # 标量梯度：自动综合
total_loss = ((predictions - y_true)**2).mean()
total_loss.backward()# 验证：标量梯度 = 向量梯度的平均
manual_average = (individual_grads[0] + individual_grads[1] + individual_grads[2]) / 3print("手动平均:", manual_average)  
print("标量结果:", w.grad)

输出结果：

预测值: tensor([1.5000, 3.5000, 5.5000], grad_fn=<MvBackward0>)
样本1梯度: tensor([1., 2.])
样本2梯度: tensor([ 9., 12.])
样本3梯度: tensor([25., 30.])
手动平均: tensor([11.6667, 14.6667])
标量结果: tensor([11.6667, 14.6667])

训练神经网络是为了最小化整体损失，不是单独优化每个样本

# 实际训练：最小化平均损失
batch_loss = individual_losses.mean()  # 标量
batch_loss.backward()  # 得到平均梯度
optimizer.step()       # 朝平均最优方向更新

2. 什么时候需要向量梯度？

仅用于研究：分析样本敏感性

def compute_sample_gradients(model, x, y):"""仅用于分析，不用于训练"""grads = []for xi, yi in zip(x, y):model.zero_grad()pred = model(xi.unsqueeze(0))loss = ((pred - yi) ** 2)loss.backward()grads.append(model.weight.grad.clone())return grads  # 每个样本的单独梯度