#神经网络中的前向传播与后向传播

$f(z)$为激励函数，关于激励函数(又称激活函数)的总结
隐藏层1输入

$$z^{(1)}=W^{(1)}x^T+b^{(1)}\tag{1}$$

隐藏层1输出

$$n^{(1)}=f^{(1)}(z^{(1)})\tag{2}$$

隐藏层2输入

$$z^{(2)}=W^{(2)}n^{(1)}+b^{(2)}\tag{3}$$

隐藏层2输出

$$n^{(2)}=f^{(2)}(z^{(2)})\tag{4}$$

隐藏层3输入

$$z^{(3)}=W^{(3)}n^{(2)}+b^{(3)}\tag{5}$$

隐藏层3输出即输出层

$$\widehat y = n^{(3)}= f^{(3)}(z{(3)})\tag{6}$$

损失函数

$$L(y,\widehat y)\tag{7}$$

即隐藏层k+1输入

$$z^{(k+1)}=W^{(k+1)}n^{(k)}+b^{(k+1)}\tag{8}$$

隐藏层k+1输出

$$n^{(k+1)}= f^{(k+1)}(z{(k+1)})\tag{9}$$

对损失函数进行总结https://blog.csdn.net/lien0906/article/details/78429768
计算偏导数

$$\frac {\partial z^{(k)}}{\partial b^{(k)}}= diag(1,1, \ldots ,1)\tag{10}$$

列向量对列向量求导参见矩阵中的求导

计算偏导数\frac {\partial L(y,\widehat y)}{\partial z^{(k)}}\

偏导数\frac {\partial L(y,\widehat y)}{\partial z^{(k)}}\ 又称误差项(error term,也称"灵敏度"),一般用$\delta$ 表示,用$\delta^{(k)}$ 表示第k层神经元的误差项,其值的大小代表了第k层神经元对最终总误差的影响大小

$$\begin{align} \delta^{(k)} & = \frac {\partial L(y,\widehat y)}{\partial z^{(k)}}\cr & =\frac {\partial n^{(k)}}{\partial z^{(k)}}* \frac {\partial z^{(k+1)}}{\partial n^{(k)}}* \frac {\partial L(y,\widehat y)}{\partial z^{(k+1)}}\cr & = {f^{(k)}}^{'}(z^{(k)}) * (W^{(k+1)})^T * \delta^{(k+1)} \end{align}\tag{11}$$

最终需要用的两个导数

$$\frac {\partial L(y,\widehat y)}{\partial W^{(k)}} =\frac {\partial L(y,\widehat y)}{\partial z^{(k)}}* \frac {\partial z^{(k)}}{\partial W^{(k)}} =\delta^{(k)}*(n^{(k-1)})^T\tag{12}$$ $$\frac {\partial L(y,\widehat y)}{\partial b^{(k)}} =\frac {\partial L(y,\widehat y)}{\partial z^{(k)}}* \frac {\partial z^{(k)}}{\partial b^{(k)}} =\delta^{(k)}\tag{13}$$

后向传播参数更新

$$W^{(k)} = W^{(k)} - \alpha(\delta^{(k)}(n^{(k-1)})^T + W^{(k)})\tag{14}$$ $$b^{(k)} = b^{(k)}-\alpha\delta^{(k)}\tag{15}$$

其中$\alpha$ 是学习率

后向传播中的正则化,L1正则化,L2正则化