神经网络中的前向传播与后向传播
$ f(z) $为激励函数，关于激励函数(又称激活函数)的总结
隐藏层1输入
z^{(1)}=W^{(1)}x^T+b^{(1)}\tag{1}
隐藏层1输出
n^{(1)}=f^{(1)}(z^{(1)})\tag{2}
隐藏层2输入
z^{(2)}=W^{(2)}n^{(1)}+b^{(2)}\tag{3}
隐藏层2输出
n^{(2)}=f^{(2)}(z^{(2)})\tag{4}
隐藏层3输入
z^{(3)}=W^{(3)}n^{(2)}+b^{(3)}\tag{5}
隐藏层3输出即输出层
\widehat y = n^{(3)}= f^{(3)}(z{(3)})\tag{6}
损失函数
L(y,\widehat y)\tag{7}
即隐藏层k+1输入
z^{(k+1)}=W^{(k+1)}n^{(k)}+b^{(k+1)}\tag{8}
隐藏层k+1输出
n^{(k+1)}= f^{(k+1)}(z{(k+1)})\tag{9}
对损失函数进行总结https://blog.csdn.net/lien0906/article/details/78429768
计算偏导数
\frac {\partial z^{(k)}}{\partial b^{(k)}}=
diag(1,1, \ldots ,1)\tag{10}
列向量对列向量求导参见矩阵中的求导
计算偏导数$\frac {\partial L(y,\widehat y)}{\partial z^{(k)}}$
偏导数$ \frac {\partial L(y,\widehat y)}{\partial z^{(k)}}\ $ 又称误差项(error term,也称"灵敏度"),一般用$ \delta $ 表示,用$ \delta^{(k)} $ 表示第k层神经元的误差项,其值的大小代表了第k层神经元对最终总误差的影响大小
\begin{align}
\delta^{(k)} & = \frac {\partial L(y,\widehat y)}{\partial z^{(k)}}\cr
& =\frac {\partial n^{(k)}}{\partial z^{(k)}}*
\frac {\partial z^{(k+1)}}{\partial n^{(k)}}*
\frac {\partial L(y,\widehat y)}{\partial z^{(k+1)}}\cr
& = {f^{(k)}}^{'}(z^{(k)}) * (W^{(k+1)})^T * \delta^{(k+1)}
\end{align}\tag{11}
最终需要用的两个导数
\frac {\partial L(y,\widehat y)}{\partial W^{(k)}}
=\frac {\partial L(y,\widehat y)}{\partial z^{(k)}}*
\frac {\partial z^{(k)}}{\partial W^{(k)}}
=\delta^{(k)}*(n^{(k-1)})^T\tag{12}
\frac {\partial L(y,\widehat y)}{\partial b^{(k)}}
=\frac {\partial L(y,\widehat y)}{\partial z^{(k)}}*
\frac {\partial z^{(k)}}{\partial b^{(k)}}
=\delta^{(k)}\tag{13}
后向传播参数更新
W^{(k)} = W^{(k)} - \alpha(\delta^{(k)}(n^{(k-1)})^T + W^{(k)})\tag{14}
b^{(k)} = b^{(k)}-\alpha\delta^{(k)}\tag{15}
其中$ \alpha $ 是学习率
后向传播中的正则化,L1正则化,L2正则化