[深度学习网络从入门到入土] 门控循环单元GRU个人导航知乎https://www.zhihu.com/people/byzh_rcCSDNhttps://blog.csdn.net/qq_54636039注本文仅对所述内容做了框架性引导具体细节可查询其余相关资料or源码参考文章各方资料文章目录[深度学习网络从入门到入土] 门控循环单元GRU个人导航参考资料背景⚙️架构(公式)1. 更新门(Update Gate) 重置门(Reset Gate)2. 候选隐藏状态3. 最终隐藏状态4. GRU结构图优点/创新点1. 结构更简单2. 能学习长程依赖缺点1. 表达能力略弱于 LSTM2. 极长序列训练困难代码实现项目实例参考资料Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.背景经典 RNN 存在一个非常严重的问题长序列训练时容易出现梯度消失 / 梯度爆炸因此LSTM被提出用三个门 memory cell来解决长期依赖问题但是 LSTM 结构较复杂参数多计算量大推理较慢- 一种更简单的结构GRUGated Recurrent Unit核心思想:用更少的门控结构实现接近 LSTM 的效果GRU 的主要改进只有两个门不再单独维护cell state参数更少, 训练更快⚙️架构(公式)1. 更新门(Update Gate) 重置门(Reset Gate)更新门控制决定最终 hidden state 用多少新信息z t σ ( W x z x t W h z h t − 1 b z ) z_t \sigma(W_{xz}x_t W_{hz}h_{t-1} b_z)zt​σ(Wxz​xt​Whz​ht−1​bz​)z t ≈ 1 z_t \approx 1zt​≈1→ 保留旧记忆z t ≈ 0 z_t \approx 0zt​≈0→ 使用新信息重置门控制决定生成新状态时要不要参考历史r t σ ( W x r x t W h r h t − 1 b r ) r_t \sigma(W_{xr}x_t W_{hr}h_{t-1} b_r)rt​σ(Wxr​xt​Whr​ht−1​br​)r t ≈ 0 r_t \approx 0rt​≈0→ 忘掉过去信息r t ≈ 1 r_t \approx 1rt​≈1→ 使用历史信息2. 候选隐藏状态候选状态h ~ t tanh ⁡ ( W x h x t W h h ( r t ⊙ h t − 1 ) b h ) \tilde{h}_t \tanh(W_{xh}x_t W_{hh}(r_t \odot h_{t-1}) b_h)h~t​tanh(Wxh​xt​Whh​(rt​⊙ht−1​)bh​)注意历史状态先经过reset gate门负责“控制比例”所以用 sigmoid - 限制在0~1候选隐藏状态负责“产生内容”所以用 tanh - 有正有负, 没有偏移问题3. 最终隐藏状态最终输出由update gate决定h t ( 1 − z t ) ⊙ h t − 1 z t ⊙ h ~ t h_t (1 - z_t)\odot h_{t-1} z_t \odot \tilde{h}_tht​(1−zt​)⊙ht−1​zt​⊙h~t​可以理解为旧状态 新状态的加权平均4. GRU结构图h_{t-1} │ ┌────┴────┐ │ │ update reset gate gate │ │ │ ▼ │ candidate │ │ └────┬────┘ ▼ h_t优点/创新点1. 结构更简单模型门数量RNN0GRU2LSTM3- 参数更少, 计算更快 - 收敛更快, 推理更快2. 能学习长程依赖通过update gate模型可以直接保留旧状态h t ≈ h t − 1 h_t \approx h_{t-1}ht​≈ht−1​这样可以缓解梯度消失问题缺点1. 表达能力略弱于 LSTMGRU 没有独立的 cell state, 只有h t h_tht​- 某些复杂任务中 LSTM 可能更强2. 极长序列训练困难这也是后来Transformer出现的重要原因代码实现importtorchimporttorch.nnasnnimporttorch.nn.functionalasFfrombyzh.ai.Butilsimportb_get_paramsclassB_GRU_Paper(nn.Module): 单层 GRU教学/论文公式对齐版 每个时间步: z_t sigmoid(x_t W_xz h_{t-1} W_hz b_z) r_t sigmoid(x_t W_xr h_{t-1} W_hr b_r) h~_t tanh( x_t W_xh (r_t * h_{t-1}) W_hh b_h ) h_t (1 - z_t) * h_{t-1} z_t * h~_t B batch size T sequence length D input_size H hidden_size C output_size def__init__(self,input_size:int,hidden_size:int,output_size:intNone,batch_first:boolTrue,bias:boolTrue):super().__init__()self.input_sizeinput_size self.hidden_sizehidden_size self.output_sizeoutput_size self.batch_firstbatch_first# 更新门: update gateself.x2znn.Linear(input_size,hidden_size,biasbias)self.h2znn.Linear(hidden_size,hidden_size,biasbias)# 重置门: reset gateself.x2rnn.Linear(input_size,hidden_size,biasbias)self.h2rnn.Linear(hidden_size,hidden_size,biasbias)# 候选隐藏状态: candidate hidden stateself.x2hnn.Linear(input_size,hidden_size,biasbias)self.h2hnn.Linear(hidden_size,hidden_size,biasbias)# (可选)输出映射: optional output projectionself.h2ynn.Linear(hidden_size,output_size,biasbias)ifoutput_sizeisnotNoneelseNoneself.reset_parameters()defstep(self,x_t,h_prev): x_t: (B, D) h_prev: (B, H) z_ttorch.sigmoid(self.x2z(x_t)self.h2z(h_prev))# 更新门r_ttorch.sigmoid(self.x2r(x_t)self.h2r(h_prev))# 重置门h_hattorch.tanh(self.x2h(x_t)self.h2h(r_t*h_prev))# 候选隐藏状态h_t(1-z_t)*h_prevz_t*h_hat# 更新隐藏状态returnh_tdefforward(self,x,h0None,return_sequencesTrue): x: batch_firstTrue - (B, T, D) batch_firstFalse - (T, B, D) h0: (B, H) return: hs: (B, T, H) or (T, B, H) or None hT: (B, H) ys: (B, T, C) or (T, B, C) or None ifnotself.batch_first:xx.transpose(0,1)# - (B, T, D)B,T,Dx.shape h_tx.new_zeros(B,self.hidden_size)ifh0isNoneelseh0# (B, H)hs[]ifreturn_sequenceselseNoneys[]if(return_sequencesandself.h2yisnotNone)elseNonefortinrange(T):x_tx[:,t,:]h_tself.step(x_t,h_t)ifreturn_sequences:hs.append(h_t)ifself.h2yisnotNone:ys.append(self.h2y(h_t))hTh_tifreturn_sequences:hstorch.stack(hs,dim1)# (B, T, H)ifysisnotNone:ystorch.stack(ys,dim1)# (B, T, C)ifnotself.batch_first:hshs.transpose(0,1)ifysisnotNone:ysys.transpose(0,1)returnhs,hT,ysdefreset_parameters(self):modules[self.x2z,self.h2z,self.x2r,self.h2r,self.x2h,self.h2h]forminmodules:nn.init.xavier_uniform_(m.weight)ifm.biasisnotNone:nn.init.zeros_(m.bias)ifself.h2yisnotNone:nn.init.xavier_uniform_(self.h2y.weight)ifself.h2y.biasisnotNone:nn.init.zeros_(self.h2y.bias)classB_GRU_Paper_Layers(nn.Module): 多层 GRU通过堆叠多个 B_GRU_Paper 实现 def__init__(self,input_size,hidden_size,num_layers2,output_sizeNone,batch_firstTrue):super().__init__()self.num_layersnum_layers self.batch_firstbatch_first layers[]foriinrange(num_layers):ifi0:in_diminput_sizeelse:in_dimhidden_size# 最后一层才接输出out_dimoutput_sizeifinum_layers-1elseNonelayers.append(B_GRU_Paper(input_sizein_dim,hidden_sizehidden_size,output_sizeout_dim,batch_firstbatch_first))self.layersnn.ModuleList(layers)defforward(self,x): return: hs: 最后一层所有时间步隐藏状态 hT_list: 每一层最后时刻隐藏状态列表 ys: 最后一层每个时间步输出如果最后一层有 output_size hsx hT_list[]ysNonefori,layerinenumerate(self.layers):hs,hT,yslayer(hs,return_sequencesTrue)hT_list.append(hT)returnhs,hT_list,ysif__name____main__:# 超参数 B50# batch sizeT6# sequence lengthD8# input_sizeH16# hidden_sizeC5# output_sizenetB_GRU_Paper(D,H,C,batch_firstTrue)atorch.randn(B,T,D)hs,hT,ysnet(a)print(hT.shape)print(f参数量:{b_get_params(net)})# 1_333项目实例库环境:numpy1.26.4 torch2.2.2cu121 byzh-core0.0.9.21 byzh-ai0.0.9.61 byzh-extra0.0.9.12 ...GRU训练MNIST数据集:# copy all the codes from here to runimporttorchimporttorch.nnasnnimporttorch.nn.functionalasFfrombyzh.ai.BtrainerimportB_Classification_Trainerfrombyzh.ai.BdataimportB_Download_MNIST,b_get_dataloader_from_tensor,b_stratified_indices# from uploadToPypi_ai.byzh.ai.Bmodel.study_rnn import B_GRU_Paperfrombyzh.ai.Bmodel.study_rnnimportB_GRU_Paperfrombyzh.ai.Butilsimportb_get_device##### hyper params #####epochs10lr1e-3batch_size128deviceb_get_device(use_idle_gpuTrue)##### data #####downloaderB_Download_MNIST(save_dirD:/study_model/datasets/MNIST)data_dictdownloader.get_data()X_traindata_dict[X_train_standard]y_traindata_dict[y_train]X_testdata_dict[X_test_standard]y_testdata_dict[y_test]num_classesdata_dict[num_classes]train_dataloader,val_dataloaderb_get_dataloader_from_tensor(X_train,y_train,X_test,y_test,batch_sizebatch_size)##### model #####classMNIST_PixelGRU(nn.Module): 用 GRU 做 pixel-wise MNIST 分类。 输入: x: (B, 1, 28, 28) 处理流程: (B, 1, 28, 28) - reshape - (B, 28, 28) - RNN - 取最终隐藏状态 hT - Linear(H, 10) - logits: (B, 10) def__init__(self,hidden_size128,num_classes10):super().__init__()self.gruB_GRU_Paper(input_size28,# 每个时间步有28个像素值hidden_sizehidden_size,output_sizeNone,# backbone 先不直接输出类别batch_firstTrue,)self.clsnn.Linear(hidden_size,num_classes,biasTrue)self.reset_parameters()defreset_parameters(self):nn.init.xavier_uniform_(self.cls.weight)ifself.cls.biasisnotNone:nn.init.zeros_(self.cls.bias)defforward(self,x): x: (B, 1, 28, 28) return: logits: (B, 10) # (B, 1, 28, 28) - (B, 28, 28)xx.squeeze(1)# RNN 编码hs,hT,ysself.gru(x,return_sequencesFalse)# 最终分类logitsself.cls(hT)# (B, 10)returnlogits modelMNIST_PixelGRU(num_classesnum_classes)##### else #####optimizertorch.optim.Adam(model.parameters(),lrlr)criteriontorch.nn.CrossEntropyLoss()##### trainer #####trainerB_Classification_Trainer(modelmodel,optimizeroptimizer,criterioncriterion,train_loadertrain_dataloader,val_loaderval_dataloader,devicedevice)trainer.set_writer1(./runs/gru/log.txt)##### run #####trainer.train_eval_s(epochsepochs)##### calculate #####trainer.draw_loss_acc(./runs/gru/loss_acc.png,y_limFalse)trainer.save_best_checkpoint(./runs/gru/best_checkpoint.pth)trainer.calculate_model()