transformer 编码器层

编码器层

目标

• 了解编编码器层的作用
• 掌握编码器层的实现过程

作用
作为编码器的组成单元, 每个编码器完成一次对输入特征提取的过程, 即编码过程

代码实现

import torch.nn as nnfrom transformer_test.attention import clones
from transformer_test.sub_layer_connection import SubLayerConnectionclass EncoderLayer(nn.Module):def __init__(self, size, self_attn, feed_forward, dropout):super(EncoderLayer, self).__init__()self.self_attn = self_attnself.feed_forward = feed_forwardself.dropout = dropoutself.laysers = clones(SubLayerConnection(size, dropout), 2)self.size = self.sizedef forward(self, x, mask):x = self.laysers[0](x, lambda x: self.self_attn(x, x, x, mask))return self.laysers[1](x, self.feed_forward)

编码器

目标

• 了解编码器的作用
• 掌握编码器的实现过程

作用
编码器用于对输入进行指定的特征提取过程, 也称为编码, 由N个编码器堆叠而成

代码实现

import torch.nn as nnfrom transformer_test.attention import clones
from transformer_test.layer_norm import LayerNormclass Encoder(nn.Module):def __init__(self, layer, N):""":param layer: 编码器层:param N: 编码器层的个数"""super(Encoder, self).__init__()# clones 函数克隆N个编码器self.layers = clones(layer, N)# 初始化规范化层, 用在编码器最后self.norm = LayerNorm(layer.size)def forward(self, x, mask):""":param x: 输入:param mask: 屏蔽:return:"""# 循环执行N个编码器for layer in self.layers:x = layer(x, mask)return self.norm(x)