当前位置：首页 > ai >正文

BERT预训练

ai 2025/5/8 21:57:38

BERT

NLP里的迁移学习

使用预训练好的模型来抽取词、句子的特征
- 例如 word2vec 或语言模型
不更新预训练好的模型
需要构建新的网络来抓取新任务需要的信息
- Word2vec 忽略了时序信息，语言模型只看了一个方向

BERT 动机

在这里插入图片描述

BERT 架构

只有编码器的 Transformer
两个版本：
- Base: #blocks = 12, hidden size = 768, #heads = 12, #parameters = 110M
- Large: #blocks = 24, hidden size = 1024, #heads = 16, #parameter = 340M
在大规模数据上训练 > 3B 词

对输入的修改

每个样本是一个句子对
加入额外的片段嵌入
位置编码可学习

在这里插入图片描述

预训练任务1：带掩码的语言模型

Transfomer的编码器是双向，标准语言模型要求单向
带掩码的语言模型每次随机（15%概率）将一些词元换成 < mask >
因为微调任务中不出现< mask >
- 80%概率下，将选中的词元变成< mask >
- 10%概率下换成一个随机词元
- 10%概率下保持原有的词元

预训练任务2：下一句子预测

预测一个句子对中两个句子是不是相邻
训练样本中：
- 50%概率选择相邻句子对：< cls > this movie is great < sep > i like it < sep >
- 50%概率选择随机句子对：< cls > this movie is great < sep > hello world < sep >
将< cls >对应的输出放到一个全连接层来预测

总结

BERT针对微调设计
基于Transformer的编码器做了如下修改
- 模型更大，训练数据更多
- 输入句子对，片段嵌入，可学习的位置编码
训练时使用两个任务：
- 带掩码的语言模型
- 下一个句子预测

BERT 代码

Bidirectional Encoder Representations from Transformers (BERT)
首先导入必要的环境

import torch
from torch import nn
from d2l import torch as d2l

输入表示

在自然语言处理中，有些任务（如情感分析）以单个文本作为输入，而有些任务（如自然语言推断）以一对文本序列作为输入。BERT输入序列明确地表示单个文本和文本对。当输入为单个文本时，BERT输入序列是特殊类别词元“<cls>”、文本序列的标记、以及特殊分隔词元“<sep>”的连结。当输入为文本对时，BERT输入序列是“<cls>”、第一个文本序列的标记、“<sep>”、第二个文本序列标记、以及“<sep>”的连结。下面将始终如一地将术语“BERT输入序列”与其他类型的“序列”区分开来。例如，一个BERT输入序列可以包括一个文本序列或两个文本序列。

为了区分文本对，根据输入序列学到的片段嵌入 $\mathbf{e}_A$ 和 $\mathbf{e}_B$ 分别被添加到第一序列和第二序列的词元嵌入中。对于单文本输入，仅使用 $\mathbf{e}_A$ 。下面将一个句子或两个句子作为输入，然后返回BERT输入序列的标记及其相应的片段索引。

def get_tokens_and_segments(tokens_a, tokens_b=None):"""获取输入序列的词元及其片段索引"""tokens = ['<cls>'] + tokens_a + ['<sep>']# 0和1分别标记片段A和Bsegments = [0] * (len(tokens_a) + 2)if tokens_b is not None:tokens += tokens_b + ['<sep>']segments += [1] * (len(tokens_b) + 1)return tokens, segments

BERT 编码器类

class BERTEncoder(nn.Module):"""BERT编码器"""def __init__(self, vocab_size, num_hiddens, norm_shape, ffn_num_input,ffn_num_hiddens, num_heads, num_layers, dropout,max_len=1000, key_size=768, query_size=768, value_size=768,**kwargs):"""初始化BERT编码器参数：vocab_size: 词表大小num_hiddens: 隐藏单元数（嵌入维度）norm_shape: 层归一化的形状ffn_num_input: 前馈网络的输入大小ffn_num_hiddens: 前馈网络的隐藏层大小num_heads: 多头注意力的头数num_layers: 编码器块的层数dropout: Dropout概率max_len: 最大序列长度key_size: 注意力机制中键的维度query_size: 注意力机制中查询的维度value_size: 注意力机制中值的维度"""super(BERTEncoder, self).__init__(**kwargs)# 词元嵌入层，将词元索引映射到嵌入向量self.token_embedding = nn.Embedding(vocab_size, num_hiddens)# 片段嵌入层，用于区分句子A和句子Bself.segment_embedding = nn.Embedding(2, num_hiddens)# 编码器块的堆叠，使用d2l提供的EncoderBlock实现self.blks = nn.Sequential()for i in range(num_layers):self.blks.add_module(f"{i}", d2l.EncoderBlock(key_size, query_size, value_size, num_hiddens, norm_shape,ffn_num_input, ffn_num_hiddens, num_heads, dropout, True))# 可学习的位置嵌入参数，用于表示序列中每个位置的信息# 形状为 (1, max_len, num_hiddens)，其中1表示批量维度self.pos_embedding = nn.Parameter(torch.randn(1, max_len, num_hiddens))def forward(self, tokens, segments, valid_lens):"""前向传播参数：tokens: 输入的词元索引，形状为 (批量大小, 最大序列长度)segments: 输入的片段索引，形状为 (批量大小, 最大序列长度)valid_lens: 有效长度，用于掩蔽填充部分，形状为 (批量大小,)返回：编码后的表示，形状为 (批量大小, 最大序列长度, num_hiddens)"""# 词元嵌入和片段嵌入相加，形状为 (批量大小, 最大序列长度, num_hiddens)X = self.token_embedding(tokens) + self.segment_embedding(segments)# 加上位置嵌入，位置嵌入的形状为 (1, 最大序列长度, num_hiddens)# 通过切片选择与输入序列长度匹配的部分X = X + self.pos_embedding.data[:, :X.shape[1], :]# 依次通过每个编码器块for blk in self.blks:X = blk(X, valid_lens)# 返回编码后的表示return X

假设词表大小为10000，喜爱卖弄演示BERTEncoder的前向推断，创建一个实例并初始化其参数。

vocab_size, num_hiddens, ffn_num_hiddens, num_heads = \10000, 768, 1024, 4
norm_shape, ffn_num_input, num_layers, dropout = [768], 768, 2, 0.2
encoder = BERTEncoder(vocab_size, num_hiddens, norm_shape, ffn_num_input,ffn_num_hiddens, num_heads, num_layers, dropout)tokens = torch.randint(0, vocab_size, (2, 8))
segments = torch.tensor([[0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 1, 1, 1, 1, 1]])
encoded_X = encoder(tokens, segments, None)
encoded_X.shape# torch.Size([2, 8, 768])

遮蔽语言模型

class MaskLM(nn.Module):"""BERT的掩蔽语言模型任务"""def __init__(self, vocab_size, num_hiddens, num_inputs=768, **kwargs):"""初始化掩蔽语言模型（Masked Language Model）参数：vocab_size: 词表大小，用于预测的输出维度num_hiddens: 隐藏单元数，用于MLP的隐藏层num_inputs: 输入特征的维度，默认为768"""super(MaskLM, self).__init__(**kwargs)# 定义一个多层感知机（MLP）用于预测掩蔽词元self.mlp = nn.Sequential(nn.Linear(num_inputs, num_hiddens),  # 全连接层，将输入映射到隐藏层nn.ReLU(),                          # 激活函数，使用ReLUnn.LayerNorm(num_hiddens),          # 层归一化，稳定训练nn.Linear(num_hiddens, vocab_size)  # 输出层，映射到词表大小的维度)def forward(self, X, pred_positions):"""前向传播参数：X: 编码后的BERT表示，形状为 (batch_size, seq_len, num_inputs)pred_positions: 掩蔽词元的位置，形状为 (batch_size, num_pred_positions)返回：mlm_Y_hat: 掩蔽位置的预测结果，形状为 (batch_size, num_pred_positions, vocab_size)"""# 获取每个序列中掩蔽词元的数量num_pred_positions = pred_positions.shape[1]# 将掩蔽位置展平为一维向量，形状为 (batch_size * num_pred_positions,)pred_positions = pred_positions.reshape(-1)# 获取批量大小batch_size = X.shape[0]# 创建一个重复的批量索引，用于从X中提取掩蔽位置的特征# 假设 batch_size=2，num_pred_positions=3# 那么 batch_idx 是 [0, 0, 0, 1, 1, 1]batch_idx = torch.arange(0, batch_size)batch_idx = torch.repeat_interleave(batch_idx, num_pred_positions)# 根据批量索引和掩蔽位置索引，从X中提取掩蔽位置的特征# masked_X 的形状为 (batch_size * num_pred_positions, num_inputs)masked_X = X[batch_idx, pred_positions]# 将提取的特征重新调整为 (batch_size, num_pred_positions, num_inputs)masked_X = masked_X.reshape((batch_size, num_pred_positions, -1))# 使用MLP对掩蔽位置的特征进行预测# mlm_Y_hat 的形状为 (batch_size, num_pred_positions, vocab_size)mlm_Y_hat = self.mlp(masked_X)return mlm_Y_hat

The forward inference of MaskLM
在这里插入图片描述
Next Sentence Prediction

#@save
class NextSentencePred(nn.Module):"""BERT的下一句预测任务"""def __init__(self, num_inputs, **kwargs):super(NextSentencePred, self).__init__(**kwargs)self.output = nn.Linear(num_inputs, 2)def forward(self, X):# X的形状：(batchsize,num_hiddens)return self.output(X)

The forward inference of an NextSentencePred
在这里插入图片描述
整合代码

class BERTModel(nn.Module):"""BERT模型"""def __init__(self, vocab_size, num_hiddens, norm_shape, ffn_num_input,ffn_num_hiddens, num_heads, num_layers, dropout,max_len=1000, key_size=768, query_size=768, value_size=768,hid_in_features=768, mlm_in_features=768,nsp_in_features=768):super(BERTModel, self).__init__()self.encoder = BERTEncoder(vocab_size, num_hiddens, norm_shape,ffn_num_input, ffn_num_hiddens, num_heads, num_layers,dropout, max_len=max_len, key_size=key_size,query_size=query_size, value_size=value_size)self.hidden = nn.Sequential(nn.Linear(hid_in_features, num_hiddens),nn.Tanh())self.mlm = MaskLM(vocab_size, num_hiddens, mlm_in_features)self.nsp = NextSentencePred(nsp_in_features)def forward(self, tokens, segments, valid_lens=None,pred_positions=None):encoded_X = self.encoder(tokens, segments, valid_lens)if pred_positions is not None:mlm_Y_hat = self.mlm(encoded_X, pred_positions)else:mlm_Y_hat = None# 用于下一句预测的多层感知机分类器的隐藏层，0是“<cls>”标记的索引nsp_Y_hat = self.nsp(self.hidden(encoded_X[:, 0, :]))return encoded_X, mlm_Y_hat, nsp_Y_hat

BERT 预训练数据代码

The WikiText-2 dataset

#@save
d2l.DATA_HUB['wikitext-2'] = ('https://s3.amazonaws.com/research.metamind.io/wikitext/''wikitext-2-v1.zip', '3c914d17d80b1459be871a5039ac23e752a53cbe')def _read_wiki(data_dir):file_name = os.path.join(data_dir, 'wiki.train.tokens')with open(file_name, 'r') as f:lines = f.readlines()paragraphs = [line.strip().lower().split(' . ')for line in lines if len(line.split(' . ')) >= 2]random.shuffle(paragraphs)return paragraphs

Generating the Next Sentence Prediction Task

def _get_next_sentence(sentence, next_sentence, paragraphs):if random.random() < 0.5:is_next = Trueelse:# paragraphs是三重列表的嵌套next_sentence = random.choice(random.choice(paragraphs))is_next = Falsereturn sentence, next_sentence, is_nextdef _get_nsp_data_from_paragraph(paragraph, paragraphs, vocab, max_len):nsp_data_from_paragraph = []for i in range(len(paragraph) - 1):tokens_a, tokens_b, is_next = _get_next_sentence(paragraph[i], paragraph[i + 1], paragraphs)# 考虑1个'<cls>'词元和2个'<sep>'词元if len(tokens_a) + len(tokens_b) + 3 > max_len:continuetokens, segments = d2l.get_tokens_and_segments(tokens_a, tokens_b)nsp_data_from_paragraph.append((tokens, segments, is_next))return nsp_data_from_paragraph

Generating the Masked Language Modeling Task

def _replace_mlm_tokens(tokens, candidate_pred_positions, num_mlm_preds,vocab):# 为遮蔽语言模型的输入创建新的词元副本，其中输入可能包含替换的“<mask>”或随机词元mlm_input_tokens = [token for token in tokens]pred_positions_and_labels = []# 打乱后用于在遮蔽语言模型任务中获取15%的随机词元进行预测random.shuffle(candidate_pred_positions)for mlm_pred_position in candidate_pred_positions:if len(pred_positions_and_labels) >= num_mlm_preds:breakmasked_token = None# 80%的时间：将词替换为“<mask>”词元if random.random() < 0.8:masked_token = '<mask>'else:# 10%的时间：保持词不变if random.random() < 0.5:masked_token = tokens[mlm_pred_position]# 10%的时间：用随机词替换该词else:masked_token = random.choice(vocab.idx_to_token)mlm_input_tokens[mlm_pred_position] = masked_tokenpred_positions_and_labels.append((mlm_pred_position, tokens[mlm_pred_position]))return mlm_input_tokens, pred_positions_and_labelsdef _get_mlm_data_from_tokens(tokens, vocab):candidate_pred_positions = []# tokens是一个字符串列表for i, token in enumerate(tokens):# 在遮蔽语言模型任务中不会预测特殊词元if token in ['<cls>', '<sep>']:continuecandidate_pred_positions.append(i)# 遮蔽语言模型任务中预测15%的随机词元num_mlm_preds = max(1, round(len(tokens) * 0.15))mlm_input_tokens, pred_positions_and_labels = _replace_mlm_tokens(tokens, candidate_pred_positions, num_mlm_preds, vocab)pred_positions_and_labels = sorted(pred_positions_and_labels,key=lambda x: x[0])pred_positions = [v[0] for v in pred_positions_and_labels]mlm_pred_labels = [v[1] for v in pred_positions_and_labels]return vocab[mlm_input_tokens], pred_positions, vocab[mlm_pred_labels]

Append the special “< mask >” tokens to the inputs

def _pad_bert_inputs(examples, max_len, vocab):max_num_mlm_preds = round(max_len * 0.15)all_token_ids, all_segments, valid_lens,  = [], [], []all_pred_positions, all_mlm_weights, all_mlm_labels = [], [], []nsp_labels = []for (token_ids, pred_positions, mlm_pred_label_ids, segments,is_next) in examples:all_token_ids.append(torch.tensor(token_ids + [vocab['<pad>']] * (max_len - len(token_ids)), dtype=torch.long))all_segments.append(torch.tensor(segments + [0] * (max_len - len(segments)), dtype=torch.long))# valid_lens不包括'<pad>'的计数valid_lens.append(torch.tensor(len(token_ids), dtype=torch.float32))all_pred_positions.append(torch.tensor(pred_positions + [0] * (max_num_mlm_preds - len(pred_positions)), dtype=torch.long))# 填充词元的预测将通过乘以0权重在损失中过滤掉all_mlm_weights.append(torch.tensor([1.0] * len(mlm_pred_label_ids) + [0.0] * (max_num_mlm_preds - len(pred_positions)),dtype=torch.float32))all_mlm_labels.append(torch.tensor(mlm_pred_label_ids + [0] * (max_num_mlm_preds - len(mlm_pred_label_ids)), dtype=torch.long))nsp_labels.append(torch.tensor(is_next, dtype=torch.long))return (all_token_ids, all_segments, valid_lens, all_pred_positions,all_mlm_weights, all_mlm_labels, nsp_labels)

The WikiText-2 dataset for pretraining BERT

#@save
class _WikiTextDataset(torch.utils.data.Dataset):def __init__(self, paragraphs, max_len):# 输入paragraphs[i]是代表段落的句子字符串列表；# 而输出paragraphs[i]是代表段落的句子列表，其中每个句子都是词元列表paragraphs = [d2l.tokenize(paragraph, token='word') for paragraph in paragraphs]sentences = [sentence for paragraph in paragraphsfor sentence in paragraph]self.vocab = d2l.Vocab(sentences, min_freq=5, reserved_tokens=['<pad>', '<mask>', '<cls>', '<sep>'])# 获取下一句子预测任务的数据examples = []for paragraph in paragraphs:examples.extend(_get_nsp_data_from_paragraph(paragraph, paragraphs, self.vocab, max_len))# 获取遮蔽语言模型任务的数据examples = [(_get_mlm_data_from_tokens(tokens, self.vocab)+ (segments, is_next))for tokens, segments, is_next in examples]# 填充输入(self.all_token_ids, self.all_segments, self.valid_lens,self.all_pred_positions, self.all_mlm_weights,self.all_mlm_labels, self.nsp_labels) = _pad_bert_inputs(examples, max_len, self.vocab)def __getitem__(self, idx):return (self.all_token_ids[idx], self.all_segments[idx],self.valid_lens[idx], self.all_pred_positions[idx],self.all_mlm_weights[idx], self.all_mlm_labels[idx],self.nsp_labels[idx])def __len__(self):return len(self.all_token_ids)

Download and WikiText-2 dataset and generate pretraining examples

def load_data_wiki(batch_size, max_len):"""加载WikiText-2数据集"""num_workers = d2l.get_dataloader_workers()data_dir = d2l.download_extract('wikitext-2', 'wikitext-2')paragraphs = _read_wiki(data_dir)train_set = _WikiTextDataset(paragraphs, max_len)train_iter = torch.utils.data.DataLoader(train_set, batch_size,shuffle=True, num_workers=num_workers)return train_iter, train_set.vocabbatch_size, max_len = 512, 64
train_iter, vocab = load_data_wiki(batch_size, max_len)for (tokens_X, segments_X, valid_lens_x, pred_positions_X, mlm_weights_X,mlm_Y, nsp_y) in train_iter:print(tokens_X.shape, segments_X.shape, valid_lens_x.shape,pred_positions_X.shape, mlm_weights_X.shape, mlm_Y.shape,nsp_y.shape)breakoutput:
Downloading ../data/wikitext-2-v1.zip from https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip...
torch.Size([512, 64]) torch.Size([512, 64]) torch.Size([512]) torch.Size([512, 10]) torch.Size([512, 10]) torch.Size([512, 10]) torch.Size([512])

在这里插入图片描述

BERT 预训练代码

A small BERT , using 2 layers , 128 hidden units , and 2 self - attention heads

net = d2l.BERTModel(len(vocab), num_hiddens=128, norm_shape=[128],ffn_num_input=128, ffn_num_hiddens=256, num_heads=2,num_layers=2, dropout=0.2, key_size=128, query_size=128,value_size=128, hid_in_features=128, mlm_in_features=128,nsp_in_features=128)
devices = d2l.try_all_gpus()
loss = nn.CrossEntropyLoss()

Computes the loss for both the masked language modeling and next sentence prediction tasks

def _get_batch_loss_bert(net, loss, vocab_size, tokens_X,segments_X, valid_lens_x,pred_positions_X, mlm_weights_X,mlm_Y, nsp_y):# 前向传播_, mlm_Y_hat, nsp_Y_hat = net(tokens_X, segments_X,valid_lens_x.reshape(-1),pred_positions_X)# 计算遮蔽语言模型损失mlm_l = loss(mlm_Y_hat.reshape(-1, vocab_size), mlm_Y.reshape(-1)) *\mlm_weights_X.reshape(-1, 1)mlm_l = mlm_l.sum() / (mlm_weights_X.sum() + 1e-8)# 计算下一句子预测任务的损失nsp_l = loss(nsp_Y_hat, nsp_y)l = mlm_l + nsp_lreturn mlm_l, nsp_l, l

Pretrain BERT (net) on the WikiText-2 (train_iter) dataset

def train_bert(train_iter, net, loss, vocab_size, devices, num_steps):net = nn.DataParallel(net, device_ids=devices).to(devices[0])trainer = torch.optim.Adam(net.parameters(), lr=0.01)step, timer = 0, d2l.Timer()animator = d2l.Animator(xlabel='step', ylabel='loss',xlim=[1, num_steps], legend=['mlm', 'nsp'])# 遮蔽语言模型损失的和，下一句预测任务损失的和，句子对的数量，计数metric = d2l.Accumulator(4)num_steps_reached = Falsewhile step < num_steps and not num_steps_reached:for tokens_X, segments_X, valid_lens_x, pred_positions_X,\mlm_weights_X, mlm_Y, nsp_y in train_iter:tokens_X = tokens_X.to(devices[0])segments_X = segments_X.to(devices[0])valid_lens_x = valid_lens_x.to(devices[0])pred_positions_X = pred_positions_X.to(devices[0])mlm_weights_X = mlm_weights_X.to(devices[0])mlm_Y, nsp_y = mlm_Y.to(devices[0]), nsp_y.to(devices[0])trainer.zero_grad()timer.start()mlm_l, nsp_l, l = _get_batch_loss_bert(net, loss, vocab_size, tokens_X, segments_X, valid_lens_x,pred_positions_X, mlm_weights_X, mlm_Y, nsp_y)l.backward()trainer.step()metric.add(mlm_l, nsp_l, tokens_X.shape[0], 1)timer.stop()animator.add(step + 1,(metric[0] / metric[3], metric[1] / metric[3]))step += 1if step == num_steps:num_steps_reached = Truebreakprint(f'MLM loss {metric[0] / metric[3]:.3f}, 'f'NSP loss {metric[1] / metric[3]:.3f}')print(f'{metric[2] / timer.sum():.1f} sentence pairs/sec on 'f'{str(devices)}')

train
在这里插入图片描述
Representing Text with BERT

def get_bert_encoding(net, tokens_a, tokens_b=None):tokens, segments = d2l.get_tokens_and_segments(tokens_a, tokens_b)token_ids = torch.tensor(vocab[tokens], device=devices[0]).unsqueeze(0)segments = torch.tensor(segments, device=devices[0]).unsqueeze(0)valid_len = torch.tensor(len(tokens), device=devices[0]).unsqueeze(0)encoded_X, _, _ = net(token_ids, segments, valid_len)return encoded_X