【llama-factory】Lora微调和DPO训练

微调参考
DPO参考

llama-factory官网

LoRA微调

数据集处理

数据集格式和注册
Alpaca数据集格式：

[{"instruction": "人类指令（必填）","input": "人类输入（选填）","output": "模型回答（必填）","system": "系统提示词（选填）","history": [["第一轮指令（选填）", "第一轮回答（选填）"],["第二轮指令（选填）", "第二轮回答（选填）"]]}
]

dataset_info.json中注册数据集：
注意可以自定义数据集路径！

"数据集名称": {"file_name": "data.json","columns": {"prompt": "instruction","query": "input","response": "output","system": "system","history": "history"}
}

SFT训练及其yaml设置

sft
llama3_lora_sft.yaml配置中：template可以参考github官方的template，比如qwen1-2.5的template都是qwen。

yaml中参数介绍

训练的时候设置一下卡:

CUDA_DEVICE_ORDER='PCI_BUS_ID' CUDA_VISIBLE_DEVICES=2 llamafactory-cli train tmp/llama3_lora_sft.yaml

merge模型

qwen2.5_merge_sft.yaml

### model
model_name_or_path: /data1/ztshao/pretrained_models/Qwen/Qwen2.5-7B-Instruct
adapter_name_or_path: /data1/ztshao/projects/projects428/military/results/lora_sft
template: qwen
finetuning_type: lora### export
export_dir: /data1/ztshao/projects/projects428/military/results/merge_sft
export_size: 2
export_device: cpu
export_legacy_format: false

merge完在对应的输出文件夹里自带了ollama导入模型需要的Modelfile文件。

merge操作：

llamafactory-cli export /data1/ztshao/projects/projects428/military/src/finetune/qwen2.5_merge_sft.yaml

DPO

数据集处理

数据集格式和注册
Alpaca数据集格式：

[{"instruction": "人类指令（必填）","input": "人类输入（选填）","chosen": "优质回答（必填）","rejected": "劣质回答（必填）"}
]

dataset_info.json中注册数据集：

"数据集名称": {"file_name": "data.json","ranking": true,"columns": {"prompt": "instruction","query": "input","chosen": "chosen","rejected": "rejected"}
}

DPO训练及其yaml设置

DPO训练
llama-factory的DPO训练大讨论

我的yaml：

### model
model_name_or_path: /data1/ztshao/pretrained_models/Qwen/Qwen2.5-7B-Instruct
trust_remote_code: true### method
stage: dpo
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all
pref_beta: 0.1
pref_loss: sigmoid  # choices: [sigmoid (dpo), orpo, simpo]### dataset
dataset: military_dpo
template: qwen
cutoff_len: 2048
max_samples: null
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4### output
output_dir: /data1/ztshao/projects/projects428/military/results/models
logging_steps: 2
save_steps: 50
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 5.0e-5
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

训练：

CUDA_DEVICE_ORDER='PCI_BUS_ID' CUDA_VISIBLE_DEVICES=2 llamafactory-cli train tmp/llama3_lora_dpo.yaml