8卡910B4-32G测试Qwen2.5-VL-72B-instruct模型兼容性
测试结论：
| 序号 | 模型名称 | 模型大小 | 显存占用 | 8卡910B4-32G（256G） | 备注 |
| 是否支持？ | |||||
| 1 | Qwen2.5-VL-32B-Instruct | 64G | 188G | 支持 | You are using a model of type qwen2_5_vl to instantiate a model of type . This is not supported for all configurations of models and can yield errors. |
| quay.io/ascend/vllm-ascend:v0.7.3 | Daemon start success! | ||||
| （https://vllm-ascend.readthedocs.io/en/stable/tutorials/single_npu_multimodal.html） | 32B的图像识别效果不满足要求,务必要继续测试，肯定支持 | ||||
| swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-800I-A2-py311-openeuler24.03-lts （https://modelers.cn/models/Models_Ecosystem/Qwen2.5-VL-32B-Instruct） | |||||
| 2 | Qwen2.5-VL-72B-Instruct | 137G | vllm框架下占用：208G | MindIE不支持(单卡分配提示显存不足) | MindIE: |
| swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-800I-A2-py311-openeuler24.03-lts | 4或8卡910B4-64G理论上可支持。 | ||||
| RuntimeError: NPU out of memory. Tried to allocate 602.00 MiB (NPU 5; 29.50 GiB total capacity; 28.10 GiB already allocated; 28.10 GiB current active; 441.27 MiB free; 28.44 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. | |||||
| VLLM支持 | vllm： | ||||
| quay.io/ascend/vllm-ascend:v0.7.3 | 输出不了json格式： | ||||
| （乱回答：quay.io/ascend/vllm-ascend:v0.8.5rc1） | WARNING 05-28 01:37:36 __init__.py:48] xgrammar is only supported on x86 CPUs. Falling back to use outlines instead. | ||||
| WARNING 05-28 01:37:36 __init__.py:84] xgrammar module cannot be imported successfully. Falling back to use outlines instead. | |||||
| WARNING 05-28 01:37:36 __init__.py:91] outlines does not support json_object. Falling back to use xgrammar instead. | |||||
| INFO: 219.141.177.34:2272 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error | |||||
| ERROR: Exception in ASGI application | |||||
| 3 | Qwen2.5-VL-72B-Instruct-AWQ | 41G | / | 都不支持 | MindIE:raise AssertionError(f"weight {tensor_name} does not exist") |
| >>> Exception:weight model.layers.0.self_attn.q_proj.weight does not exist | |||||
| vllm:ERROR 05-28 05:49:02 engine.py:400] KeyError: 'visual.blocks.0.attn.qkv.weight' |
测试命令：
| 推理框架镜像 | 启动命令 | 测试命令 |
| quay.io/ascend/vllm-ascend:v0.7.3 | 启动容器： docker run \ --name vllm-ascend \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /root/.cache:/root/.cache \ -p 8000:8000 \ -itd quay.io/ascend/vllm-ascend:v0.7.3 进入容器： docker exec -it vllm-ascend bash 启动vllm服务： vllm serve Qwen/Qwen2.5-VL-32B-Instruct --tokenizer_mode="auto" --tokenizer_mode="auto" --dtype=bfloat16 --max_num_seqs=256 --tensor_parallel_size=8 --gpu-memory-utilization=0.98 --max-model-len=32768 & 停vllm服务： ps -ef|grep python3|grep -v grep|awk '{print $2}'|xargs kill -9 停容器： docker stop vllm-ascend docker rm vllm-ascend | curl http://110.165.26.90:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-VL-72B-Instruct", "messages": [ {"role": "system", "content": "你是专业图片识别助手"}, {"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://i-blog.csdnimg.cn/img_convert/4f479548fee019db69efeb29cacc1a94.jpeg"}}, {"type": "text", "text": "请描述图片内容？"} ]} ] }' |
| swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-800I-A2-py311-openeuler24.03-lts | 启动容器： docker run --name m1 --privileged=true -it -d --net=host --shm-size=200g --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 --device=/dev/davinci_manager --device=/dev/hisi_hdc --device=/dev/devmm_svm --entrypoint=bash -w /models -v /usr/local/Ascend/driver:/usr/local/Ascend/driver -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi -v /usr/local/sbin:/usr/local/sbin -v /root/.cache/modelscope/hub/models:/models -v /tmp:/tmp -v /etc/hccn.conf:/etc/hccn.conf -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-800I-A2-py311-openeuler24.03-lts 进入容器： docker exec -it m1 bash 启动服务： cd /usr/local/Ascend/mindie/latest/mindie-service nohup ./bin/mindieservice_daemon > ./q.log 2>&1 & tail -f ./q.log 停服务： ps -ef|grep mindieservice_daemon|grep -v grep|awk '{print $2}'|xargs kill -9 停容器： docker stop m1 docker rm m1 | curl 110.165.26.90:28858/v1/chat/completions -d ' { "model": "qwen2_vl", "messages": [{ "role": "user", "content": [ {"type": "image_url", "image_url": "https://i-blog.csdnimg.cn/img_convert/4f479548fee019db69efeb29cacc1a94.jpeg"}, {"type": "text", "text": "请描述图片内容"} ] }], "max_tokens": 512, "do_sample": true, "repetition_penalty": 1.00, "temperature": 0.01, "top_p": 0.001, "top_k": 1 }' |
系统环境：
参考链接：
https://vllm-ascend.readthedocs.io/en/latest/quick_start.html
https://modelers.cn/models/Models_Ecosystem/Qwen2.5-VL-32B-Instruct