跳到主要内容

vLLM 速查表

快速查找 vLLM 常用命令、参数和代码片段。

安装命令​

# pip 安装
pip install vllm

# 开发版本
pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

# Docker
docker pull vllm/vllm-openai:latest

启动服务​

# 基本启动
vllm serve MODEL_NAME

# 完整参数
vllm serve MODEL_NAME \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 4096 \
    --quantization awq

Python API​

基础推理​

from vllm import LLM, SamplingParams

llm = LLM(model="MODEL_NAME")
sampling_params = SamplingParams(temperature=0.7, max_tokens=100)

outputs = llm.generate(["prompt"], sampling_params)
print(outputs[0].outputs[0].text)

采样参数​

SamplingParams(
    n=1,                    # 生成结果数
    temperature=0.7,        # 温度
    top_p=0.95,             # 核采样
    top_k=-1,               # Top-k 采样
    max_tokens=100,         # 最大 token 数
    presence_penalty=0.0,   # 存在惩罚
    frequency_penalty=0.0,  # 频率惩罚
    stop=None,              # 停止词
)

LLM 初始化参数​

LLM(
    model="MODEL_NAME",
    tensor_parallel_size=1,      # 张量并行
    pipeline_parallel_size=1,    # 流水线并行
    gpu_memory_utilization=0.9,  # GPU 显存利用率
    max_model_len=4096,          # 最大模型长度
    quantization=None,           # 量化类型
    dtype="auto",                # 数据类型
    max_num_seqs=256,            # 最大并发序列数
    enable_prefix_caching=False, # 前缀缓存
)

HTTP API​

Completions​

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MODEL_NAME",
        "prompt": "PROMPT",
        "max_tokens": 100,
        "temperature": 0.7
    }'

Chat Completions​

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MODEL_NAME",
        "messages": [{"role": "user", "content": "MESSAGE"}],
        "max_tokens": 100
    }'

Python 客户端​

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

# Completions
response = client.completions.create(
    model="MODEL_NAME",
    prompt="PROMPT",
    max_tokens=100
)

# Chat
response = client.chat.completions.create(
    model="MODEL_NAME",
    messages=[{"role": "user", "content": "MESSAGE"}]
)

量化选项​

类型命令说明
FP16--dtype float16默认
BF16--dtype bfloat16推荐
FP8--quantization fp8Hopper GPU
INT8--quantization awq通用
GPTQ--quantization gptq4-bit
AWQ--quantization awq4-bit

并行配置​

# 张量并行 (TP)
--tensor-parallel-size 4

# 流水线并行 (PP)
--pipeline-parallel-size 2

# 混合并行
--tensor-parallel-size 2 --pipeline-parallel-size 2

性能优化参数​

# 启用前缀缓存
--enable-prefix-caching

# 启用 chunked prefill
--enable-chunked-prefill

# 推测解码
--speculative-model MODEL_NAME --num-speculative-tokens 5

# 调整批处理
--max-num-seqs 512 --max-num-batched-tokens 16384

常见问题​

显存不足​

llm = LLM(
    model="MODEL_NAME",
    gpu_memory_utilization=0.7,
    max_model_len=2048,
    quantization="awq"
)

模型下载​

# 设置镜像
export HF_ENDPOINT=https://hf-mirror.com

# 手动下载
huggingface-cli download MODEL_NAME --local-dir ./models

多 GPU 启动​

# 2 卡
CUDA_VISIBLE_DEVICES=0,1 vllm serve MODEL_NAME --tensor-parallel-size 2

# 4 卡
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve MODEL_NAME --tensor-parallel-size 4

参考链接​