⚙️ Engineering
Scanned
vLLM Model Serving
Serve LLMs at high throughput with vLLM. Configures PagedAttention, continuous batching, tensor parallelism, and OpenAI-compatible endpoints, plus GPTQ/AWQ/FP8 quantization for serving on limited GPU memory.
This skill deploys production LLM inference with vLLM. It sets up OpenAI-compatible API endpoints, tunes PagedAttention and continuous batching for throughput, configures tensor and pipeline parallelism across GPUs, applies GPTQ/AWQ/FP8 quantization, and benchmarks latency and tokens-per-second under load.
vllm inference serving gpu quantization
When to use
Use when deploying a self-hosted LLM API, optimizing inference latency or throughput, or fitting a large model onto limited GPU memory with quantization and parallelism.
Examples
Stand up an LLM API
OpenAI-compatible endpoint
Serve Mistral 7B with vLLM behind an OpenAI-compatible endpoint, with continuous batching and AWQ quantization on a single 24GB GPU
Boost throughput
Tune batching and parallelism
My vLLM server saturates at 30 req/s — help me tune max-num-seqs, KV cache, and tensor parallelism to raise throughput