tools
LoRAX
LoRAX is an open-source server for serving thousands of fine-tuned LoRA models on shared GPU infrastructure.
In inglese
LoRAX dynamically loads LoRA adapters and serves them with a shared base model. It supports continuous batching, adapter scheduling, quantization, token streaming, and OpenAI-compatible chat requests.
Teams use it to deploy many fine-tuned language models while sharing GPU memory. It runs through Docker, Kubernetes, SkyPilot, or locally on Linux with a supported NVIDIA GPU. No hosted LoRAX pricing was found; users provide their own infrastructure.
Features
- Dynamically loads LoRA adapters from Hugging Face, Predibase, or local files
- Batches requests using different adapters in the same inference batch
- Schedules adapter prefetching and offloading between GPU and CPU memory
- Supports tensor parallelism, quantization, token streaming, and CUDA kernels
- Provides Docker images, Helm charts, Prometheus metrics, and OpenTelemetry tracing
- Offers an OpenAI-compatible API for multi-turn chat conversations
- Supports structured JSON output and per-request tenant isolation
- Apache 2.0 license with free commercial use
Use cases
- Serve many fine-tuned language models on shared GPU infrastructure
- Route requests to different LoRA adapters without restarting the server
- Combine multiple adapters per request to create model ensembles
- Deploy inference servers with Docker or Kubernetes
- Expose fine-tuned models through REST, Python, or OpenAI-compatible APIs
Pros
Cons
Capabilities
- Choice of models — “LoRAX supports a number of Large Language Models as the base model including Llama (including CodeLlama ), Mistral (including Zephyr ), and Qwen .” source
- Self-hosted — “LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.” source
- API — “LoRAX supports multi-turn chat conversations combined with dynamic adapter loading through an OpenAI compatible API.” source
- Official SDKs — “For a full tutorial including token streaming and the Python client, see Getting Started - Docker .” source
- Runs models for you — “Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs” source
- Traces and evaluates — “prebuilt Docker images, Helm charts for Kubernetes, Prometheus metrics, and distributed tracing with Open Telemetry.” source
Pricing
- Prices checked
- 2026-09-25