tools
LoRAX
LoRAX is an open-source server for serving thousands of fine-tuned LoRA models on shared GPU infrastructure.
LoRAX dynamically loads LoRA adapters and serves them with a shared base model. It supports continuous batching, adapter scheduling, quantization, token streaming, and OpenAI-compatible chat requests.
Teams use it to deploy many fine-tuned language models while sharing GPU memory. It runs through Docker, Kubernetes, SkyPilot, or locally on Linux with a supported NVIDIA GPU. No hosted LoRAX pricing was found; users provide their own infrastructure.
Features
- Dynamically loads LoRA adapters from Hugging Face, Predibase, or local files
- Batches requests using different adapters in the same inference batch
- Schedules adapter prefetching and offloading between GPU and CPU memory
- Supports tensor parallelism, quantization, token streaming, and CUDA kernels
- Provides Docker images, Helm charts, Prometheus metrics, and OpenTelemetry tracing
- Offers an OpenAI-compatible API for multi-turn chat conversations
- Supports structured JSON output and per-request tenant isolation
- Apache 2.0 license with free commercial use
Use cases
- Serve many fine-tuned language models on shared GPU infrastructure
- Route requests to different LoRA adapters without restarting the server
- Combine multiple adapters per request to create model ensembles
- Deploy inference servers with Docker or Kubernetes
- Expose fine-tuned models through REST, Python, or OpenAI-compatible APIs