Skip to content
AI.info

tools

LoRAX

LoRAX is an open-source server for serving thousands of fine-tuned LoRA models on shared GPU infrastructure.

LoRAX

LoRAX dynamically loads LoRA adapters and serves them with a shared base model. It supports continuous batching, adapter scheduling, quantization, token streaming, and OpenAI-compatible chat requests.

Teams use it to deploy many fine-tuned language models while sharing GPU memory. It runs through Docker, Kubernetes, SkyPilot, or locally on Linux with a supported NVIDIA GPU. No hosted LoRAX pricing was found; users provide their own infrastructure.

Features

  • Dynamically loads LoRA adapters from Hugging Face, Predibase, or local files
  • Batches requests using different adapters in the same inference batch
  • Schedules adapter prefetching and offloading between GPU and CPU memory
  • Supports tensor parallelism, quantization, token streaming, and CUDA kernels
  • Provides Docker images, Helm charts, Prometheus metrics, and OpenTelemetry tracing
  • Offers an OpenAI-compatible API for multi-turn chat conversations
  • Supports structured JSON output and per-request tenant isolation
  • Apache 2.0 license with free commercial use

Use cases

  • Serve many fine-tuned language models on shared GPU infrastructure
  • Route requests to different LoRA adapters without restarting the server
  • Combine multiple adapters per request to create model ensembles
  • Deploy inference servers with Docker or Kubernetes
  • Expose fine-tuned models through REST, Python, or OpenAI-compatible APIs

Pros

    Cons

      Capabilities

      • Choice of models — “Just specify any adapter as the model parameter.” source
      • Self-hosted — “We recommend starting with our pre-built Docker image to avoid compiling custom CUDA kernels and other dependencies.” source
      • API — “LoRAX supports multi-turn chat conversations combined with dynamic adapter loading through an OpenAI compatible API.” source
      • Official SDKs — “For a full tutorial including token streaming and the Python client, see Getting Started - Docker .” source
      • Runs models for you — “LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.” source
      • Traces and evaluates — “prebuilt Docker images, Helm charts for Kubernetes, Prometheus metrics, and distributed tracing with Open Telemetry.” source

      Pricing

      Prices checked
      2026-10-25
      Official website