EcoServe

Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration (OSDI '26)

EcoServe is an LLM serving system built for commodity GPU clusters — clusters without high-performance interconnects such as NVLink or InfiniBand. It follows a partially disaggregated (PaDG) strategy: within each instance, prefill and decode are disaggregated along the time dimension to mitigate prefill–decode interference; across instances, a cooperating group is cyclically ("rolling") activated — forming a macro instance — so that prefills are always in flight and TTFT stays low. An adaptive scheduler routes requests within a macro instance while a mitosis scaling mechanism adjusts capacity online, together improving goodput.

This repository provides EcoServe's macro instance implementation — the core cross-instance orchestration layer (rolling activation, adaptive scheduling, and the API-server / macro-instance / instance roles) that realizes the PaDG strategy on top of vLLM.

Built on vLLM 0.7.3 — EcoServe code lives in ecoserve/, and most of the vLLM source is kept as-is apart from a few minor changes to the scheduler. See the paper for details.

Architecture

EcoServe wires the API server, the macro-instance, and the vLLM instances together over ZeroMQ — only requests, control signals, and generated tokens cross the network, never KV cache (the "data-reduced" part of the design).

              request                route + control
  clients ──►  API server  ─────────►  Macro instance  ─────────►  Instances
          ◄──  (/generate)             (scheduler)     ◄─ state ──  (vLLM engines)
              tokens  ▲                                                  │
                      └──────────────  generated tokens  ────────────────┘

• API server (head node) — the client-facing /generate endpoint; forwards each request to the macro instance and streams back the tokens it receives from instances.
• Macro instance (head node) — the cross-instance scheduler. It tracks every instance's state and decides when and where requests run, driving rolling activation by sending control signals that toggle which instance currently takes prefills.
• *Instances