EcoServe is an open-source LLM serving system that uses a partially disaggregated architecture with cross-instance orchestration to efficiently run LLM inference on commodity GPU clusters lacking high-performance interconnects.
23
No data
1
0
Apache-2.0
2026-07-05
It addresses the practical challenge of serving LLMs on affordable commodity hardware by avoiding KV cache transfer and using a novel rolling activation mechanism to maintain low TTFT, with a published paper at OSDI '26.
Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration (OSDI '26)
EcoServe is an LLM serving system built for commodity GPU clusters — clusters without high-performance interconnects such as NVLink or InfiniBand. It follows a partially disaggregated (PaDG) strategy: within each instance, prefill and decode are disaggregated along the time dimension to mitigate prefill–decode interference; across instances, a cooperating group is cyclically ("rolling") activated — forming a macro instance — so that prefills are always in flight and TTFT stays low. An adaptive scheduler routes requests within a macro instance while a mitosis scaling mechanism adjusts capacity online, together improving goodput.
This repository provides EcoServe's macro instance implementation — the core cross-instance orchestration layer (rolling activation, adaptive scheduling, and the API-server / macro-instance / instance roles) that realizes the PaDG strategy on top of vLLM.
Built on vLLM 0.7.3 — EcoServe code lives in
ecoserve/, and most of the vLLM source is kept as-is apart from a few minor changes to the scheduler. See the paper for details.
EcoServe wires the API server, the macro-instance, and the vLLM instances together over ZeroMQ — only requests, control signals, and generated tokens cross the network, never KV cache (the "data-reduced" part of the design).
request route + control
clients ──► API server ─────────► Macro instance ─────────► Instances
◄── (/generate) (scheduler) ◄─ state ── (vLLM engines)
tokens ▲ │
└────────────── generated tokens ────────────────┘
/generate endpoint; forwards each
request to the macro instance and streams back the tokens it receives from instances.[OSDI' 26] Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration
Similar projects matched by category, topics, and programming language.
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.