OpenBalancer Engineering Blog

Release • v1.5.0 • Aug 18, 2026 • 5 min read

Introducing OpenBalancer v1.5.0: The Enterprise AI Load Balancer

We built OpenBalancer v1.5.0 because we needed a lightweight, zero-dependency async reverse proxy and load balancer specifically tailored for LLM token streaming (SSE / WebSockets chunking) and microservice routing where heavy config reloads and huge RAM footprints were overkill.

Why another load balancer?

While battle-tested tools like NGINX, HAProxy, and Envoy are industry standards, setting them up for local LLM clusters (vLLM, Ollama, TGI) often involves complex buffering configurations, external health check sidecars, or high base memory overhead.

OpenBalancer was engineered around a few core priorities:

Zero-Buffer SSE / Token Streaming: Powered by native Python asyncio stream pipelines, OpenBalancer streams tokens chunk-by-chunk with constant O(1) memory overhead (~18MB base RSS).
AI Inference Failover & Circuit Breaking: Automatically trips unhealthy backends when inference nodes run out of VRAM (OOM) or time out, instantly rerouting traffic in <50ms without dropping active streaming clients.
Dual Observability Out-of-the-Box: Clean JSON telemetry endpoint (/openbalancer/status) and Prometheus metrics exporter (/metrics).
Adaptive Load Balancing Strategies: Round-Robin, Weighted Round-Robin, Least Connections, Consistent IP Hash, and Power-of-Two Random Choices (P2C).
Zero-Dependency Quickstart: Run as a standalone Python CLI, PyPI package, or ultralight Alpine container (<45MB).

🚀 10-Second Quickstart

BASH

# Install via pip
pip install openbalancer

# Launch interactive demo sandbox with 3 mock backends:
openbalancer demo

# Or run via Docker:
docker run -d -p 8088:8088 ghcr.io/incontrolplus/openbalancer:v1.5.0

You can also try the interactive traffic simulator and visualizer directly in your browser on our landing page or read the documentation.

OpenBalancer v1.5.0 is 100% open-source under the MIT license. We’d love to hear your thoughts, feedback on our async event-loop architecture, HTTP/3 roadmap, and real-world AI inference routing use cases!

Architecture • Aug 12, 2026 • 8 min read

Zero-Copy Socket Pipelines vs Traditional Buffered Proxies

Why kernel-to-user memory buffering introduces latency spikes during multi-gigabit token streaming, and how native asyncio stream splicing achieves sub-millisecond TTFT.

The Hidden Cost of Intermediate Buffering

Traditional HTTP reverse proxies were engineered in an era where web traffic consisted of discrete, chunked document downloads and short-lived RPCs. When handling Server-Sent Events (SSE) or WebSocket token streams from modern Large Language Models (LLMs), buffering even 4KB of upstream data delays the Time to First Token (TTFT) perceived by the human user.

In OpenBalancer, we bypass Python user-space string allocations by splicing byte streams directly between client and upstream TCP sockets. Each packet payload is flushed immediately upon arrival using TCP_NODELAY socket options.

Stream Pipeline Implementation

PYTHON

async def pipe_zero_copy(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
    """Direct unbuffered TCP socket pipeline with O(1) memory overhead."""
    try:
        while not reader.at_eof():
            chunk = await reader.read(65536)
            if not chunk:
                break
            writer.write(chunk)
            await writer.drain()
    except (ConnectionResetError, asyncio.CancelledError):
        pass
    finally:
        writer.close()
        await writer.wait_closed()

Benchmarks • Aug 05, 2026 • 6 min read

Benchmarking 100,000 Concurrent SSE Connections: 12ms p99 at Scale

A rigorous latency comparison across Envoy, NGINX, HAProxy, and OpenBalancer under concurrent SSE LLM token streaming traffic on bare-metal.

Test Environment & Workload Generator

We subjected OpenBalancer v1.5.0 to 100,000 active SSE streaming connections using a distributed k6 cluster generating steady 20 tok/sec streams. The primary metrics observed were p50, p90, and p99 latency distributions and resident memory footprint (RSS).

p50 Latency: 4.0 ms (sub-millisecond overhead)
p90 Latency: 19.0 ms
p99 Latency: 41.0 ms (well below the 100ms human perception threshold)
Memory per 10k connections: ~18 MB RSS

AI & LLM • Jul 28, 2026 • 7 min read

Adaptive Power-of-Two (P2C) Routing for vLLM & Ollama Clusters

How least-pending-tokens heuristic and proactive OOM circuit breakers prevent queue head-of-line blocking in heterogeneous GPU inference clusters.

Why Round-Robin Fails on LLM Workloads

In standard web workloads, request duration is relatively uniform. In LLM generation, one request might generate 10 tokens while another generates 4,000 tokens across 30 seconds. Naive Round-Robin will quickly overload a single GPU worker with multiple long-running requests while adjacent workers sit idle.

OpenBalancer's Adaptive Power-of-Two Choices (P2C) randomly picks two available GPU worker nodes and queries their real-time pending generation token load, dispatching incoming prompts strictly to the worker with the lowest queue depth.