OpenBalancer Logo OpenBalancer
Retainers Request Enterprise SLA
Home / API & Architecture Documentation
ℹ️

System Overview & Capabilities #

OpenBalancer is an ultralight, non-blocking asynchronous load balancer and API reverse proxy engineered for AI token streaming, high-frequency microservices, and zero-downtime operations.

💡
AI Agent & LLM Integration Context: For automated coding agents (Claude, ChatGPT, Gemini), we provide structured API specifications and OpenAPI 3.1 endpoints directly at /openbalancer/status and /metrics.

Traditional reverse proxies incur significant buffer allocation and thread contention penalties when handling long-lived Server-Sent Events (SSE) or bi-directional WebSockets. OpenBalancer eliminates intermediate buffers by piping raw socket descriptors directly through Python's non-blocking asyncio event loop.

Core Architectural Capabilities

  • Zero-Buffer SSE Streaming: Unbuffered chunk-by-chunk piping for OpenAI/Claude/vLLM completion endpoints.
  • Proactive Circuit Breaking: Tripping unhealth backends in <50ms upon consecutive 5xx errors or connection timeouts.
  • Multi-Algorithm Routing: Least Latency, Weighted Round-Robin, Least Connections, IP Hash, and Power-of-Two (P2C).
  • Built-in Prometheus Exporter: Native /metrics scraper without external exporter sidecars.
  • Zero-Downtime Hot Reload: Immediate in-flight re-configuration via POSIX SIGHUP signals.
🚀

Quickstart & Installation #

Get OpenBalancer running in under 60 seconds using Docker, Docker Compose, or standalone Python.

1. Run via Docker (Recommended)

BASH / DOCKER 
docker run -d --name openbalancer \
  -p 8080:8080 \
  -v $(pwd)/config.json:/app/config.json:ro \
  ghcr.io/incontrolplus/openbalancer:v1.5.0

2. Run via Standalone Python (Zero Dependencies)

BASH 
# Requires Python 3.11+ (Standard Library only - 0 pip packages required)
python3 openbalancer.py config.json

3. Verify Ingress & Prometheus Telemetry

BASH 
# Test Load Balancer Ingress
curl -i http://localhost:8080/

# Query JSON Cluster Telemetry
curl http://localhost:8080/openbalancer/status

# Scrape Prometheus Metrics
curl http://localhost:8080/metrics
⚡

Kernel & Event Loop Architecture #

OpenBalancer utilizes single-threaded non-blocking cooperative multitasking via Python's asyncio socket event loop (~18MB RSS footprint).

⚡
Zero-Copy Bufferless Streaming Pipeline: When proxying streaming responses (e.g. OpenAI/Claude SSE completion streams or gRPC chunks), OpenBalancer bypasses internal aggregation buffers and streams chunks directly between socket descriptors in real time.

Circuit Breaker & Health Probing State Machine

State Trigger Condition Routing Behavior Recovery Process
HEALTHY Health check returns HTTP 200 OK within timeout limit. Receives full traffic proportional to its configured weight. Steady-state health probes every probe_interval seconds.
PROBING / DEGRADED Single probe failure or latency spike > 500ms. Traffic throttled; candidate for isolation. Probed with accelerated frequency until confirmed or tripped.
TRIPPED / DOWN consecutive_failures >= threshold (default: 3). Immediately isolated. Zero ingress traffic routed to this node. Background probes continue until 2 consecutive 200 OK probes restore health.
🔀

Routing Algorithms #

OpenBalancer provides 6 production-ready load balancing strategies switchable at runtime or via configuration:

Algorithm Identifier Routing Logic Best For Complexity
round_robin Distributes requests evenly in circular sequence across all healthy backends. Homogeneous backend nodes with uniform request cost. O(1)
weighted / weighted_round_robin Routes traffic proportional to each node's capacity weight integer (e.g. 3:2:1). Heterogeneous servers (e.g., 8-GPU node vs 2-GPU node). O(1)
least_latency / latency Directs incoming requests to the backend with the lowest recent health check latency. Multi-region backends or variable inference workloads. O(N)
least_connections / least_conn Selects the healthy node with the lowest number of active open TCP sockets. Long-lived HTTP/1.1 connections, WebSockets, SSE streams. O(N)
ip_hash / consistent_hash Hashes client source IP (MD5 hash ring) to bind client sessions to a sticky backend. Stateful web sessions, cache locality, sticky user state. O(1)
power_of_two / p2c Picks two random healthy nodes and routes to the one with fewer active connections. Massive clusters avoiding the herd effect with O(1) selection overhead. O(1)
⚙️

Configuration Schema (`config.json`) #

OpenBalancer reads its initial topology from config.json. All parameters can be hot-reloaded without downtime via SIGHUP.

Property Type Default Description
host string "0.0.0.0" Network interface to bind the ingress listener to.
port integer 8080 Port for the load balancer ingress proxy.
algorithm string "least_latency" Dispatch algorithm: least_latency, round_robin, weighted, least_connections, ip_hash, power_of_two.
health_check_interval integer 3 Interval in seconds between background HTTP health probes.
circuit_breaker_threshold integer 3 Consecutive failures required before tripping a backend node.
backends array [] Array of upstream nodes with host, port, weight, and health_path.
🛠️

Interactive JSON Configuration Builder #

Configure your cluster parameters visually below. The schema updates in real time with 1-click clipboard copy and direct JSON file download.

Health Probe Interval 3s
Circuit Breaker Failures Threshold 3 failures
config.json (Live Validated)
🎨

Custom Error Page Studio #

Design, live-preview, and export branded, lightweight standalone HTML error pages for 429 Rate Limiting, 502 Bad Gateway, and 503 Outage scenarios.

LIVE INTERACTIVE PREVIEW
🛡️
HTTP 429 TOO MANY REQUESTS

Rate Limit Exceeded (429)

Too many concurrent requests were received. Your token bucket will automatically refill shortly.

Automatic reconnection in 15 seconds
Ray ID: ob_9a8f2c3d • OpenBalancer AI Edge
📊

Telemetry & Prometheus API #

OpenBalancer features built-in, zero-dependency observability endpoints natively exposing cluster telemetry in both structured JSON and standard Prometheus scrape formats.

1. Query JSON Cluster Telemetry

Returns real-time uptime, request counters, active strategy, and per-node health status & latency.

HTTP / JSON 
GET /openbalancer/status HTTP/1.1
Host: localhost:8080

HTTP/1.1 200 OK
Content-Type: application/json

{
  "status": "healthy",
  "version": "1.5.0",
  "uptime_seconds": 84210,
  "algorithm": "least_latency",
  "active_connections": 142,
  "total_requests": 1428904,
  "backends": [
    { "url": "http://127.0.0.1:8001", "healthy": true, "latency_ms": 2.1, "active_conns": 45 },
    { "url": "http://127.0.0.1:8002", "healthy": true, "latency_ms": 1.8, "active_conns": 52 },
    { "url": "http://127.0.0.1:8003", "healthy": true, "latency_ms": 3.4, "active_conns": 45 }
  ]
}

Real-Time Latency Distribution Heatmap & Flamegraph

Observe real-time latency distribution across 4 discrete threshold buckets (<10ms green, 10-50ms cyan, 50-200ms yellow, >200ms red) with live percentile calculations and execution waterfall stages.

DYNAMIC 60FPS
<10ms (Optimal)74%
10-50ms (Normal)18%
50-200ms (Spike)6%
>200ms (Degraded)2%
🧪

Interactive API Tester ("Try it Out") #

Test OpenBalancer management and telemetry endpoints in real time. Choose an endpoint below and click Execute Request.

Target Environment:
200 OK Time: 1.2ms Size: 420 B
📈

Benchmarks & High-Load Audit #

Measured on bare-metal Linux server (AMD EPYC 7763, 64 Cores, 10GbE Network) with wrk -t16 -c1000 -d30s against 3 backend instances.

Concurrency (Connections) Throughput (Reqs/sec) p50 Latency p90 Latency p99 Latency Error Rate
100 conns 18,420 req/s 0.82 ms 1.45 ms 3.10 ms 0.00%
500 conns 34,180 req/s 1.15 ms 2.80 ms 5.40 ms 0.00%
1,000 conns 42,500 req/s 1.40 ms 4.20 ms 8.90 ms 0.00%
10,000 conns (SSE Streams) 10,000 streams 1.85 ms 5.10 ms 12.20 ms 0.00%
🐳

Production Deployment & Docker #

Deploying OpenBalancer in mission-critical environments with automatic restart policies, healthchecks, and resource constraints.

Complete Production Docker Compose Stack

YAML 
services:
  openbalancer:
    image: ghcr.io/incontrolplus/openbalancer:v1.5.0
    container_name: openbalancer-core
    restart: always
    ports:
      - "8080:8080"
    volumes:
      - ./config.json:/app/config.json:ro
    environment:
      - OPENBALANCER_LOG_LEVEL=INFO
      - OPENBALANCER_WORKERS=1
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen(\"http://localhost:8080/openbalancer/health\", timeout=2)"]
      interval: 10s
      timeout: 3s
      retries: 3
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 256M
🔄

Zero-Downtime Hot Reload (`SIGHUP`) #

OpenBalancer listens for POSIX SIGHUP signals. When received, the core re-reads config.json from disk, reconstructs upstream pools, updates routing weights, and resets connection counters without closing active socket streams.

BASH 
# Send SIGHUP to running Docker container
docker kill -s HUP openbalancer-core

# Or via PID on standalone host:
kill -HUP $(pgrep -f openbalancer.py)
🛡️

Enterprise Support & Legal Verification #

Commercial SLA retainers, 24/7 incident response, bespoke load balancing algorithms, and on-premise high-availability deployment services are provided by INCONTROL PLUS ЕООД.

🛡️

Contractual 99.9% Uptime Guarantee

Backed by formal contractual service credits for any monthly downtime below 99.9%. Net-14 corporate invoicing and instant Stripe Card checkout supported.

Request Enterprise SLA Contract Company Impressum & Verification