Skip to content
凌霄剑域

Local LLM Gateway

Local LLM Gateway

Section titled “Local LLM Gateway”

LingXiao can serve as an OpenAI/Anthropic dual-format LLM proxy, providing virtual key management, per-key rate limiting, and request tracing. Default port: 62000.

Architecture

Section titled “Architecture”

Dual-Format Proxy

Section titled “Dual-Format Proxy”

The Gateway supports both OpenAI and Anthropic API formats simultaneously:

OpenAI Format

Section titled “OpenAI Format”
POST /llm/openai/v1/chat/completions
Authorization: Bearer <virtual-key>

Compatible with OpenAI SDK and all OpenAI-format compatible services (DeepSeek, Qwen, Moonshot, etc.).

Anthropic Format

Section titled “Anthropic Format”
POST /llm/anthropic/v1/messages
x-api-key: <virtual-key>

Compatible with Anthropic SDK and Claude model family.

Virtual Keys

Section titled “Virtual Keys”

The Gateway uses a virtual key system, isolated from the upstream Provider’s real API keys:

  • Root key: Gateway master key for managing virtual keys
  • Virtual keys: Generated on demand, bound to rate limit policies and allowed models
  • Key management: Via WebUI Settings panel or API

Virtual Key Properties

Section titled “Virtual Key Properties”
{
  "key": "sk-lx-xxxxxxxx",
  "name": "dev-team-key",
  "rpm": 60,
  "tpm": 200000,
  "daily_token_budget": 1000000,
  "allowed_models": ["gpt-4o", "claude-sonnet-4"],
  "trace_enabled": true
}

Rate Limiting

Section titled “Rate Limiting”
PolicyDescription
rpmRequests per minute limit
tpmTokens per minute limit
daily_token_budgetDaily token budget

When rate limited, returns HTTP 429 + error body:

{
  "error": {
    "type": "rate_limit_exceeded",
    "message": "RPM limit exceeded: 60/min",
    "retry_after": 30
  }
}

Request Tracing

Section titled “Request Tracing”

All request traces are recorded to the SQLite llm_gateway_requests table:

  • Request timestamp and virtual key
  • Upstream Provider and model
  • Token usage (prompt/completion/total)
  • Latency and status code
  • Request and response bodies (configurable)

Configuration

Section titled “Configuration”

Configure LLM Gateway in ~/.lingxiao/settings.json:

{
  "llm_gateway": {
    "enabled": true,
    "port": 62000,
    "provider": "openai",
    "upstream": {
      "openai_api_key": "sk-xxx",
      "openai_base_url": "https://api.openai.com/v1",
      "anthropic_api_key": "sk-ant-xxx"
    },
    "default_rpm": 60,
    "default_tpm": 200000,
    "default_daily_budget": 1000000,
    "trace_enabled": true,
    "log_body": false
  }
}

Authentication Matrix

Section titled “Authentication Matrix”
Endpoint groupAuth method
/llm/openai/v1/**Gateway virtual key (Authorization: Bearer or x-api-key)
/llm/anthropic/v1/**Gateway virtual key (x-api-key or Authorization: Bearer)

LLM Gateway endpoints do not use Server Token; they are governed by the llm_gateway configuration group independently.