ai-gateway

Project 3 of a 3-project AI-engineering portfolio suite. ai-gateway is a production-shaped FastAPI microservice that fronts two existing backends — hybrid-rag-platform (Project 1) and db-action-agent (Project 2) — with its own authentication, per-tenant rate limiting and token budgeting, OpenTelemetry tracing, and one genuinely-streamable endpoint direct to Claude.

Architecture

Three routes, three different backend-wrapping strategies (see docs/adr/0001-*.md for why):

Route	Backend	How it's called
POST /v1/rag/query	hybrid-rag-platform	HTTP proxy (RAG_SERVICE_URL)
POST /v1/agent/query	db-action-agent	In-process library import (db_action_agent.run())
POST /v1/chat/stream	Claude directly	Gateway's own Anthropic client, SSE

caller
  │  X-Api-Key
  ▼
auth/deps.py: resolve_tenant  ──►  gateway Postgres (tenants, api_keys, rag_credentials,
  │  (hash key, check rate limit,      rate_limit_buckets, token_budget_usage)
  │   check token budget)
  ▼
┌─────────────┬──────────────┬────────────────────┐
│ /v1/rag/    │ /v1/agent/   │ /v1/chat/stream     │
│ query       │ query        │                     │
│             │              │                     │
│ rag_client/ │ agent_client/│ passthrough/         │
│ proxy.py    │ wrapper.py   │ claude_stream.py     │
│  (httpx →   │  (in-process │  (direct Anthropic   │
│  hybrid-    │  db_action_  │  streaming call)     │
│  rag-       │  agent.run)  │                      │
│  platform)  │              │                      │
└─────────────┴──────────────┴────────────────────┘
       │              │                │
       ▼              ▼                ▼
  rag.proxy_call   agent.run      generation.claude   (one OTel child span each,
       └──────────────┴────────────────┘               under an auto http.ingress span)
                       │
                       ▼
                 OTLP → Jaeger (localhost:16686)

Every route sits behind the same resolve_tenant FastAPI dependency (src/ai_gateway/auth/deps.py): resolve X-Api-Key → tenant, check the per-tenant token-bucket rate limit, check the daily LLM-token budget, reject with 429 (+ Retry-After or X-RateLimit-Reset headers) before any backend call is dispatched.

Why each backend is wrapped differently

• RAG is proxied over HTTP because hybrid-rag-platform already runs as its own service — there's nothing to gain from importing it in-process, and it keeps this gateway's own image small.
• The agent is imported in-process because db-action-agent has no HTTP service of its own, and its own ADR assigns that trade-off to whichever project wraps it next — this one.
• Only the passthrough route can genuinely stream. Neither hybrid-rag-platform nor db-action-agent streams internally, so wrapping either one can only ever return a completed response. POST /v1/chat/stream talks to Claude directly for real token-by-token SSE.

Full architectural rationale lives in the four ADRs under docs/adr/:

• 0001-in-process-agent-vs-http-rag-proxy.md
• 0002-gateway-owns-its-own-auth-and-quota.md
• 0003-honest-streaming-scope.md
• 0004-honest-tracing-scope.md

See Getting started for installation and usage.