Kishin Technologies iconKishinTechnologies

Case Studies

Production reference architectures

There is more than one way to build an AI gateway or platform, depending on your cloud, providers, and data requirements. Explore the reference architectures below to see the tradeoffs.

AWS

Multi-Region AI Gateway on AWS with LiteLLM

A regulated enterprise needed centralized, auditable access to multiple LLM providers across multiple AWS regions for resilience and data residency, without individual teams calling provider APIs directly.

TeamsApplicationsWAF + Application Load BalancerLiteLLM Gateway — ECS FargateAuth · Quotas · Model Routing · Request/Response LoggingAurora (Multi-AZ)Config & audit trailElastiCache RedisResponse cacheBedrockVertex AIOpenAISecrets Manager · KMS · CloudWatchActive-active across two AWS regions, cross-region replication

Key Capabilities

  • ✓Centralized authentication and per-team quotas at the gateway
  • ✓Multi-provider routing across Bedrock, Vertex AI, and OpenAI behind one API
  • ✓Active-active deployment across multiple AWS regions with automatic failover
  • ✓Full audit trail of prompts and responses for compliance reporting
  • ✓Infrastructure as code with versioned, replicated state

Stack

AWSLiteLLMECS FargateAuroraElastiCacheRoute 53Secrets ManagerTerraform
Azure

Azure API Management as an AI Gateway

A multi-team organization needed governed, quota-controlled access to AI providers through their existing Azure identity and API management investment, with per-team usage policies and native streaming support.

Internal TeamsEntra IDAzure API ManagementAuth · Quotas · Token Limits · Model RoutingEvent Hub(Observability)Azure AIFoundryVertex AIOutbound Response PoliciesNon-stream: parse, enrich, log · Stream: SSE passthroughResponse to teams (OpenAI-compatible)

Key Capabilities

  • ✓Governed multi-team access using existing Entra ID identity
  • ✓Per-team quotas, token limits, and renewal periods via APIM products
  • ✓Unified OpenAI-compatible response format across Azure AI Foundry and Vertex AI
  • ✓Native streaming (SSE) alongside standard non-streamed responses
  • ✓Usage data published to Event Hub for observability and compliance reporting

Stack

AzureAPI ManagementEntra IDEvent HubAzure AI FoundryVertex AI
Kubernetes

Private Inference Platform on Kubernetes with vLLM

An organization with strict data residency and latency requirements needed inference and retrieval to run entirely within their own infrastructure, with no prompts or documents leaving the cluster.

RAG ApplicationIngress GatewayAuth · RoutingQdrantVector search (RAG)RedisCachevLLM Inference ServiceOpenAI-compatible servingNVIDIA GPU Node Pool — KubernetesAutoscaling via HPA on GPU utilizationObservabilityPrometheus · DCGM GPU metrics · Model registry

Key Capabilities

  • ✓Self-hosted model serving with vLLM on an NVIDIA GPU node pool, no data leaving the cluster
  • ✓Retrieval-augmented generation backed by Qdrant for private knowledge bases
  • ✓Redis caching to reduce redundant inference and retrieval latency
  • ✓GPU-aware autoscaling driven by real-time utilization metrics
  • ✓Observability into GPU usage, latency, and throughput

Stack

KubernetesvLLMNVIDIA GPUQdrantRedisPrometheus

Need a similar gateway for your organization?

Kishin Technologies can design and build a production AI gateway tailored to your cloud, providers, and governance requirements.

Book a Readiness CallView Services