Kishin Technologies iconKishinTechnologies

Case Studies

Production reference architectures

There is more than one way to build an AI gateway or platform, depending on your cloud, providers, and data requirements. Explore the reference architectures below to see the tradeoffs.

AWS

Multi-Region AI Gateway on AWS with LiteLLM

A regulated enterprise needed centralized, auditable access to multiple LLM providers across multiple AWS regions for resilience and data residency, without individual teams calling provider APIs directly.

TeamsApplicationsWAF + Application Load BalancerLiteLLM Gateway — ECS FargateAuth · Quotas · Model Routing · Request/Response LoggingAurora (Multi-AZ)Config & audit trailElastiCache RedisResponse cacheBedrockVertex AIOpenAISecrets Manager · KMS · CloudWatchActive-active across two AWS regions, cross-region replication

Key Capabilities

  • ✓Centralized authentication and per-team quotas at the gateway
  • ✓Multi-provider routing across Bedrock, Vertex AI, and OpenAI behind one API
  • ✓Active-active deployment across multiple AWS regions with automatic failover
  • ✓Full audit trail of prompts and responses for compliance reporting
  • ✓Infrastructure as code with versioned, replicated state

Stack

AWSLiteLLMECS FargateAuroraElastiCacheRoute 53Secrets ManagerTerraform

This AI gateway runs on AWS, using LiteLLM for provider routing, ECS Fargate for compute, Aurora and ElastiCache for state and caching, Route 53 for regional failover, Secrets Manager for credential handling, and Terraform for infrastructure as code.

Azure

Azure API Management as an AI Gateway

A multi-team organization needed governed, quota-controlled access to AI providers through their existing Azure identity and API management investment, with per-team usage policies and native streaming support.

Internal TeamsEntra IDAzure API ManagementAuth · Quotas · Token Limits · Model RoutingEvent Hub(Observability)Azure AIFoundryVertex AIOutbound Response PoliciesNon-stream: parse, enrich, log · Stream: SSE passthroughResponse to teams (OpenAI-compatible)

Key Capabilities

  • ✓Governed multi-team access using existing Entra ID identity
  • ✓Per-team quotas, token limits, and renewal periods via APIM products
  • ✓Unified OpenAI-compatible response format across Azure AI Foundry and Vertex AI
  • ✓Native streaming (SSE) alongside standard non-streamed responses
  • ✓Usage data published to Event Hub for observability and compliance reporting

Stack

AzureAPI ManagementEntra IDEvent HubAzure AI FoundryVertex AI

This AI gateway runs on Azure API Management, using Entra ID for identity, Event Hub for usage telemetry, and unifies access to Azure AI Foundry and Vertex AI behind one OpenAI-compatible interface.

Kubernetes

Private Inference Platform on Kubernetes with vLLM

An organization with strict data residency and latency requirements needed inference and retrieval to run entirely within their own infrastructure, with no prompts or documents leaving the cluster.

RAG ApplicationIngress GatewayAuth · RoutingQdrantVector search (RAG)RedisCachevLLM Inference ServiceOpenAI-compatible servingNVIDIA GPU Node Pool — KubernetesAutoscaling via HPA on GPU utilizationObservabilityPrometheus · DCGM GPU metrics · Model registry

Key Capabilities

  • ✓Self-hosted model serving with vLLM on an NVIDIA GPU node pool, no data leaving the cluster
  • ✓Retrieval-augmented generation backed by Qdrant for private knowledge bases
  • ✓Redis caching to reduce redundant inference and retrieval latency
  • ✓GPU-aware autoscaling driven by real-time utilization metrics
  • ✓Observability into GPU usage, latency, and throughput

Stack

KubernetesvLLMNVIDIA GPUQdrantRedisPrometheus

This private inference platform runs on Kubernetes, serving models with vLLM on NVIDIA GPU nodes, using Qdrant for retrieval, Redis for caching, and Prometheus for observability, with no data leaving the cluster.

Need a similar gateway for your organization?

Kishin Technologies can design and build a production AI gateway tailored to your cloud, providers, and governance requirements.

Discuss a Gateway BuildView Services