HyperPod Inference Gateway: GPU-Aware Routing
Updated
AWS announced a Kubernetes-native, GPU-aware routing add-on for SageMaker HyperPod on Amazon EKS.
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing system for scalable LLM inference. In a post dated September 24, 2026, AWS says it deploys as a single EKS managed add-on on existing SageMaker HyperPod infrastructure, with zero application changes.
AWS says it replaces round-robin load balancing with routing driven by live inference signals, and that this reduces first-token latency by up to 82%. Three components do that work: an Envoy endpoint that exposes one private endpoint per cluster, a body-based router that reads the model name, and an endpoint picker that scores each model-server pod. It works with any OpenAI-compatible model server, including vLLM and SGLang, and needs no application code changes. Per-cluster routing is available in every AWS Region where the SageMaker HyperPod inference add-on is supported.
Key facts
- Posted September 24, 2026: AWS describes the gateway as a Kubernetes-native, GPU-aware routing system that deploys as a single EKS managed add-on on existing SageMaker HyperPod infrastructure, with no application changes.
- AWS says it replaces round-robin balancing and reduces first-token latency by up to 82%, with p99 time-to-first-token reductions of 97–98% in mixed-hardware and burst scenarios.
- AWS names three components: an Envoy endpoint, a body-based router that reads the model name, and an endpoint picker that scores pods on six signals, including KV cache use and queue depth.
- AWS says it works with any OpenAI-compatible model server, including vLLM and SGLang. Per-cluster routing is in every Region that supports the HyperPod inference add-on.
Source: Amazon SageMaker HyperPod Inference Gateway for scalable LLM inference · Amazon Web Services