News/Product Launch
AWS launches SageMaker HyperPod Inference Gateway as a GPU-aware EKS add-on
The Kubernetes-native router uses live GPU signals to pick the best pod for each inference request and can cut first-token latency by up to 82%.
What happened
Amazon Web Services has launched Amazon SageMaker HyperPod Inference Gateway, a Kubernetes-native add-on for Amazon EKS that routes inference traffic with GPU awareness. The add-on reads real-time GPU signals and sends each request to the pod best suited to serve it.
AWS says the gateway can cut first-token latency by up to 82%. Model servers and client applications stay as they are. The product sits on Amazon SageMaker HyperPod and was announced on the AWS Machine Learning Blog.
Why it matters
GPU-aware routing that can cut first-token latency by up to 82% without model-server or client changes can make large-scale LLM inference on EKS cheaper and more responsive, tightening HyperPod’s place as a production inference stack.