2 posts tagged with "ai"

If you're running self-hosted AI models on AKS, routing inference traffic is a different problem to routing web traffic. A standard HTTP load balancer doesn't know about GPU queue depth or KV cache utilisation — it just sees HTTP requests. The new inference gateway for Application Gateway for Containers changes that.

Microsoft put this feature into public preview on 24 June 2026. It brings the Kubernetes Gateway API Inference Extension into Application Gateway for Containers, giving you model-aware routing decisions at the ingress layer without building your own sidecar logic.

The short version: your gateway can now pick which model server pod handles a request based on real-time signals from the model itself, not just round-robin or connection count.

Microsoft has made managed virtual network support for evaluations in Microsoft Foundry generally available. You can now keep evaluation workloads inside a Microsoft-managed private network boundary without having to build and run your own virtual network just to get started.

If you work in a regulated environment, or you simply want tighter control of outbound traffic, this is a useful step. It gives you a cleaner path to private connectivity for evaluation runs, while still letting you use Foundry's hosted evaluation features.

The nice part is that Microsoft handles most of the plumbing. You still choose the isolation mode and approve access to the services you need, but you don't have to manage the underlying network estate yourself.