Comparing BGP communities in AWS and Azure

I like to point out to people that it's easier to train a network person on cloud than it is to train a cloud person on networks. It's a glib generalisation but it holds true for the most part because there is so much to networking that comes from history and quite a lot of grounding that a seasoned network engineer or architect will already understand. A big chunk of the AWS and Azure networking certification covers BGP and that's one of the reasons they are considered quite hard for some but quite easy for others. BGP is a topic that many very experienced network engineers in enterprise networking can get through their entire career without touching, but for those who operate at scale or work with MSP and telco networks it's bread and butter.

Both Azure and AWS use BGP for route exchange on ExpressRoute and Direct Connect respectively to on premise networks; they also use it optionally for route exchange on VPN connections from on premise networks. Both AWS and Azure have a public and private type peering - the public type to give access to public services over the cloud backbone and private ER or DX connections

I use ER and DX a lot instead of ExpressRoute and Direct Connect. You probably realised that though.

Both Azure and AWS tag their route prefixes with BGP communities and support specific BGP communities from customers to influence routing decisions.

Did you spot that? 'Influence' routing decisions. It's pretty important to remember that as BGP operates across autonomous system boundaries you can only influence your peer's routing, not control it. Conversely they can influence your routing in specific and defined ways but cannot directly control it; it's one of the important tenets of BGP.

ER and DX public connections

On Azure ExpressRoute you can create a Microsoft Peering to route your traffic to public services over your ExpressRoute rather than over the internet. Microsoft aren't all that keen on this, expecially for Microsoft 365. On an AWS Direct Connect you can create a public vif which performs the same function allowing you to route connectivity to public AWS services over a low (or at least predictable) latency private connection rather than over the internet.

NO_EXPORT

AWS explicitly supports the NO_EXPORT community defined in RFC 1997. This is defined as:

NO_EXPORT (0xFFFFFF01)

All routes received carrying a communities attribute containing this value MUST NOT be advertised outside a BGP confederation boundary (a stand-alone autonomous system that is not part of a confederation should be considered a confederation itself).

This means that prefixes learned over a DX public peering cannot be exported to other connected ASNs - this prevents customers from unwittingly becoming transit networks for other people to reach AWS. AWS also treats all prefixes received from customers as if they were tagged with NO_EXPORT which means that your public DX prefixes will never be exported beyond AWS.

The Azure ExpressRoute documentation doesn't explicitly state that NO_EXPORT is used however there is an expectation that customers will not export routes learned from ExpressRoute to elsewhere. This is also enforced by internet providers that peer with Microsoft who are required to reject any ExpressRoute prefixes on the internet.

All parties peering with Microsoft agree not to accept routes from AS12076 (ExpressRoute) under any circumstances and should filter out AS12076 on all peers.

Regional Communities

Both AWS DX and Azure ER public connections support a range of regional and geographic communities. For Azure the full list is here and you can filter routes based on region for both public clouds and soverign national clouds (US Government and China). In AWS there is a simpler scoping mechanism where routes learned from the same region as your DX peering will be tagged with 7224:8100, those from the same continent will be tagged with 7224:8200 and everything else is untagged. This means you can prefer, for example, US DX for US regions and a European DX for traffic to European regions. In order to ensure that traffic is routed symetrically you can apply corresponding communities to the prefixes you advertise to AWS: adding 7224:9100 will instruct AWS to only allow that prefix to the local region of the DX connection, 7224:9200 will instruct AWS to allow it to regions in the same continental area and leaving them untagged will instruct AWS to allow those prefixes to any region

Service Communities

Azure supports an extensive list of communities for public services while AWS publishes a detailed JSON formatted list of their prefixes along with information about the services and regions they support; this leaves it for the customer to implement service filtering without communities.

Local Preference Communities

A rather neat thing from AWS is the ability to influence Local Preference by setting specific communities. Setting one of the communities on prefixes advertised to AWS will instruct AWS to apply the corresponding Local Preference for traffic outbound from AWS to the customer.

7224:7100 - Low preference

7224:7200 — Medium preference

7224:7300 — High preference

This is particularly useful because Local Preference Always Wins so you can override AS PATH length and home region association to influence path selection and ECMP across regions.

OK, you're right, longest prefix beats LOCAL_PREF. It's probably better to say Local Preference Always Wins In BGP but that's not as catchy.

ER and DX private connections

AWS support is basically the same over private vifs with regard to the scoping and preference. Azure gives you granular control to set your own community values for routes exported from Azure to your own network over a private peering.