Tech Notes:- Nginx reverse proxy issues in AWS due to name resolution failure

The Problem

There are certain failure scenarios in AWS which is hard to detect. One of them is Nginx reverse proxy failure due to name resolution issues.

AWS has certain services like ELB, RDS with dynamic changes in underlying host IP. The problem here is that some applications resolve the IP only once mostly during the startup, restart or reload of that service. One such application is the Nginx web server. 

Nginx resolves all the names to their IP and caches it during the start, restart or reload. If the DNS resource record changes in between, Nginx fails. This typically happens when Amazon ELB IP address changes. Amazon updates the DNS record, but Nginx never re-resolves the DNS record and stays pointing at the old IP address. Subsequently requests to the backend start failing once Amazon drops services from the old IP address.

Analysis

But how do you find out if the Nginx issues are caused due to name resolution? One of the best options in front of us is Amazon CloudTrail. Normally ELB IP changes when the ELB scales or there is a scale up or down of hosts behind the ELB.

Refer the below Amazon link to read about the ELB behavior

https://aws.amazon.com/articles/best-practices-in-evaluating-elastic-load-balancing/

AWS CloudTrail logs all the API activities associated with your AWS account and this includes the changes related to ELB as well.

Solution

Now how do we resolve this problem? If we are using the community edition of Nginx, the best option is to configure a resolver entry.

But how do you know the IP address of the DNS that you should use in resolver?AWS documentation comes handy here

https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html

The first four IP addresses and the last IP address in each subnet CIDR block are not available for you to use, and cannot be assigned to an instance. For example, in a subnet with CIDR block 10.0.0.0/24, the following five IP addresses are reserved:

• 10.0.0.0: Network address.
• 10.0.0.1: Reserved by AWS for the VPC router.
• 10.0.0.2: Reserved by AWS. The IP address of the DNS server is always the base of the VPC network range plus two; however, we also reserve the base of each subnet range plus two. For VPCs with multiple CIDR blocks, the IP address of the DNS server is located in the primary CIDR. For more information, see Amazon DNS Server.
• 10.0.0.3: Reserved by AWS for future use.
• 10.0.0.255: Network broadcast address. We do not support broadcast in a VPC, therefore we reserve this address.

Apart from that there is VPC independent up address which can be used as resolver and that is 169.254.169.253. Here the catch is you lose the ability to resolve up outside AWS from nginx point of view.