I have two servers: ExternalSrv and InternalSrv, on the same EC2 VPC.
I have a very simple setup using Nodejs, Express and Axios.
ExternalSrv handles requests from the public, which, of course, come in to ExternalSrv's public IP address. ExternalSrv calls InternalSrv to do some of the work.
In order to simplify the security group inbound rules on InternalSrv, I would like to allow ALL VPC IP address, but nothing else.
I find that ExternalSrv always uses its Public IP address when making requests to InternalSrv's Private IP address. Therefore, the security group needs to be updated with ExternalSrv's Public IP address whenever that address changes (Stop/Start, new instance, more instances, etc.). That seems like a fragility point in ongoing maintenance.
This seems like this should be easy, but I've been searching for an answer for quite some time.
Any insight would be appreciated.
Bill
When two Amazon EC2 instances in the same VPC communicate with each other, it is best to perform this communication via private IP addresses. This has several benefits:
Security Groups can refer to other Security Groups
Traffic stays within the VPC (if communicating via Public IP addresses, the traffic will exit the VPC and then come back in)
It is cheaper (there is a 1c/GB charge for traffic going out of the VPC and then back in)
The best-practice security setup for your situation would be:
Create a Security Group on ExternalSrv (SG-External) that would allow inbound traffic as necessary (eg port 80, 443), together with default "Allow All" outbound traffic
Create a Security Group on InternalSrv (SG-Internal) that allows inbound traffic from SG-External
That is, SG-Internal specifically references SG-External in its rules. This way, inbound traffic will be accepted from ExternalSrv without needing to know its IP address. It also allows other servers to be added to the Security Group in future and they will also be permitted access.
Yes, you could simply add a rule that limits inbound access to the CIDR of the VPC, but good security is always about having multiple layers of security. Restricting access will cut-down potential attack vectors.
Related
How can I create a VPC/subnet on AWS and launch a Windows instance and Linux instance in that same subnet. whenever I try to create a vpc it will not give ssh access to other terminals even though I give permissions in route tables.
This could be many things. I'd probably recommend looking at security groups (adding TCP/22 inbound from your IP address) and depending on your setup (make sure an IGW is attached), making sure a public IP address is assigned to your instances (add an Elastic IP if your instance doesn't already have one). Without more information on your environment, I can only provide some background to help you troubleshoot.
The network path to an instance in a VPC looks like this:
Internet -> AWS Boundary -> VPC Boundary -> VPC Router -> Subnet boundary -> Elastic Network Interface
AWS Boundary
Most of the time this part isn't important. From the Internet, eventually your traffic will be routed into an Amazon autonomous system. This is generally transparent. But if you have a public-type Direct Connect, the Amazon side of the connection dumps you here. In this area, you can access public API endpoints, but you cannot access anything within a VPC without progressing further down the stack. See below.
VPC Boundary
This is the point at which traffic enters/leaves the VPC. There are a number of ways this can occur.
Internet Gateway (IGW)
This performs a one-to-one NAT between an amazon-owned publicly routable IP address and a VPC internal private IP address (do not confuse this with a NAT Gateway, as that's different and described below). Only instances assigned a public IP address can use an IGW. Being a 1-to-1 NAT, to the Instances perspective it's a connection between it's private IP address and an Internet-routable IP. From the outside it's a connection to the instance's public IP address. To stress, instances without a public IP address assigned (elastic IP or otherwise) will not be able to communicate with the Internet in either direction through an IGW. Conversely, if you don't have an IGW, there is no use for public IP addresses on your instances. Also worth mentioning, there are also egress-only gateways which allow IPv6 connectivity for outbound initiated connections only.
Virtual Private Gateway (VGW)
This can be seen as a router of sorts. If you have a VPN connecting to AWS' VPN service or if you're using Private Direct Connect, it will traverse this virtual device, which will be associated with a VPC. What's noteworthy, is that is a peering of sorts and uses VPC private IP addresses for communication. You don't need a public IP address to talk through a VGW, and you cannot traverse a VGW by using the public IP address of instances in your VPC.
VPC Peering Connections (PCX)
This is very similar to a VGW. It can only connect two VPCs (with transit gateway, I suppose this is an oversimplification), and it connects them on the Private IP address layer. You cannot reference resources across a PCX by using their Public IP addresses.
VPC Endpoint (VPC-E)
This is only accessible from inside of the VPC with connections going out (obviously return traffic with come back through this, but . It is a connection to a specific AWS Endpoint within the AWS Public Boundary (like the S3 API endpoint). This doesn't use an instance's public IP address.
VPC Router
All traffic exiting or entering the VPC hits this router, and it routes to/from all of the egress/ingress points at the VPC boundary and to/from each of the subnets. You can adjust this router to control which traffic goes where, and you can have different route tables for each subnet. Thus a "public" subnet is one in a VPC that has a IGW and has a default route (0.0.0.0/0) to that IGW. A private subnet has no route to an IGW. If there isn't a route to an IGW, having a public IP address on your instance is useless and wasteful.
You can also route to an ENI if you want to control traffic within your VPC and send it to an EC2 Instance (web proxying / IDS / traffic capturing / etc.), however, that ENI has to reside in a subnet with a different route table, or otherwise it's own outbound traffic will be routed back to itself. All traffic exiting/entering any subnet and all traffic exiting/entering the VPC traverses this router and is subject to the routes you configure. You cannot configure routing within your subnets, any packet destined for an address within your VPC's private IP space will be automatically routed to that particular subnet, and you cannot override this functionality with more specific routes.
Subnet Boundary
At the subnet boundary, traffic is subject to the Network Access Control Lists (NACLs). This is a stateless, rule-based firewall. By default it is wide-open and requires no configuration to allow traffic. There is no rule to allow "existing connections" so if you start to lock down a subnet with NACLs, you'll probably need to open up all of the ephemeral ports in the direction you're expecting return traffic. Any traffic between instances within the same subnet will not hit the NACL. But anything that leaves or enters the subnet (whether it's going to another subnet in the same VPC, or leaving the VPC altogether) will hit the NACL and be subject to its rules. Generally leave these alone, unless you need to protect traffic at the subnet level, NACLs are a little unwieldy.
Elastic Network Interface (ENI)
Finally traffic goes through an ENI, where it is subject to a security group. This is a stateful implicit-deny firewall for which you can only add allow rules. If the security group doesn't have a rule allowing outbound traffic from the instance, that traffic will never leave the ENI. If the security group doesn't have a rule allowing a kind of inbound traffic, that kind of inbound traffic will never be sent to the Instance (i.e. the OS will not be able to detect it).
NAT Gateway
This is a device that can reside in a subnet. It will need a route to an IGW and it will need a public IP address (Elastic IPs work). It performs a many-to-one NAT of any private IP addresses within your VPC to a publicly-routable IP (technically it performs a many-to-one NAT translating the many private IP addresses within your VPC to its one own private IP address, and when it communicates with the IGW, that does a one-to-one NAT to translate the NAT Gateway's private IP address into a public IP address). This only works for IPv4. And it only works if instances send their traffic to the NAT Gateway's ENI. Generally, you'd have a NAT Gateway reside in a public Subnet (with a default route to the IGW), and all of your instances in private subnets would have a default route to the NAT Gateway's ENI.
Summary
Bare minimum to connect to an EC2 Instance:
VPC has an IGW attached
NACL has allow rules for the desired traffic in both directions (configured by default)
EC2 Instance has a security group allowing the desired traffic (TCP/22 for SSH, etc.)
EC2 Instance has a public IP address associated with it (must be configured when you launch the instance, or can be added afterward by attaching an Elastic IP).
This configuration allows you to directly connect to the instance over the public internet.
Well-Architected VPC Pattern
The generic architectural pattern advised by AWS for simple best-practice networks is:
VPC with an IGW attached
two-or-more public subnets (each in separate availability zones) with a default route to the IGW.
a NAT Gateway in each of the public subnets
two-or-more private subnets (each in the same AZ as the public subnets) each with default routes to the NAT Gateway in the public subnet in the same AZ.
a bastion host in an auto-scaling group of 1 spanning all of the public subnets allowing SSH inbound (Whether this is advisable is debatable)
if needed, a VPN connection from your corporate network to the VPC
security groups on the private instances allowing inbound from specific resources within the VPC (referencing by security group ID where possible) and whatever inbound traffic is needed through the VPC, and outbound traffic TCP/443 outbound to the world (or more/less depending on your risk tolerance and needs).
if needed, and VPC Endpoint to S3 or whatever API endpoints you expect to send large volumes of traffic.
This architecture allows your private instances to connect to public internet resources outbound (using IPv4, at least), and inbound traffic has to go through the VPN or the bastion host. For public-facing services, setting up an Elastic Load Balancer in the public subnets is the desired way to provide public connectivity for your instances, so that you can continue to keep them protected in a private subnet.
Even when I create EC2 instances in a private subnet, they must be able to send traffic to the Internet if I want to register them to a ECS cluster.
I am using a NAT gateway to do this, but I still feel insecure that the instances can send private information to anywhere in case of takeover.
What would be the most compact CIDR range that I can use for the instances' security group, instead of 0.0.0.0/0?
For the moment, you may have to rely on the list of public IP address ranges for AWS, allowing traffic bound for all the CIDR blocks associated with your region.
Part of design for resiliency of much of what AWS does relies on the ability of their service endpoints not to depend on static address assigments and instead to use DNS... but their service endpoints should always be on addresses associated with your region, since very few services violate their practice strict regional separation of service infrastructure.
(CloudFront, Route 53, and IAM do, maybe others, but these are provisioning endpoints, not operational ones. These provisioning-only endpoints do not need to be accessible for most applications to function normally.)
A super common pattern here is to create a proxy in a narrow CIDR and only allow outbound traffic to flow through the proxy. White-list AWS endpoints and log all traffic flowing through the proxy for monitoring/auditing.
AWS Endpoints = https://docs.aws.amazon.com/general/latest/gr/rande.html
I have 2 AWS EC2 instances living inside 2 different subnets of my vpc.
I would like to allow the ruby app running on the first instance (say App#1) to call the endpoints of the app (say App#2) running on the 2nd instance.
I would also like my users to directly call the endpoints of App#2 from their browser.
Here is what I have tried (and mostly failed):
[Sucess!] I added the known IP addresses of my users to the inbound rules of Load Balancer Security Group of App#2 and have confirmed that they can access App#2 endpoints from their browsers.
[Fail!] I added the Load Balancer Security Group ID of App#1 to the inbound rules to the Load Balancer Security Group of App#2. But my logs tell me App#1 cannot access the endpoints of App#2.
[Fail!] I added the VPC Security Group ID of App#1 to the inbound rules of the Load Balancer Security Group of App#2 - nope, still doesn't work.
(Somehow, when I launched the instance for App#1, aws automatically created 2 security groups for this instance - one for VPC and one for load balancer... I have no idea why/how this happened...)
[Fail!] I added the CIDR for the subnet App#1 was in to the inbound rules of the Load Balancer Security Group of App#2. Still no joy.
[Success...Sort Of] I assigned an elastic IP for the instance running App#1 and added that to the inbound rules of the Load Balancer Security Group of App#2. This works but I would rather not use this method since I would like to elastically scale my App#1 in the future and I do not know how to automatically assign more elastic IPs for the new instances when they spin up, add them to the inbound rules, and then somehow remove them when they shut down.
I feel like there has got to be a really clean solution to this problem and I am probably missing something painfully obvious. Can someone please give me a hint?
Any help would be appreciated!
It sounds like you might be using the public IP address of your load balancer, so it looks like the traffic is coming from the outside. Try using the private IP/DNS if there is one, or setting up a second, internally-facing load balancer.
So App#2 is in public subnet, App#1 is in private subnet. For example, the diagram will be something like:
Internet => LB#2 => App#2:80 (in public subnet) => LB#1 => App#1:4567 (in private subnet)
Let's open all inbound rules in all instances and loadbalancers, check if you can access it via internet,
then apply security groups on each layer each time, don't change all of them at same time.
Let me know which layer has issue.
I have an EC2 node, node1 (security group SG1) which should be accessible from another EC2 node, node2 (security group SG2) on port 9200. Now, when I add an inbound rule in SG1 with port 9200 and specify SG2 as source in Custom IP section, I can't access node1 from node2. On the other hand, if I specify an inbound rule in SG1 with source as 0.0.0.0/0 or IP of node2, it works fine. What is wrong in my approach?
Are you attempting to connect to node1's public or private address? From the documentation:
When you specify a security group as the source or destination for a
rule, the rule affects all instances associated with the security
group. For example, incoming traffic is allowed based on the private
IP addresses of the instances that are associated with the source
security group.
I've been burned on this before by trying to connect to an EC2 instance's public address... sounds very similar to your setup, actually. When you wire up the inbound rule so that the source is a security group, you must communicate through the source instance's private address.
Some things to be aware of:
In EC2 Classic, private IP addresses can change on stop/start of an EC2 instance. If you're using EC2 classic you may want to look into this discussion on Elastic DNS Names for a more static addressing solution.
If you set up your environment in VPC, private IP addresses are static. You can also change security group membership of running instances.
Reason: Inter security-group communication works over private addressing. If you use the public IP address the firewall rule will not recognise the source security group.
Solution: You should address your instances using the Public DNS record - this will actually be pointed at the private IP address when one of your instances queries the DNS name.
e.g. if your instance has public IP 203.0.113.185 and private IP 10.1.234.12, you are given a public DNS name like ec2-203-0-113-185.eu-west-1.compute.amazonaws.com.
ec2-203-0-113-185.eu-west-1.compute.amazonaws.com will resolve to 203.0.113.185 if queried externally, or 10.1.234.12 if queried internally. This will enable your security groups to work as intended.
This will enable you to use an elastic IP as you simply use the Public DNS entry of the elastic IP. Also, having the DNS resolve to the internal IP means that you are not incurring bandwidth charges for your data between instances:
Instances that access other instances through their public NAT IP
address are charged for regional or Internet data transfer, depending
on whether the instances are in the same region.
The Public DNS didn't work for me.
What I did instead was create a custom inbound rule using the security group of the other instance.
I would like to use my internal exchange servers for email notifications from an azure Web Role. The role is set up on the west us region. I need to allow this role, with multiple instances, access my corporate smtp server on port 25. IS there an appropriate range of IPs that I could open up on my corporate firewall, or is there a more secure option.
We have experience setting up networks in Azure as well as VPN gateways, I am not sure that this would be part of the solution as my understanding is not broad enough to see how I could implement something that would allow secure access to corporate networks while continuing to be publicly available.
Any help or direction would be appreciated.
Thanks
Your web role has a single (and fairly static) ip address of the load balancer, but this is of no use to you as the source ip address for outgoing data will be the ip address of the particular instance. You can get hold of this address, but not easily, and it is subject to change — every time an instance recycles, or is added, the ip address will change (although the incoming ip address, of the load balancer, remains static). So you can't provide instance public ip address ranges for the corporate firewall.
If you are familiar with VPNs and private networks on Azure, then that will be your simplest option. Your worker role ip addresses will still change, but the range is more predictable, and is defined by your subnet configuration. A worker role on a private network (VPN) will have both public and private ip addresses, so will be publicly accessible, but will send to a VPN ip address via the gateway.
Another option is to do some sort of smtp relay, either on a VM in Azure, or a managed hosting service, where the ip address is more static. This would require that a machine is setup, maintained, and running, but could probably be easily done on an extra small linux vm.
Your web roles will be part of an Azure Cloud Service. That cloud service is assigned a static public IP address by Azure. No matter how many roles you add/remove from the service the public IP will remain unchanged. You can set your SMTP server to only allow requests from that IP address.