Is NetworkPolicy a stateful firewall?
For example, if I allow ingress from a certain IPs on certain ports, is the return traffic automatically allowed on ephemeral? Ditto for allowed egress.
How does this play with a default block policy in place?
Are there any other considerations here?
NetworkPolicy is stateful and will allow an established connection to communicate both ways.
Related
Explaining my confusion / lack of understanding
When reading about the external LoadBalancer in K8s, which is a cloud provider only feature, I don't quite understand when it should be used, as when one creates a Deployment K8s will do Round Robin load balancing on the pods in that Deployment.
So from my current understanding all one would need to do is make a NodeIP, and you have the equivalent of an external load balancer?
Or should I think of the LoadBalancer type as haproxy/nginx/Envoy, where one can do SSL, reverse proxy, and many other useful things?
My current guess is that the proper use of LoadBalancer is to add many NodeIP's, but I can't find anything to back that up.
Question
Can anyone explain when and why to use LoadBalancer and not just using the NodeIP?
For example, You want to deploy multiple applications in your cluster, say 10 apps.
You would like to access these 10 apps over internet. One way is to set those 10 application services as nodeport so you can access them from outside. For this to happen kubernetes opens 10 nodeports on each cluster node. This is a security risk.
In most of the enterprises where they work behind firewall in a closed network dont allow external traffic to/from any ports other than http/https ( 80/443 ).
One way is to set service type as Loadbalancer for each application service. So, to access 10 app, you will be provisioning 10 load balancers to access the app servers over http/https ports. Since loadbalancers are charged resources, economically it is not viable to have one load balancer for each service that you want to access over itnernet.
Is there a way to access all those 10 app services running inside kubernetes over single port. This is where ingress controller comes into picture.
Ingress controller allows single ip-port to access all services running in k8s through ingress rules. The ingress controller service is set to load balancer so it is accessible from public internet
I am trying to send an http request from my pod to the outside, but it seems impossible.
I currently have implemented a loadbalancer with fixed IP, but so far I just tested connections to the service.
Is there any specific constraint for it? Is it possible to overcome the issue?
Your worker nodes where your pods live are probably in private subnets (it's good practice to keep them there) and if that is the case then it's not Kubernetes problem. You should setup NAT to allow outbound traffic. I'm not familiar with Azure, but you should also check other abstractions that control your traffic (like Security Group or NACLs in AWS)
I have a few questions regarding Kubernetes: How to secure Kubernetes cluster?
My plan is to develop an application that secures Kubernetes cluster by default. I have written and tested a few Network Policy successfully.
As a second step I want to set these information dynamically in my application based on cloud provider and so one.
1.) I want to block access the host network as well as the meta data services (my cluster runs on AWS):
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.250.0.0/16 # host network
- 169.254.169.254/32 # metadata service
Does anyone know how I can access the host network dynamically?
I found an issue that says that you must use the Meta Data Service: https://github.com/kubernetes/kubernetes/issues/24657
Does anyone know how I can find out on which cloud provider I am currently running?
Based on that information, I want to set the meta data service IP.
2.) I want to block access to the "kube-system" namespace:
egress:
- to:
- podSelector:
matchExpressions:
- key: namespace
operator: NotIn
values:
- kube-system
Does anyone know how I can enforce the actual denied access?
As far as I understood, the key labled "namespace" is just a name that I choosed. How does Kubernetes know that I actually mean the namespace and nothing else?
3.) I want to block Internet access:
spec:
podSelector: {}
policyTypes:
- Egress
Does anyone know, if something like the DNS server in the DMZ zone is still reachable?
4.) I want to block communication to pods with a different namespace:
egress:
- to:
- namespaceSelector:
matchLabels:
project: default
Here, I developed a controller that set the namespace dynamically.
Your ideas are good in terms of a least-privilege policy but the implementation is problematic due to the following reasons.
The logic you are trying to achieve it beyond the capabilities of Kubernetes network policies. It is very difficult to combine multiple block and allow policies in k8s without them conflicting with each other. For example, your first snippet allows access to any IP outside of the cluster and then your 3rd question is about blocking access to the internet - these two policies can't work simultaneously.
You shouldn't block access to the kube-system namespace because that's where the k8s DNS service is deployed and blocking access to it will prevent all communications in the cluster.
To answer your 1st question specifically:
How I can access the host network dynamically?
The cluster subnet is defined when you deploy it on AWS - you should store it during creation and inject it to your policies. Alternatively, you may be able to get it by calling an AWS API.
You can also get the cluster node IPs from Kubernetes: kubectl get nodes -o wide
How I can find out on which cloud provider I am currently running?
Kubernetes doesn't know which platform it is running on, but you can guess it based on the node name prefix, for example: aks-nodepool1-18370774-0 or gke-...
Your 4th point about blocking access between namespaces is good but it would be better to do it with an ingress policy like this:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: default
spec:
podSelector: {}
policyTypes:
- Ingress
For more details, I recommend this blog post that explains the complexities of k8s network policies: https://medium.com/#reuvenharrison/an-introduction-to-kubernetes-network-policies-for-security-people-ba92dd4c809d
As Mark pointed out, NP may not be able to address all your use cases. You might want to check out the Open Policy Agent project and there specifically the Gatekeeper tool, which could be utilized, to at least in part cover some of your needs.
My understanding is that setting the Service type to LoadBalancer creates a new Azure Load Balancer and assigns an IP address to the Service. Does this mean that I can have multiple Services using port 80? If the app behind my Service (an ASP.NET Core app) can handle TLS and HTTPS why shouldn't I just use LoadBalancer's for any Service I want to expose to the internet?
What is the advantage of using an Ingress if I don't care about TLS termination (You can let Cloudflare handle TLS termination)? If anything, it slows things down by adding an extra hop for every request.
Update
Some answers below mention that creating load balancers is costly. It should be noted that load balancers on Azure are free but they do charge for IP addresses of which they give you five for free. So for small projects where you want to expose up to five IP addresses, it's essentially free. Any more than that, then you may want to look ad usign Ingress.
Some answers also mention extra complexity if you don't use Ingress. I have already mentioned that Cloudflare can handle TLS termination for me. I've also discovered the external-dns Kubernetes project to create DNS entries in Cloudflare pointing at the load balancers IP address? It seems to me that cutting out Ingress reduces complexity as it's one less thing that I have to configure and manage. The choice of Ingress is also massive, it's likely that I'll pick the wrong one which will end up unmaintained after some time.
There is a nice article here which describe the differences on Service(Load Balancer) and Ingress.
In summary, you can have multiple Service(Load Balancer) in the cluster, where each application is exposed independently from each other. The main issue is that each Load Balancer added will increase the cost of your solution, and does not have to be this way, unless you strictly need this.
If multiple applications listen on port 80, and they are inside the container, there is no reason you also need to map it to the port 80 in the host node. You can assign it to any port, because the Service will handle the dynamic port mappings for you.
The ingress is best in this scenario, because you can have one ingress listing on port 80, and route the traffic to the right service based on many variables, like:
Domain
Url Path
Query String
And many other
Ingress in not just for TLS termination, it is in simple terms a proxy\gateway that will control the routing to the right service, TLS termination is just one of the features.
No, you cant have multiple services listening on port 80, as load balancer wont know where to route them (ingress will, however). If you can affort to host each service on different port you could use load balancer. alternatively, if you have public ip for each service and different backend port on each service you can achieve this.
quote: The protocol and port combination you entered matches another rule used by this load balancer. The protocol and port combination of each load balancing rule and inbound NAT rule on a load balancer must be unique.
again, if you are a developer, you probably do not realize how much more convenient it is to manage certificate on ingress, and not on all individual containers that are supposed to be accessible
Even when I create EC2 instances in a private subnet, they must be able to send traffic to the Internet if I want to register them to a ECS cluster.
I am using a NAT gateway to do this, but I still feel insecure that the instances can send private information to anywhere in case of takeover.
What would be the most compact CIDR range that I can use for the instances' security group, instead of 0.0.0.0/0?
For the moment, you may have to rely on the list of public IP address ranges for AWS, allowing traffic bound for all the CIDR blocks associated with your region.
Part of design for resiliency of much of what AWS does relies on the ability of their service endpoints not to depend on static address assigments and instead to use DNS... but their service endpoints should always be on addresses associated with your region, since very few services violate their practice strict regional separation of service infrastructure.
(CloudFront, Route 53, and IAM do, maybe others, but these are provisioning endpoints, not operational ones. These provisioning-only endpoints do not need to be accessible for most applications to function normally.)
A super common pattern here is to create a proxy in a narrow CIDR and only allow outbound traffic to flow through the proxy. White-list AWS endpoints and log all traffic flowing through the proxy for monitoring/auditing.
AWS Endpoints = https://docs.aws.amazon.com/general/latest/gr/rande.html