I'm using Google's Container Engine service, and got a pod running a server listening on port 3000. I set up the service to connect port 80 to that pod's port 3000. I am able to curl the service using its local and public ip from within the node, but not from outside. I set up a firewall rule to allow port 80 and send it to the node, but I keep getting 'connection refused' from outside the network. I'm trying to do this without a forwarding rule, since there's only one pod and it looked like forwarding rules cost money and do load balancing. I think the firewall rule works, because when I add the createExternalLoadBalancer: true to the service's spec, the external IP created by the forwarding rule works as expected. Do I need to do something else? Set up a route or something?
controller.yaml
kind: ReplicationController
apiVersion: v1beta3
metadata:
name: app-frontend
labels:
name: app-frontend
app: app
role: frontend
spec:
replicas: 1
selector:
name: app-frontend
template:
metadata:
labels:
name: app-frontend
app: app
role: frontend
spec:
containers:
- name: node-frontend
image: gcr.io/project_id/app-frontend
ports:
- name: app-frontend-port
containerPort: 3000
targetPort: 3000
protocol: TCP
service.yaml
kind: Service
apiVersion: v1beta3
metadata:
name: app-frontend-service
labels:
name: app-frontend-service
app: app
role: frontend
spec:
ports:
- port: 80
targetPort: app-frontend-port
protocol: TCP
publicIPs:
- 123.45.67.89
selector:
name: app-frontend
Edit (additional details):
Creating this service adds these additional rules, found when I run iptables -L -t nat
Chain KUBE-PORTALS-CONTAINER (1 references)
target prot opt source destination
REDIRECT tcp -- anywhere 10.247.247.206 /* default/app-frontend-service: */ tcp dpt:http redir ports 56859
REDIRECT tcp -- anywhere 89.67.45.123.bc.googleusercontent.com /* default/app-frontend-service: */ tcp dpt:http redir ports 56859
Chain KUBE-PORTALS-HOST (1 references)
target prot opt source destination
DNAT tcp -- anywhere 10.247.247.206 /* default/app-frontend-service: */ tcp dpt:http to:10.241.69.28:56859
DNAT tcp -- anywhere 89.67.45.123.bc.googleusercontent.com /* default/app-frontend-service: */ tcp dpt:http to:10.241.69.28:56859
I don't fully understand iptables, so I'm not sure how the destination port matches my service. I found that the DNS for 89.67.45.123.bc.googleusercontent.com resolves to 123.45.67.89.
kubectl get services shows the IP address and port I specified:
NAME IP(S) PORT(S)
app-frontend-service 10.247.243.151 80/TCP
123.45.67.89
Nothing recent from external IPs is showing up in /var/log/kube-proxy.log
TL;DR: Use the Internal IP of your node as the public IP in your service definition.
If you enable verbose logging on the kube-proxy you will see that it appears to be creating the appropriate IP tables rule:
I0602 04:07:32.046823 24360 roundrobin.go:98] LoadBalancerRR service "default/app-frontend-service:" did not exist, created
I0602 04:07:32.047153 24360 iptables.go:186] running iptables -A [KUBE-PORTALS-HOST -t nat -m comment --comment default/app-frontend-service: -p tcp -m tcp -d 10.119.244.130/32 --dport 80 -j DNAT --to-destination 10.240.121.42:36970]
I0602 04:07:32.048446 24360 proxier.go:606] Opened iptables from-host portal for service "default/app-frontend-service:" on TCP 10.119.244.130:80
I0602 04:07:32.049525 24360 iptables.go:186] running iptables -C [KUBE-PORTALS-CONTAINER -t nat -m comment --comment default/app-frontend-service: -p tcp -m tcp -d 23.251.156.36/32 --dport 80 -j REDIRECT --to-ports 36970]
I0602 04:07:32.050872 24360 iptables.go:186] running iptables -A [KUBE-PORTALS-CONTAINER -t nat -m comment --comment default/app-frontend-service: -p tcp -m tcp -d 23.251.156.36/32 --dport 80 -j REDIRECT --to-ports 36970]
I0602 04:07:32.052247 24360 proxier.go:595] Opened iptables from-containers portal for service "default/app-frontend-service:" on TCP 23.251.156.36:80
I0602 04:07:32.053222 24360 iptables.go:186] running iptables -C [KUBE-PORTALS-HOST -t nat -m comment --comment default/app-frontend-service: -p tcp -m tcp -d 23.251.156.36/32 --dport 80 -j DNAT --to-destination 10.240.121.42:36970]
I0602 04:07:32.054491 24360 iptables.go:186] running iptables -A [KUBE-PORTALS-HOST -t nat -m comment --comment default/app-frontend-service: -p tcp -m tcp -d 23.251.156.36/32 --dport 80 -j DNAT --to-destination 10.240.121.42:36970]
I0602 04:07:32.055848 24360 proxier.go:606] Opened iptables from-host portal for service "default/app-frontend-service:" on TCP 23.251.156.36:80
Listing the iptables entries using -L -t shows the public IP turned into the reverse DNS name like you saw:
Chain KUBE-PORTALS-CONTAINER (1 references)
target prot opt source destination
REDIRECT tcp -- anywhere 10.119.240.2 /* default/kubernetes: */ tcp dpt:https redir ports 50353
REDIRECT tcp -- anywhere 10.119.240.1 /* default/kubernetes-ro: */ tcp dpt:http redir ports 54605
REDIRECT udp -- anywhere 10.119.240.10 /* default/kube-dns:dns */ udp dpt:domain redir ports 37723
REDIRECT tcp -- anywhere 10.119.240.10 /* default/kube-dns:dns-tcp */ tcp dpt:domain redir ports 50126
REDIRECT tcp -- anywhere 10.119.244.130 /* default/app-frontend-service: */ tcp dpt:http redir ports 36970
REDIRECT tcp -- anywhere 36.156.251.23.bc.googleusercontent.com /* default/app-frontend-service: */ tcp dpt:http redir ports 36970
But adding the -n option shows the IP address (by default, -L does a reverse lookup on the ip address, which is why you see the DNS name):
Chain KUBE-PORTALS-CONTAINER (1 references)
target prot opt source destination
REDIRECT tcp -- 0.0.0.0/0 10.119.240.2 /* default/kubernetes: */ tcp dpt:443 redir ports 50353
REDIRECT tcp -- 0.0.0.0/0 10.119.240.1 /* default/kubernetes-ro: */ tcp dpt:80 redir ports 54605
REDIRECT udp -- 0.0.0.0/0 10.119.240.10 /* default/kube-dns:dns */ udp dpt:53 redir ports 37723
REDIRECT tcp -- 0.0.0.0/0 10.119.240.10 /* default/kube-dns:dns-tcp */ tcp dpt:53 redir ports 50126
REDIRECT tcp -- 0.0.0.0/0 10.119.244.130 /* default/app-frontend-service: */ tcp dpt:80 redir ports 36970
REDIRECT tcp -- 0.0.0.0/0 23.251.156.36 /* default/app-frontend-service: */ tcp dpt:80 redir ports 36970
At this point, you can access the service from within the cluster using both the internal and external IPs:
$ curl 10.119.244.130:80
app-frontend-5pl5s
$ curl 23.251.156.36:80
app-frontend-5pl5s
Without adding a firewall rule, attempting to connect to the public ip remotely times out. If you add a firewall rule then you will reliably get connection refused:
$ curl 23.251.156.36
curl: (7) Failed to connect to 23.251.156.36 port 80: Connection refused
If you enable some iptables logging:
sudo iptables -t nat -I KUBE-PORTALS-CONTAINER -m tcp -p tcp --dport
80 -j LOG --log-prefix "WTF: "
And then grep the output of dmesg for WTF it's clear that the packets are arriving on the 10. IP address of the VM rather than the ephemeral external IP address that had been set as the public IP on the service.
It turns out that the problem is that GCE has two types of external IPs: ForwardingRules (which forward with the DSTIP intact) and 1-to-1 NAT (which actually rewrites the DSTIP to the internal IP). The external IP of the VM is the later type so when the node receives the packets the IP tables rule doesn't match.
The fix is actually pretty simple (but non-intuitive): Use the Internal IP of your node as the public IP in your service definition. After updating your service.yaml file to set publicIPs to the Internal IP (e.g. 10.240.121.42) you will be able to hit your application from outside of the GCE network.
If you add the node's external IP address to the service's publicIPs field, you should then be able to access it at the node's IP address. If your cluster has multiple nodes, you can put more than one of their IP addresses into the field if you'd like to enable accessing the pod on any of them.
In an upcoming release there will be a simpler built-in option for setting up an external service without a load balancer. If you're curious or reading this sometime in the future, check out the updated "External Services" section of this doc to see how you'll be able to use use NodePort to accomplish the same thing much more easily.
Answer by #Robert Bailey is absolutely correct. publicIPs has been deprecated in kubernetes 1.5.1, you can use externalIPs instead.
get the internal ip of the node, kubectl describe node | grep Address
Addresses: 10.119.244.130,101.192.150.200,gke-...
or you can run the ifconfig eth0 inside the node terminal to get the internal ip
set the ip in service.yaml
spec:
type: NodePort
externalIPs:
- 10.119.244.130
can curl with resolve to test
curl --resolve 'example.com:443:23.251.156.36' https://example.com -k
Related
I'm running Debian 8 with iptables.
I have the following rule:
iptables -t mangle -A PREROUTING -p tcp --dport 5000 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 4000
I simply want to redirect all traffic going with destination port 5000 to port 4000.
The standard iptables REDIRECT is not usable in my case, as it alters the packet and changes the original destination port.
Looking at iptables -t mangle -nvL I can see the rule being hit:
Chain PREROUTING (policy ACCEPT 5056 packets, 13M bytes)
pkts bytes target prot opt in out source destination
12 720 TPROXY tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5000 TPROXY redirect 0.0.0.0:4000 mark
0x1/0x1
But my service running on port 4000 doesn't intercept the packets.
I have a simple NodeJS application listening for all TCP on port 4000, which doesn't get any packets:
server.listen(4000, () => { console.log('listening on 4000'); });
Also, running wireshark on TCP port 4000 on all interfaces doesn't show anything.
You also need to set up the routing rule:
# 1 is --tproxy-mark parameter in iptables command
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
I'd like to capture a HTTP service call from HostA -> HostB to test the client on HostA. Both OS are Linux. I tried following but fail.
What's the recommended way to do this?
I would like to use transparent proxy mode because I cannot modify client and cannot redirect all traffic from HostA to hostB as other service also running on HostA. I'd like only redirect the connection of the client from host A to host B.
The client in Host A call a service on Host B on a certain port 10001 by HTTP.
I tried setup HostC with mitmproxy (HostA and HostC are in the same subnet)
HostA (ip_A) -> HostC(ip_C) with mitmproxy-> HostB(ip_B) , I set the ip table to build transparent mode.
Following is what I setup for on HostA
sudo iptables -t mangle -I OUTPUT -p tcp --dport 10001 -j MARK --set-mark 1
sudo ip route add default via ip_C table 100
sudo ip rule add fwmark 0x1 table 100
On HostC
sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -t nat -A PREROUTING -o eth0 -p tcp --dport 10001 -j REDIRECT --to-port 8080
mitmproxy -T --host
This doesn't work. client on HostA connection timeout.
If I try traceroute on HostA
traceroute ip_B -p 10000 -T
It shows ip_B is unreachable on TCP from HostA
I also tried setup mitmproxy on HostA, but when I try to redirect traffic of port 10001
on HostA
sudo iptables -t nat -A OUTPUT -p tcp --dport 10001 -j REDIRECT --to-port 8080
mitmproxy -T --host
The service call could be capture by mitmproxy on HostA but cannot get response.
Thanks a lot for your help.
I'm stumped.
This is how my iptables are configured on Debian 7.
sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 8090 -j ACCEPT
-A FORWARD -d 127.0.0.1/32 -i eth0 -p tcp -m state --state NEW -m tcp --dport 8090 -j ACCEPT
-A FORWARD -d 10.1.130.5/32 -i eth0 -p tcp -m state --state NEW -m tcp --dport 8090 -j ACCEPT
Basically forwarding port 80 to port 8090.
I also have an instance of Apache Tomcat running and listening on port 8090. e.g.
sudo lsof -i :8090
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 15081 user1 36u IPv6 164737 0t0 TCP *:8090 (LISTEN)
However, whenever I try to connect via a browser nothing get returned. Its the same using Wget. e.g.
wget www.test.com/confluence
--2016-04-22 16:59:22-- http://www.test.com/confluence
Resolving www.test.com... 10.1.130.5
Connecting to www.test.com|10.1.130.5|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: /bootstrap/selectsetupstep.action [following]
--2016-04-22 16:59:22-- http://se- www.test.com/bootstrap/selectsetupstep.action
Reusing existing connection to www.test.com:80.
HTTP request sent, awaiting response...
There is nothing in your ruleset that actually forwards ports. You have INPUT rules, which will accept or reject packets destined for the local host, and you have FORWARD rules, which will accept or reject rules transiting the machine to another address, but you don't have anything that actually changes the target port of a connection.
If you actually want to change some aspect of a connection, this falls into the broad category of "network address translation" (NAT), which is carried out in the nat table, rather than the default filter table.
Possibly you need REDIRECT rule in your nat table:
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 \
-j REDIRECT --to-ports 8090
Note that a REDIRECT rule will only operate on traffic that does not originate on the local host. You will need to test this from another host on the network (or from inside a container or a vm).
Alternatively, you could accomplish something similar using proxy software such as haproxy.
Some additional reading:
http://www.cyberciti.biz/faq/linux-port-redirection-with-iptables/
https://serverfault.com/questions/179200/difference-beetween-dnat-and-redirect-in-iptables
https://wiki.debian.org/Firewalls-local-port-redirection
Master node has 2 nic one with ip 10.0.0.1 and one with ip 192.168.2.1.
Slaves have ip on 192.168.2.0 network.
I would like to submit programs to the cluster from client on network 10.0.0.0.
is it possible ?
working with the spark-env.sh SPARK_MASTER_IP parameter is not enough.
Edit:
Even port forwarding is not enough.
iptables -A PREROUTING -t nat -i eth1 -p tcp --dport 9999 -j DNAT --to 192.168.2.11:7077
iptables -A FORWARD -p tcp -d 192.168.2.11 --dport 7077 -j ACCEPT
and for example pyspark --master=spark://publicipmaster:9999 returns the error
WARN ReliableDeliverySupervisor: Association with remote system
[akka.tcp://sparkMaster#publicipmaster:9999] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
I have a Node.js express website that was listening on port 9000, the thing which was fine until I changed the port to 80, now it accepts only connection from local:
wget http://127.0.0.1/ -O -
curl 127.0.0.1:80
Locally it's working fine and they return the html page. But it does not accept remote connections from browser, either using the external IP address or the domain name.
# iptables
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
Well, i have figured the issue, it was my iptables rules forwarding remote connections from outside to another port.