Slow network on AKS ith Egress policies - azure

Question

I've setup a Kube cluster with AKS, Calico Policy and Azure CNI plugin.
Some of my pods need to connect to external services and I want to setup Egress rules to limit traffic on the pods. Ingress is doing just fine but for whatever reason when I add my Egress rules it works but it seems to be painfully slow.
For example : without Egress my logs telling me my pod is connected to the DB are instant. With egress rules up, it can takes 3 to 10 minutes to connect to my DB.
Before connecting to the DB my pod is getting values from a KeyVault. (I don't use Vault mounting because the variables fetched in the vault are dynamic depending on a certain configuration). So I'm not sure what takes times here.
I've done a tcpdump and ifconfig but I can't see any dropped packets.
I haven't set ports on CIDR yet because I wanted to make sure it was not a matter of port mapping first.
The pod runs on port 3000 and here are my configs for the network policies :
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
namespace: client2
name: default-deny
spec:
podSelector:
matchLabels: {}
policyTypes:
- Ingress
- Egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: client2-network-policy
namespace: client2
spec:
podSelector:
matchLabels:
io.kompose.service: backend
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: ingress-basic
podSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/component: controller
app.kubernetes.io/instance: nginx-ingress
ports:
- protocol: TCP
port: 3000
- from:
- namespaceSelector:
matchLabels:
networking/namespace: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: TCP
port: 53
- protocol: UDP
port: 53
- from: # Keyvault ips fetched from dns lookup
- ipBlock:
cidr: 1.2.3.4/32
- from:
- ipBlock:
cidr: 3.4.5.6/32
- from:
- ipBlock:
cidr: 34.5.4.6/32
- from:
- ipBlock:
cidr: 2.5.6.2/32 #db shard1
- from:
- ipBlock:
cidr: 2.4.5.2/32 #db shard2
- from:
- ipBlock:
cidr: 1.2.4.3/32 #db shard3
egress:
- to:
- namespaceSelector:
matchLabels:
name: ingress-basic
podSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/component: controller
app.kubernetes.io/instance: nginx-ingress
- to:
- namespaceSelector:
matchLabels:
networking/namespace: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: TCP
port: 53
- protocol: UDP
port: 53
- to:
- ipBlock:
cidr: 1.3.8.0/24 #unknown azure service i saw in tcpdump
- to: # Keyvault ips fetched from dns lookup
- ipBlock:
cidr: 1.2.3.4/32
- to:
- ipBlock:
cidr: 3.4.5.6/32
- to:
- ipBlock:
cidr: 34.5.4.6/32
- to:
- ipBlock:
cidr: 2.5.6.2/32 #db shard1
- to:
- ipBlock:
cidr: 2.4.5.2/32 #db shard2
- to:
- ipBlock:
cidr: 1.2.4.3/32 #db shard3
I don't know where to look at to debug the slowness, any ideas or tips ? Thanks !