hyperledger on multiple clusters : ingress and dns routing failing - azure

Question

I am using 2 AKS(azure k8s clusters) with net peering to spin up hyperledger fabric components(one org on cluster-1 and org2 and orderers on cluster-2).
the issue I am facing is about using the ingress for the services/pods to communicate with each other.
between the 2 clusters we have a pvt DNS to route the requests to appropriate ingress points in the clusters. and have ingress resources configured(with FQDN) to further route them to the services.
but, they are failing to talk to each other. As a first priority, I am concentrating on getting the orderers talk and elect a leader. All the orderers are in the same cluster. so using the FQDN it queries the DNS for IP of the Ingress point(which it is able to do. confirmed by using ping in the pod. but the ping fails) and then onwards I am not sure what's happening.
The requests gets into an error at IP addresses of 2 pods strangely every time.
2022-06-21 01:28:24.294 UTC [core.comm] ServerHandshake -> ERRO 19558 Server TLS handshake failed in 5.000908746s with error read tcp 10.192.0.72:7050->10.192.0.55:24182: i/o timeout server=Orderer remoteaddress=**10.192.0.55**:24182
2022-06-21 01:28:24.294 UTC [grpc] Warningf -> DEBU 19559 grpc: Server.Serve failed to complete security handshake from "10.192.0.55:24182": read tcp 10.192.0.72:7050->10.192.0.55:24182: i/o timeout
2022-06-21 01:28:25.495 UTC [core.comm] ServerHandshake -> ERRO 1955a Server TLS handshake failed in 5.000462596s with error read tcp 10.192.0.72:7050->**10.192.0.4**:11226: i/o timeout server=Orderer remoteaddress=10.192.0.4:11226
2022-06-21 01:28:25.495 UTC [grpc] Warningf -> DEBU 1955b grpc: Server.Serve failed to complete security handshake from "10.192.0.4:11226": read tcp 10.192.0.72:7050->10.192.0.4:11226: i/o timeout
2022-06-21 01:28:28.781 UTC [orderer.consensus.etcdraft] Step -> INFO 1955c 2 is starting a new election at term 1 channel=syschannel node=2
the 2 ip addresses 10.192.0.4 and 10.192.0.55 are of kubernetes azure-ip-masq-agent/ calico-node. below is my ingress resource.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-orderer3
namespace: istakapazatestnet
uid: 03736608-2e29-4744-8b07-eaa67408f72e
resourceVersion: '6818484'
generation: 1
creationTimestamp: '2022-06-20T14:47:55Z'
labels:
component: orderer3
type: orderer
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/rewrite-target":"/","nginx.ingress.kubernetes.io/ssl-redirect":"true"},"labels":{"component":"orderer3","type":"orderer"},"name":"ingress-orderer3","namespace":"istakapazatestnet"},"spec":{"rules":[{"host":"orderer3.istakapaza.com","http":{"paths":[{"backend":{"service":{"name":"orderer3-service","port":{"number":7050}}},"path":"/","pathType":"ImplementationSpecific"}]}}],"tls":[{"hosts":["orderer3.istakapaza.com"],"secretName":"ssl-secret"}]}}
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
selfLink: >-
/apis/networking.k8s.io/v1/namespaces/istakapazatestnet/ingresses/ingress-orderer3
status:
loadBalancer:
ingress:
- ip: 10.224.0.6
spec:
tls:
- hosts:
- orderer3.istakapaza.com
secretName: ssl-secret
rules:
- host: orderer3.istakapaza.com
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: orderer3-service
port:
number: 7050
I did also try a couple of combinations for the ports in configtx.yaml for orderers along with addresses and also attaching ssl certs to ingress. Looks like I am not able to figure this out. Any help will be much appreciated.
I know it sounds simple to route all the services through ingress but somehow I am missing something.
if anyone need further resources please just ping me. below is one combination I tried for configtx. I think we are not supposed to change configtx but I tried it anyway.
Orderer: &OrdererDefaults
OrdererType: etcdraft
EtcdRaft:
Consenters:
- Host: orderer.istakapaza.com
Port: 443
ClientTLSCert: /scripts/crypto-config/ordererOrganizations/istakapaza.com/orderers/orderer.istakapaza.com/tls/server.crt
ServerTLSCert: /scripts/crypto-config/ordererOrganizations/istakapaza.com/orderers/orderer.istakapaza.com/tls/server.crt
- Host: orderer2.istakapaza.com
Port: 443
ClientTLSCert: /scripts/crypto-config/ordererOrganizations/istakapaza.com/orderers/orderer2.istakapaza.com/tls/server.crt
ServerTLSCert: /scripts/crypto-config/ordererOrganizations/istakapaza.com/orderers/orderer2.istakapaza.com/tls/server.crt
- Host: orderer3.istakapaza.com
Port: 443
ClientTLSCert: /scripts/crypto-config/ordererOrganizations/istakapaza.com/orderers/orderer3.istakapaza.com/tls/server.crt
ServerTLSCert: /scripts/crypto-config/ordererOrganizations/istakapaza.com/orderers/orderer3.istakapaza.com/tls/server.crt
Addresses:
- https://orderer.istakapaza.com
- https://orderer2.istakapaza.com
- https://orderer3.istakapaza.com
BatchTimeout: 2s