net.DialTCP produces "connection refused" error on Linux but not on Windows - linux

Question

Code
To reproduce requires two application running and connecting to each other through TCP. So I've made a tiny repo that also includes the powershell build script. link to the full repo
However to avoid the extra click, here is the code for clientA.go.
package main
import (
"fmt"
"net"
"time"
)
func main() {
clientA, err := net.ResolveTCPAddr("tcp4", fmt.Sprintf(":%v", "2222"))
if err != nil {
fmt.Println(err)
return
}
clientB, err := net.ResolveTCPAddr("tcp4", fmt.Sprintf(":%v", "3333"))
if err != nil {
fmt.Println(err)
return
}
for {
clientAtoB, err := net.DialTCP("tcp4", clientA, clientB)
if err != nil {
fmt.Println(err)
} else {
defer clientAtoB.Close()
clientAtoB.SetLinger(0)
clientAtoB.SetNoDelay(true)
clientAtoB.SetKeepAlive(false)
fmt.Println("connected as Client A!")
buffer := make([]byte, 64)
_, err = clientAtoB.Read(buffer)
if err != nil {
continue
}
}
time.Sleep(time.Second)
}
}
The code for clientB.go is identical except the local and remote endpoints are swapped around:
clientBtoA, err := net.DialTCP("tcp4", clientB, clientA)
Problem
I build the same go code for both Windows and Linux but at runtime the applications produce different results. Specifically with how TCP connections are dialed on each platform.
On Windows, when I run the two executables clientA.exe and clientB.exe (built from the build.ps1 script) I get the desired result. As seen in this screenshot:
However when I upload and execute the Linux binaries, the result is different:
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ sudo chmod +x clientA clientB
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ ls -la
total 10984
drwxrwxr-x 3 ubuntu ubuntu 4096 Apr 27 03:09 .
drwxrwxr-x 4 ubuntu ubuntu 4096 Apr 27 03:08 ..
drwxrwxr-x 8 ubuntu ubuntu 4096 Apr 27 03:08 .git
-rw-rw-r-- 1 ubuntu ubuntu 11255 Apr 27 03:12 A.txt
-rw-rw-r-- 1 ubuntu ubuntu 11255 Apr 27 03:12 B.txt
-rw-rw-r-- 1 ubuntu ubuntu 247 Apr 27 03:08 build.ps1
-rwxrwxr-x 1 ubuntu ubuntu 2950662 Apr 27 03:08 clientA
-rw-rw-r-- 1 ubuntu ubuntu 2642944 Apr 27 03:08 clientA.exe
-rw-rw-r-- 1 ubuntu ubuntu 718 Apr 27 03:08 clientA.go
-rwxrwxr-x 1 ubuntu ubuntu 2950662 Apr 27 03:08 clientB
-rw-rw-r-- 1 ubuntu ubuntu 2642944 Apr 27 03:08 clientB.exe
-rw-rw-r-- 1 ubuntu ubuntu 718 Apr 27 03:08 clientB.go
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ ./clientA > A.txt & ./clientB > B.txt &
[1] 24914
[2] 24915
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ cat A.txt
dial tcp4 :2222->:3333: connect: connection refused
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ cat B.txt
dial tcp4 :3333->:2222: connect: connection refused
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$
I don't expect the connection refused error since these two applications are running under the same environment, so no firewalls are in effect, and the permissions are identical.
How can I get the same result regardless of platform? Or why are they different in the first place?
Edit
The successful connection on Windows is not just the luck of good timing. On Windows, I can run A for 5 minutes, then when I run B, both connect successfully.
Update (2020-04-27)
After receiving feedback from Go developers, I've been told that this is likely a Linux configuration issue and not specific to Go. Other than permissions, I can't thing of anything that would prevent two applications in the same environment from establishing a TCP connection like this? (These low level Linux stuff isn't really my forte.)

Answer 1

Why this doesn't work on Linux is quite obvious. Both A and B are clients that are connecting to counterpart that needs to listen. On Linux (or UNIX) if you try to run ClientA it will try to dial in to ClientB's address and port. If there's no process already listening on this address and port to accept the connection in that moment ClientA will immediately end up with connection refused error (this is not entirely true, but most of time is, see my EDIT at the end of answer).
On Windows, under the hood Golang uses (for tcp, tcp4 and tcp6 protocols) ConnectEx API which is for connection-oriented sockets. This API behaves different from Linux connect API. If ConnectEx cannot connect immediately it returns error code ERROR_IO_PENDING and behind the scenes OS waits/retries until connection is accepted and established (or it gives up and makes it definitively failed) and then notifies back - this is called overlapped I/O.
Relevant part of MSDN ConnectEx documentation:
Connection-oriented sockets are often unable to complete their connection immediately, and therefore the operation is initiated and the function immediately returns with the ERROR_IO_PENDING or WSA_IO_PENDING error. When the connect operation completes and success or failure is achieved, status is reported using the completion notification mechanism indicated in lpOverlapped.
Now, what happens in your case on Windows is that you try to ConnectEx from both sides and OS connects those sockets for you. This will only work if other side gets connected within certain period. If you try to reasonably increase time.Sleep interval in both clients (e.g. 17 and 28), you can see even on Windows they will have hard time to connect anymore.
Answer to your question is that your code as it is written now depends on OS-specific behavior of TCP dialing in Golang on Windows and is not portable. To fix your software to be portable on any platform supported by Golang you probably want to change logic so both ClientA and ClientB listen for incoming connection and also periodically try to connect to the opposite side.
EDIT: I'm not saying your code can not work on Linux at all. It actually uses rare connection mode called TCP simultaneous connect where you can connect two processes without having any of them listen. Both dialing sides send their SYN simultaneously, so each side responds with SYN/ACK and then ACK to complete the 3-way handshake and ESTABLISH connection. That requires very precise timing and syncing of the connect call in both clients. Both sides would connect if TCP simultaneous connect is allowed in Linux kernel and that sync between connects is achieved (hardly done by just running both clients by hand or from same script; even simulating within same process and thread is not that easy).