Firestore / gRPC behind a corporate firewall / proxy - node.js

Question

Our company has built an electron application using Firestore and now we are trying to deploy the application behind a corporate proxy and firewall (customer environment). After setting the proxy authentication settings using electrons app.on('login') all network requests in the application are successful, except for the firestore connection.
We receive the following error:
[2018-09-21T09:09:13.556Z] #firebase/firestore: Firestore (5.5.0) [Connection]: GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Connect Failed
[2018-09-21T09:09:13.557Z] #firebase/firestore: Firestore (5.5.0) [PersistentStream]: close with error: FirebaseError: [code=unavailable]: 14 UNAVAILABLE: Connect Failed
[2018-09-21T09:09:13.557Z] #firebase/firestore: Firestore (5.5.0): Could not reach Cloud Firestore backend. Connection failed 1 times. Most recent error: FirebaseError: [code=unavailable]: 14 UNAVAILABLE: Connect Failed
This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend.
We have also tried to debug gRPC which is used by Firestore with the following log output:
I0921 11:39:18.014000000 8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:292] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): starting watch: requesting connectivity change notification (from IDLE)
I0921 11:39:18.016000000 8920 connectivity_state.cc:116] CONWATCH: 138B91A8 subchannel: from IDLE [cur=IDLE] notify=08F0B0C4
I0921 11:39:18.016000000 8920 connectivity_state.cc:164] SET: 138B91A8 subchannel: IDLE --> CONNECTING [state_change] error=00000000 "No Error"
I0921 11:39:18.018000000 8920 connectivity_state.cc:190] NOTIFY: 138B91A8 subchannel: 08F0B0C4
I0921 11:39:18.019000000 8920 tcp_client_custom.cc:139] CLIENT_CONNECT: 134C5B98 ipv4:172.217.16.202:443: asynchronously connecting
I0921 11:39:18.020000000 8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:404] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): connectivity changed: state=CONNECTING, error="No Error", shutting_down=0
I0921 11:39:18.021000000 8920 connectivity_state.cc:164] SET: 138D9848 pick_first: CONNECTING --> CONNECTING [connecting_changed] error=00000000 "No Error"
I0921 11:39:18.021000000 8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:313] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): renewing watch: requesting connectivity change notification (from CONNECTING)
I0921 11:39:18.026000000 8920 connectivity_state.cc:116] CONWATCH: 138B91A8 subchannel: from CONNECTING [cur=CONNECTING] notify=08F0B0C4
I0921 11:39:18.027000000 8920 completion_queue.cc:851] grpc_completion_queue_next(cq=08DED7E0, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=00000000)
I0921 11:39:18.028000000 8920 completion_queue.cc:951] RETURN_EVENT[08DED7E0]: QUEUE_TIMEOUT
I0921 11:39:18.030000000 8920 tcp_custom.cc:348] Creating TCP endpoint 134C5B98
I0921 11:39:18.030000000 8920 tcp_client_custom.cc:69] CLIENT_CONNECT: ipv4:172.217.16.202:443: on_alarm: error="Cancelled"
I0921 11:39:18.030000000 8920 handshaker.cc:141] handshake_manager 034B6840: adding handshaker http_connect [137A6430] at index 0
I0921 11:39:18.030000000 8920 ssl_transport_security.cc:211] HANDSHAKE START - TLS client start_connect - !!!!!!
I0921 11:39:18.030000000 8920 ssl_transport_security.cc:211] LOOP - TLS client enter_early_data - !!!!!!
I0921 11:39:18.030000000 8920 ssl_transport_security.cc:211] LOOP - TLS client read_server_hello - !!!!!!
I0921 11:39:18.031000000 8920 handshaker.cc:141] handshake_manager 034B6840: adding handshaker security [08F078C8] at index 1
I0921 11:39:18.031000000 8920 handshaker.cc:212] handshake_manager 034B6840: error="No Error" shutdown=0 index=0, args={endpoint=13454A30, args=08E1AC60 {size=8: grpc.primary_user_agent=grpc-node/1.13.1, grpc.client_channel_factory=58CBB994, grpc.channel_credentials=134C5318, grpc.server_uri=dns:///firestore.googleapis.com, grpc.default_authority=firestore.googleapis.com, grpc.http2_scheme=https,
grpc.security_connector=08EDF460, grpc.subchannel_address=ipv4:172.217.16.202:443}, read_buffer=08D55870 (length=0), exit_early=0}
I0921 11:39:18.031000000 8920 handshaker.cc:253] handshake_manager 034B6840: calling handshaker http_connect [137A6430] at index 0
I0921 11:39:18.031000000 8920 handshaker.cc:212] handshake_manager 034B6840: error="No Error" shutdown=0 index=1, args={endpoint=13454A30, args=08E1AC60 {size=8: grpc.primary_user_agent=grpc-node/1.13.1, grpc.client_channel_factory=58CBB994, grpc.channel_credentials=134C5318, grpc.server_uri=dns:///firestore.googleapis.com, grpc.default_authority=firestore.googleapis.com, grpc.http2_scheme=https,
grpc.security_connector=08EDF460, grpc.subchannel_address=ipv4:172.217.16.202:443}, read_buffer=08D55870 (length=0), exit_early=0}
I0921 11:39:18.031000000 8920 handshaker.cc:253] handshake_manager 034B6840: calling handshaker security [08F078C8] at index 1
I0921 11:39:18.032000000 8920 tcp_custom.cc:234] WRITE 134C5B98 (peer=ipv4:172.217.16.202:443): 16 03 01 00 9f 01 00 00 9b 03 03 20 0d 33 b4 b9 36 9c bc b1 56 cf f9 8b 2c 96 35 7a 10 05 c7 42 8f 0b e8 f5 55 b9 5b d2 03 9e 40 00 00 08 c0 2b c0 2c c0 2f c0 30 01 00 00 6a ff 01 00 01 00 00 00 00 1d 00 1b 00 00 18 66 69 72 65 73 74 6f 72 65 2e 67 6f 6f 67 6c 65 61 70 69 73 2e 63 6f 6d 00 17 00 00 00
23 00 00 00 0d 00 14 00 12 04 03 08 04 04 01 05 03 08 05 05 01 08 06 06 01 02 01 33 74 00 00 00 10 00 0e 00 0c 08 67 72 70 63 2d 65 78 70 02 68 32 00 0b 00 02 01 00 00 0a 00 04 00 02 00 17 '........... .3..6...V...,.5z...B....U.[...#....+.,./.0...j..............firestore.googleapis.com.....#..........................3t.........grpc-exp.h2..............'
I0921 11:39:18.033000000 8920 completion_queue.cc:851] grpc_completion_queue_next(cq=08DED7E0, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=00000000)
I0921 11:39:18.044000000 8920 completion_queue.cc:951] RETURN_EVENT[08DED7E0]: QUEUE_TIMEOUT
I0921 11:39:18.046000000 8920 tcp_custom.cc:217] write complete on 134C5B98: error="No Error"
I0921 11:39:18.046000000 8920 resource_quota.cc:795] RQ anonymous_pool_8f00c70 ipv4:172.217.16.202:443: alloc 8192; free_pool -> -8192
I0921 11:39:18.046000000 8920 resource_quota.cc:292] RQ: check allocation for user 08DDBA00 shutdown=0 free_pool=-8192
I0921 11:39:18.047000000 8920 resource_quota.cc:318] RQ anonymous_pool_8f00c70 ipv4:172.217.16.202:443: grant alloc 8192 bytes; rq_free_pool -> 9223372036854767615
I0921 11:39:18.048000000 8920 tcp_custom.cc:174] TCP:134C5B98 read_allocation_done: "No Error"
I0921 11:39:18.048000000 8920 tcp_custom.cc:191] Initiating read on 134C5B98: error="No Error"
I0921 11:39:18.050000000 8920 completion_queue.cc:851] grpc_completion_queue_next(cq=08DED7E0, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=00000000)
I0921 11:39:18.052000000 8920 completion_queue.cc:951] RETURN_EVENT[08DED7E0]: QUEUE_TIMEOUT
I0921 11:39:18.056000000 8920 resource_quota.cc:818] RQ anonymous_pool_8f00c70 ipv4:172.217.16.202:443: free 8192; free_pool -> 8192
I0921 11:39:18.057000000 8920 tcp_custom.cc:128] TCP:134C5B98 call_cb 08F079B8 58B7D4B0:08F078C8
I0921 11:39:18.064000000 8920 tcp_custom.cc:132] read: error={"created":"#1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}
D0921 11:39:18.065000000 8920 security_handshaker.cc:129] Security handshake failed: {"created":"#1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"#1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}
I0921 11:39:18.066000000 8920 tcp_custom.cc:286] TCP 134C5B98 shutdown why={"created":"#1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"#1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}
I0921 11:39:18.067000000 8920 handshaker.cc:212] handshake_manager 034B6840: error={"created":"#1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"#1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]} shut
down=0 index=2, args={endpoint=00000000, args=00000000 {size=0: (null)}, read_buffer=00000000 (length=0), exit_early=0}
I0921 11:39:18.069000000 8920 handshaker.cc:240] handshake_manager 034B6840: handshaking complete -- scheduling on_handshake_done with error={"created":"#1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"#1537522758.056000000","description":"EOF","file":"..\deps\
grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}
I0921 11:39:18.072000000 8920 connectivity_state.cc:164] SET: 138B91A8 subchannel: CONNECTING --> TRANSIENT_FAILURE [connect_failed] error=08E97DF8 {"created":"#1537522758.071000000","description":"Connect Failed","file":"..\deps\grpc\src\core\ext\filters\client_channel\subchannel.cc","file_line":641,"grpc_status":14,"referenced_errors":[{"created":"#1537522758.065000000","description":"Handshake
read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"#1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}]}
I0921 11:39:18.075000000 8920 connectivity_state.cc:190] NOTIFY: 138B91A8 subchannel: 08F0B0C4
I0921 11:39:18.076000000 8920 subchannel.cc:646] Connect failed: {"created":"#1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"#1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}
I0921 11:39:18.076000000 8920 resource_quota.cc:508] RU shutdown 08DDBA00
I0921 11:39:18.078000000 8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:404] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): connectivity changed: state=TRANSIENT_FAILURE, error={"created":"#1537522758.071000000","description":"Connect Failed","file":"..\deps\grpc\src\core\ext\filters\client_channel\subchannel.cc","file_line":641,"grpc_sta
tus":14,"referenced_errors":[{"created":"#1537522758.065000000","description":"Handshake read failed","file":"..\deps\grpc\src\core\lib\security\transport\security_handshaker.cc","file_line":321,"referenced_errors":[{"created":"#1537522758.056000000","description":"EOF","file":"..\deps\grpc\src\core\lib\iomgr\tcp_uv.cc","file_line":107}]}]}, shutting_down=0
I0921 11:39:18.079000000 8920 src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:332] [pick_first 138D9818] subchannel list 08E19208 index 7 of 11 (subchannel 138B9138): stopping connectivity watch
How can this connectivity problem be solved? Which settings need to be changed in the corporate proxy or firewall in order for Firestore / gRPC to be able to connect?

Answer 1

As can be seen in the source code, the node.js client library accesses Firestore through gRPC at firestore.googleapis.com on port 443.
So for your application to work with Firestore successfully, the firewall/proxy must allow access to firestore.googleapis.com on port 443 via TCP. Why this doesn't work at the moment is a question to the current configuration of the firewall/proxy, most probably it is blocked by default in some catch-all rule.

Answer 2

Warning: GRPC currently only supports HTTP Basic Auth for proxies. The proxy authentication method here will currently not work, if the proxy requires authentication via Digest. I have filed an issue for it here: https://github.com/grpc/grpc/issues/18250
There seem to be two workarounds currently:
Possibility 1: The solution is to set the environment variables for http_proxy and https_proxy before loading the Electron BrowserWindow that initiates the Firestore connection.
An possible way is to ask for proxy credentials in the BrowserWindow and return them to the main process using an event (in this example set-proxy-url). After setting the environment variables correctly for gRPC to pick them up, reload the BrowserWindow (or open a new one).
Example code (main):
ipcMain.on('set-proxy-url', (event: Electron.Event, arg: any) => {
if (arg.authInfo.isProxy) {
// In our case this is a http proxy (HTTPS is tunneled)
const httpUrl = `http://${arg.user}:${arg.pass}#${arg.authInfo.host}:${arg.authInfo.port}`
process.env.http_proxy = httpUrl
process.env.https_proxy = httpUrl
// TODO: Reload the BrowserWindow that uses Firestore
}
})
Example code (renderer):
ipcRenderer.send('set-proxy-url', {
authInfo, // This comes from app.on('login', (event, webContents, request, authInfo, callback) => { ... }
user: usernameRetrievedFromUser,
pass: passwordRetrievedFromUser
})
Possibility 2: Configure a VPN tunnel and configure CNAME in your internal DNS to point *.googleapis.com to private.googleapis.com as explained here.