Context
I've been working on GraphQL subscriptions for my team and have been trying to set up an API Gateway WebSocket API that listens to incoming requests and forwards the subscriptions to our backend services. The integration for the WebSocket API is a VPC Link that targets an ECS cluster which is running a server. One issue with VPC Link integrations for WebSocket APIs is that the WebSocket request is automatically downgraded to HTTP. Therefore, I've been exploring an alternative solution involving a proxy HTTP server which listens to the downgraded WebSocket request and then establishes a WebSocket request with the actual WebSocket server. The proxy HTTP server is defined with ExpressJS and the WebSocket server uses graphql-ws (similar to this example from Apollo https://www.apollographql.com/docs/apollo-server/data/subscriptions/#enabling-subscriptions, the full server code example listed in item #7).
Problem
My problem is that I am unable to successfully establish a WebSocket connection between the two servers when testing the docker containers locally. The current setup is two local docker containers (HTTP proxy and WebSocket server) that are running on the default "bridge" network. Once both servers are running, I use Postman to send a PUT request to the HTTP proxy server. Ideally, the HTTP proxy server should then open a WebSocket connection with the WebSocket server. However, this leads to a 4406 error: Subprotocol not acceptable. graphql-ws uses the graphql-transport-ws protocol (https://github.com/enisdenjo/graphql-ws/blob/master/PROTOCOL.md) and I've tried to change the protocol defined in the WebSocket class constructor but 4406 error is happening regardless of whether or not I specify a protocol on the new WebSocket.
I've tested the WebSocket server by sending requests directly to it and it works as expected. This is why I believe the issue is the HTTP proxy server and not the WebSocket server. Here is a TypeScript code sample that demonstrates how I'm creating the WebSocket on the proxy server:
import express from 'express';
import WebSocket from 'ws';
const app = express();
app.put('/', (req, res) => {
const body = req.body ? req.body : {};
const socket = new WebSocket(wsEndpoint, {
headers: {
'Sec-WebSocket-Version': 13,
'Sec-WebSocket-Key': 'Ab2SMOa5hg0YUyC1OELtyQ==',
Connection: 'Upgrade',
Upgrade: 'websocket',
'Sec-WebSocket-Extensions':
'permessage-deflate; client_max_window_bits',
Host: 'localhost',
},
protocol: 'graphql-transport-ws',
});
socket.on('error', (err) => {
console.log(JSON.stringify(err));
});
socket.on('open', (s: any) => {
console.log('socket open');
});
socket.on('close', (code, reason) => {
console.log(`socket close: ${code}, reason: ${reason.toString()}`);
});
socket.on('error', (error) => {
console.log('socket error');
console.log(error);
});
socket.on('upgrade', (socket: any, request: any) => {
console.log('socket upgrade');
console.log(request);
});
res.status(200).json({
message: 'Connection established to WebSocket server',
});
});
Log snippet:
Proxy server listening on port 80 <-- app.listen() on port 80
socket upgrade <-- starts once I make the PUT request via Postman
socket open
socket close: 4406, reason: Subprotocol not acceptable
What I've Tried
So far I've tried to open the WebSocket both with a protocol in the WebSocket class constructor, and without a value. However, the 4406 error still shows up in both cases.
I've also tried to follow advice that was discussed in this question: How to create websocket connection between two Docker containers. When I create a user defined network, I still run into this exact same 4406 error.
I suspect that it might be that I'm missing something in the WebSocket constructor or some low-level knowledge about how graphql-ws works.
Related
What are the steps to complete a poll call directly from Javascript (NodeJS) to Kafka (not the HTTP endpoints or proxies), without a library? Examples would be wonderful, but even steps and directions are great.
So for instance, authentication, subscription, etc?
PS: This is mostly to learn the internals of communicating with Kafka.
to Kafka (not the HTTP endpoints or proxies)
Kafka uses a custom TCP protocol, and not HTTP, so what you're looking for doesn't exist.
It'd be suggested to use a library rather than attempt to rewrite this TCP socket communication yourself, but if you wanted to, a completely blank slate would start with
const net = require('net');
const client = new net.Socket();
// Send a connection request to the server.
client.connect({ port: 9092, host: 'kafka' }), function() {
console.log('TCP connection established with the server.');
client.write(...); // some binary data matching Kafka spec
});
client.on('data', function(chunk) {
console.log(`Data received from the server: ${chunk}.`);
});
client.on('end', function() {
console.log('Requested an end to the TCP connection');
});
I'm using Socket.IO to run a WebSocket server locally in NodeJS using the following code:
import express = require('express');
const path = require('path');
import http = require('http');
import { Socket } from 'socket.io';
const app = express();
const server = http.createServer(app);
const socketio = require('socket.io')(server);
app.get('/', (req, res) => {
res.send("Node Server is running");
});
server.listen(3000, function () {
console.log('Example app listening on port 3000!');
});
socketio.on("connection", (socket: Socket) => {
console.log(`connect ${socket.id}`);
console.log(`connect ${socket.handshake.url}`);
socket.on("disconnect", () => {
console.log(`disconnect ${socket.id}`);
});
});
Using a tool like Firecamp, I try to establish a connection on ws://localhost:3000, but to no avail. I eventually use the Socket.IO client to connect from a simple web page by running let socket = io(). It seems the only reason this works is because that call connects to the host serving the page by default, as stated here. Running console.log(socket) and looking at the output, I eventually find that the URL inside the engine field is ws://localhost:3000/socket.io/?EIO=4&transport=websocket&sid=qerg3iHm3IKMOjdNAAAA.
My question is why is the URL so complicated rather than simply ws://localhost:3000? And is there no easier way to get the URL instead of having to access it through dev tools?
A socket.io server does not accept generic webSocket connections. It only accepts socket.io connections as socket.io goes through an extra layer of preparation stuff (over http) before establishing the actual webSocket connection. It then also adds a layer on top of the regular webSocket packet format to support some of its features (such as message names).
When using a socket client to connect to a socket.io server in the default configuration, socket.io first makes a few regular http requests to the socket.io server and with those http requests it sends a few parameters. In your URL:
ws://localhost:3000/socket.io/?EIO=4&transport=websocket&sid=qerg3iHm3IKMOjdNAAAA
The path:
/socket.io/
Is the path that the socket.io server is looking for requests on as destined for the socket.io server. Since this is a unique path and not generally used by other requests, this allows you to share an http server between socket.io and other http requests. In fact, this is a common way to deploy a socket.io server (hooking into an http server that you are already using for http requests).
In fact, the path /socket.io/socket.io.js is also served by the socket.io server and that will return the client-side socket.io.js file. So, clients often use this in their HTML files:
<script src="/socket.io/socket.io.js"></script>
as a means of getting the socket.io client code. Again you see the use of the path prefix /socket.io on all socket.io related URLs.
In your original URL, you can see parameters for:
EIO=4 // engine.io protocol version
transport=websocket // desired transport once both sides agree
sid=qerg3iHm3IKMOjdNAAAA // client identifier so the server knows which client this
// is before the actual webSocket connection is established
Once both sides agree that the connection looks OK, then the client will make a webSocket connection to the server. In cases where webSocket connections are blocked (by network equipment that doesn't support them or blocks them), then socket.io will use a form of http polling where it repeatedly "polls" the server asking for any more data and it will attempt to simulate a continuous connection. The client configuration can avoid this http polling and go straight to a webSocket connection if you want, but you would give up the fallback behavior in case continuous webSocket connections are blocked.
And is there no easier way to get the URL instead of having to access it through dev tools?
Not really. This URL is not something you have to know at all. The socket.io client will construct this URL for you. You just specify http://localhost:3000 as the URL you want to connect to and the socket.io client will add the other parameters to it.
I am trying to build a command-line chat room using Node.js and Socket.io.
This is my server-side code so far, I have tried this with both http initialisations (with express, like on the official website's tutorial, and without it):
#app = require('express')()
#http = require('http').Server(app)
http = require('http').createServer()
io = require('socket.io')(http)
io.sockets.on 'connect', (socket) ->
console.log 'a user connected'
http.listen 3000, () ->
console.log 'listening on *:3000'
I start this with nodejs server.js, the "Listening on" is showing up.
If I run lsof -i tcp:3000, the server.js process shows up.
However, when I start this client-side code:
socket = require('socket.io-client')('localhost:3000', {})
socket.on 'connect', (socket) ->
console.log "Connected"
No luck... When I run nodejs client.js, neither "connect" events, from server nor client, are fired!
My questions are :
- What am I doing wrong?
- Is it necessary to start a HTTP server to use it? Sockets are on the transport layer, right? So in theory I don't need a HTTP protocol to trade messages.
If this is a server to server connection and you're only making a socket.io connection (not also setting it up for regular HTTP connections), then this code shows the simple way for just a socket.io connection:
Listening socket.io-only server
// Load the library and initialize a server on port 3000
// This will create an underlying HTTP server, start it and bind socket.io to it
const io = require('socket.io')(3000);
// listen for incoming client connections and log connect and disconnect events
io.on('connection', function (socket) {
console.log("socket.io connect: ", socket.id);
socket.on('disconnect', function() {
console.log("socket.io disconnect: ", socket.id);
});
});
Node.js socket.io client - connects to another socket.io server
// load the client-side library
const io = require('socket.io-client');
// connect to a server and port
const socket = io('http://localhost:3000');
// listen for successful connection to the server
socket.on('connect', function() {
console.log("socket.io connection: ", socket.id);
});
This code works on my computer. I can run two separate node.js apps on the same host and they can talk to one another and both see the connect and disconnect events.
Some Explaining
The socket.io protocol is initiated by making an HTTP connection to an HTTP server. So, anytime you have a socket.io connection, there is an HTTP server listening somewhere. That HTTP connection is initially sent with some special headers that indicate to the server that this is a request to "upgrade" to the webSocket protocol and some additional security info is included.
This is pretty great reference on how a webSocket connection is initially established. It will show you step by step what happens.
Once both sides agree on the "upgrade" in protocol, then the protocol is switched to webSocket (socket.io is then an additional protocol layer on top of the base webSocket protocol, but the connection is all established at the HTTP/webSocket level). Once the upgrade is agreed upon, the exact same TCP connection that was originally the incoming HTTP connection is repurposed and becomes the webSocket/socket.io connection.
With the socket.io server-side library, you can either create the HTTP server yourself and then pass that to socket.io or you can have socket.io just create one for you. If you're only using socket.io on this server and not also sharing using http server for regular http requests, then you can go either way. The minimal code example above, just lets socket.io create the http server for you transparently and then socket.io binds to it. If you are also fielding regular web requests from the http server, then you would typically create the http server first and then pass it to socket.io so socket.io could bind to the http server you already have.
Then, keep in mind that socket.io is using the webSocket transport. It's just some additional packet structure on top of the webSocket transport. It would akin to agreeing to send JSON across an HTTP connection. HTTP is the host transport and underlying data format. Both sides then agree to format some data in JSON format and send it across HTTP. The socket.io message format sits on top of webSocket in that way.
Your Questions
Is it necessary to start a HTTP server to use it?
Yes, an HTTP server must exist somewhere because all socket.io connections start with an HTTP request to an HTTP server.
Sockets are on the transport layer, right?
The initial connection protocol stack works like this:
TCP <- HTTP protocol
Then, after the protocol upgrade:
TCP <- webSocket <- socket.io
So after the protocol upgrade from HTTP to the webSocket transport, you then have socket.io packet format sitting on top of the webSocket format sitting on top of TCP.
So in theory I don't need a HTTP protocol to trade messages.
No, that is not correct. All connections are initially established with HTTP. Once the upgrade happens to the webSocket transport, HTTP is no longer used.
I am running a websocket service using the nodejs-websocket module.
I would like to detect and take some action when a connection attempt is made that is not using websocket protocols.
I've thought of using setTimeout(n) and clearing the timeout when conn.readystate changes from 'connecting' to 'open' or taking an action upon the timer expiring but I'm hoping that there is a more direct way.
How can I detect a non-websocket connection attempt?
The webocket protocol starts every connection with an HTTP request with a header set that "requests" an upgrade to the webSocket protocol. When both the server agrees, it responds with the upgrade and both client and server then change the protocol to the webSocket protocol.
So, a non-webSocket connection attempt will either be a plain HTTP connection or it will be just a socket connection attempt that isn't even HTTP. So, in either case, the connection will die at the HTTP server either as a 404 (an HTTP request that the web server doesn't have a response for) or as an invalid protocol (not HTTP).
Connection has an error event, so you can do Connection.on('error', function(err) .... This will mean something went wrong on the connection, but I don't believe you can do anything with the connection at that point.
One suggestion would be to use Socket.io which can party on an HTTP/Connect/Express server. It can speak HTTP(Kinda) or WebSockets. Then you could do something like:
var app = require('express')();
var server = require('http')
.Server(app);
var io = require('socket.io')(server);
server.listen(3000);
app.get('/*', function(req, res) {
res.json({
hello: 'world'
});
});
io.on('connection', function(socket) {
socket.emit('news', {
hello: 'world'
});
socket.on('my other event', function(data) {
console.log(data);
});
});
This has all the Socket.io connections under the /Socket.io/* path. Any other connection will get handled by the app.get('/*',...) call. This would only catch http requests.
I may not be entering the correct search terms but I cannot seem to find good examples that allow my node application to initiate a socket.io client connection to another telnet server (non-node).
Below is my node app trying to connect to a telnet server
var ioc = require('socket.io-client'),
clientSocket = ioc.connect('192.168.1.97', {
port: 23
});
clientSocket.on('connect', function(){
console.log('connected to to telnet');
});
clientSocket.on('connect_error', function(data){
console.log('connection error to telnet');
console.log(data);
});
clientSocket.on('connect_timeout', function(data){
console.log('connection timeout to telnet');
console.log(data);
});
Here is the error I get
connection error to telnet
timeout
connection timeout to telnet
20000
I've telneted directly to the telnet server successfully from the terminal. Bad code?
You can't.
Socket.IO has nothing to do with regular TCP network sockets. Socket.IO is an RPC layer providing web-socket-like functionality over several transports (Web Sockets, long-polling AJAX, etc.). You can't just connect to any server you want, you must connect to a Socket.IO server. Even Web Sockets itself has a whole protocol built on top of HTTP that must be set up.
If you want to connect to an arbitrary server to send/receive data, that connection must be proxied server-side through your Node.js application. Socket.IO is only for communication between a Socket.IO client and a Socket.IO server.
Not Sure if this can be done, but have a look at this package
https://www.npmjs.org/package/node-telnet-client