What are the steps to complete a poll call directly from Javascript (NodeJS) to Kafka (not the HTTP endpoints or proxies), without a library? Examples would be wonderful, but even steps and directions are great.
So for instance, authentication, subscription, etc?
PS: This is mostly to learn the internals of communicating with Kafka.
to Kafka (not the HTTP endpoints or proxies)
Kafka uses a custom TCP protocol, and not HTTP, so what you're looking for doesn't exist.
It'd be suggested to use a library rather than attempt to rewrite this TCP socket communication yourself, but if you wanted to, a completely blank slate would start with
const net = require('net');
const client = new net.Socket();
// Send a connection request to the server.
client.connect({ port: 9092, host: 'kafka' }), function() {
console.log('TCP connection established with the server.');
client.write(...); // some binary data matching Kafka spec
});
client.on('data', function(chunk) {
console.log(`Data received from the server: ${chunk}.`);
});
client.on('end', function() {
console.log('Requested an end to the TCP connection');
});
Related
Context
I've been working on GraphQL subscriptions for my team and have been trying to set up an API Gateway WebSocket API that listens to incoming requests and forwards the subscriptions to our backend services. The integration for the WebSocket API is a VPC Link that targets an ECS cluster which is running a server. One issue with VPC Link integrations for WebSocket APIs is that the WebSocket request is automatically downgraded to HTTP. Therefore, I've been exploring an alternative solution involving a proxy HTTP server which listens to the downgraded WebSocket request and then establishes a WebSocket request with the actual WebSocket server. The proxy HTTP server is defined with ExpressJS and the WebSocket server uses graphql-ws (similar to this example from Apollo https://www.apollographql.com/docs/apollo-server/data/subscriptions/#enabling-subscriptions, the full server code example listed in item #7).
Problem
My problem is that I am unable to successfully establish a WebSocket connection between the two servers when testing the docker containers locally. The current setup is two local docker containers (HTTP proxy and WebSocket server) that are running on the default "bridge" network. Once both servers are running, I use Postman to send a PUT request to the HTTP proxy server. Ideally, the HTTP proxy server should then open a WebSocket connection with the WebSocket server. However, this leads to a 4406 error: Subprotocol not acceptable. graphql-ws uses the graphql-transport-ws protocol (https://github.com/enisdenjo/graphql-ws/blob/master/PROTOCOL.md) and I've tried to change the protocol defined in the WebSocket class constructor but 4406 error is happening regardless of whether or not I specify a protocol on the new WebSocket.
I've tested the WebSocket server by sending requests directly to it and it works as expected. This is why I believe the issue is the HTTP proxy server and not the WebSocket server. Here is a TypeScript code sample that demonstrates how I'm creating the WebSocket on the proxy server:
import express from 'express';
import WebSocket from 'ws';
const app = express();
app.put('/', (req, res) => {
const body = req.body ? req.body : {};
const socket = new WebSocket(wsEndpoint, {
headers: {
'Sec-WebSocket-Version': 13,
'Sec-WebSocket-Key': 'Ab2SMOa5hg0YUyC1OELtyQ==',
Connection: 'Upgrade',
Upgrade: 'websocket',
'Sec-WebSocket-Extensions':
'permessage-deflate; client_max_window_bits',
Host: 'localhost',
},
protocol: 'graphql-transport-ws',
});
socket.on('error', (err) => {
console.log(JSON.stringify(err));
});
socket.on('open', (s: any) => {
console.log('socket open');
});
socket.on('close', (code, reason) => {
console.log(`socket close: ${code}, reason: ${reason.toString()}`);
});
socket.on('error', (error) => {
console.log('socket error');
console.log(error);
});
socket.on('upgrade', (socket: any, request: any) => {
console.log('socket upgrade');
console.log(request);
});
res.status(200).json({
message: 'Connection established to WebSocket server',
});
});
Log snippet:
Proxy server listening on port 80 <-- app.listen() on port 80
socket upgrade <-- starts once I make the PUT request via Postman
socket open
socket close: 4406, reason: Subprotocol not acceptable
What I've Tried
So far I've tried to open the WebSocket both with a protocol in the WebSocket class constructor, and without a value. However, the 4406 error still shows up in both cases.
I've also tried to follow advice that was discussed in this question: How to create websocket connection between two Docker containers. When I create a user defined network, I still run into this exact same 4406 error.
I suspect that it might be that I'm missing something in the WebSocket constructor or some low-level knowledge about how graphql-ws works.
I've a GPS tracker device that sends data to my Node.js TCP server and I can send data back to device from the TCP server.
const net = require('net');
const port = 6565;
const host = '127.0.0.1';
const server = net.createServer(onClientConnection);
function onClientConnection(sock){
console.log(`${sock.remoteAddress}:${sock.remotePort} Connected`);
sock.on('data',function(data){
console.log(`${sock.remoteAddress}:${sock.remotePort} Says : ${data} `);
sock.write("Hello World!");
});
//Handle client connection termination.
sock.on('close',function(){
console.log(`${sock.remoteAddress}:${sock.remotePort} Terminated the connection`);
});
//Handle Client connection error.
sock.on('error',function(error){
console.error(`${sock.remoteAddress}:${sock.remotePort} Connection Error ${error}`);
});
};
server.listen(port,host,function(){
console.log(`Server started on port ${port} at ${host}`);
});
However I'm looking to extend this device/server interaction to a web portal from where I want to send/receive data from the device in real-time and I can't seem to wrap my head around how to approach this problem. Is it possible to do this? The device itself is a low-end device that don't seem to have a embedded web server. Can we create a REST API like interface between the web portal and TCP server to accomplish this task? I'm really lost. Any pointers?
Please help.
We are attempting to ingest data from a streaming sports API in Node.Js, for our React app (a MERN app). According to their API docs: "The livestream read APIs are long running HTTP GET calls. They are designed to be used to receive a continuous stream of game actions." So we am attempting to ingest data from long-running HTTP GET call in Node Js, to use in our React app.
Are long-running HTTP GET calls the same as websockets? We have not ingested streaming data previously, so not sure about either of these.
So far, we have added the following code into the index.js of our node app:
index.js
...
// long-running HTTP request
http.get(`https://live.test.wh.geniussports.com/v2/basketball/read/1234567?ak=OUR_KEY`, res => {
res.on('data', chunk => {
console.log(`NEW CHUNK`);
console.log(Buffer.from(chunk).toString('utf-8')); // simply console log for now
});
res.on('end', () => {
console.log('ENDING');
});
});
// Start Up The Server
const PORT = process.env.PORT || 8080;
app.listen(PORT, () => console.log(`Express server up and running on port: ${PORT}`));
This successfully connects and console-log's data in the terminal, and continues to console log new data as it becomes available in the long-running GET call.
Is it possible to send this data from our http.get() request up to React? Can we create a route for a long-running request in the same way that other routes are made? And then call that routes in React?
Server-sent events works like this, with a text/event-stream content-type header.
Server-sent events have a number of benefits over Websockets, so I wouldn't say that it's an outdated technique, but it looks like they rolled their own SSE, which is definitely not great.
My recommendation is to actually just use SSE for your use-case. Built-in browser support and ideal for a stream of read-only data.
To answer question #1:
Websockets are different from long-running HTTP GET calls. Websockets are full-duplex connections that let you send messages of any length in either direction. When a Websocket is created, it uses HTTP for the handshake, but then changes the protocol using the Upgrade: header. The Websockets protocol itself is just a thin layer over TCP, to enable discrete messages rather than a stream. It's pretty easy to design your own protocol on top of Websockets.
To answer question #2:
I find Websockets flexible and easy to use, though I haven't used the server-sent events that Evert describes in his answer. Websockets could do what you need (as could SSE according to Evert).
If you go with Websockets, note that it's possible to queue a message to be sent, and then (like any network connection) have the Websocket close before it's sent, losing any unsent messages. Thus, you probably want to build in acknowledgements for important messages. Also note that Websockets close after about a minute of being idle, at least on Android, so you need to either send heartbeat messages or be prepared to reconnect as needed.
If you decided to go on with websockets , I would recommend this approach using socket.io for both client and server:
Server-Side code:
const server = require('http').createServer();
const io = require('socket.io')(server);
io.on("connection", (socket) => {
http.get(`https://live.test.wh.geniussports.com/v2/basketball/read/1234567?ak=OUR_KEY`, res => {
res.on('data', chunk => {
console.log(`NEW CHUNK`);
let dataString = Buffer.from(chunk).toString('utf-8)
socket.emit('socketData' , {data:dataString});
});
res.on('end', () => {
console.log('ENDING');
});
});
});
server.liste(PORT);
Client-Side code:
import {
io
} from 'socket.io-client';
const socket = io('<your-backend-url>');
socket.on('socketData', (data) => {
//data is the same data object that we emited from server
console.log(data)
//use your websocket data
})
I've created a very simple socket server with NodeJS (v8.11.3) and it's working fine. My goal is to keep a socket connection opened with an electronic device (IoT).
QUESTION: How to make communication secure, that is, how to make socket/SSL socket communication? NOTE: I have also created a self-signed certificate for testing.
The test socket server (without security) is the one below. I have no experience with NodeJS, so I think there are a lot better ways to establish a socket connection...
const net = require('net')
net.createServer(socket => {
socket.on('error', (err) => {
console.log('Socket Error: ')
console.log(err.stack)
})
socket.on('data', function(data){
msg = ''
msg = data.toString();
socket.write(msg)
console.log(msg)
})
}).listen(8001)
You can use the built-in tls module, which provides extensions of net.Server and net.Socket. As such, it works about the same as the net server creation, with more options and more events. There is a simple example in the tls.createServer section, which shows a basic server close to your code here.
I have node TCP socket server that i implemented using net module as shown in below
const net = require("net");
const server = net.createServer();
server.on("connection", function (socket) {
socket.setEncoding('utf8');
socket.on("data", function (d) {
}
}
socket.on("end", function () {
clients.splice(clients.indexOf(socket), 1);
console.log('end event on socket fired');
});
and i want my angular 6 app as a client to this TCP server. So i explore on the internet i only get with socket.io. Basic scenario is i have two clients one is raspberry which communicate on TCP/IP to my server and one is angular app that communicate with server using http. Any idea how to achieve this?
As of my knowledge plain TCP/UDP connections from the browser is currently deprecated because of security issues. I believe you may have to use WebSockets both on angular and node sides.