I am in the process of writing an intercepting proxy tool like Burpsuite for security testing. An important part of that would be sending malformed HTTP requests in the case of which we would have to give the user full control over the request!
So, I can't have complete control while using a library! I need to be able to send raw HTTP requests to the target hosts like,
GET / HTTP/1.1
Host: google.com
My attempt :-
I tried using the node JS net module, and I was able to connect to host on port 80 (HTTP), and while connecting to port 443 (HTTPS), a connection is established but returns an empty response!
On some researching, I found out that this has something to do with SSL, as I tried telnet and it too failed for HTTPS connections and by looking at some stackoverflow answers!
Is there any option through which I can directly send raw HTTP/HTTPS requests directly from my node application?
Thanks!
There is a module http-tag, which allow writing literal http messages like -
const net = require('net')
const HTTPTag = require('http-tag')
const socket = net.createConnection({
host: 'localhost',
port: 8000,
}, () => {
// This callback is run once, when socket connected
// Instead of manually writing like this:
// socket.write('GET / HTTP/1.1\r\n')
// socket.write('My-Custom-Header: Header1\r\n\r\n')
// You will be able to write your request(or response) like this:
const xHeader = 'Header1' // here in the epressions you can pass any characters you want
socket.write(
HTTPTag`
GET / HTTP/1.1
My-Custom-Header: ${xHeader}
`
)
socket.end()
})
socket.on('close', hasError => console.log(`Socket Closed, hasError: ${hasError}`))
// set readable stream encoding
socket.setEncoding('utf-8')
socket.on('data', data => console.log(data))
Regarding TLS, currently i am in research on built-in node modules, and I haven’t view the tls yet.
Related
I have been really struggling to send data from Matlab over a network to a series of 'Dashboards' written in HTML/JS that essentially just display the data.
In Matlab I use uSend = udpport("datagram","IPV4","LocalHost","127.0.0.1","LocalPort",3333) then write(uSend,D,"char","LocalHost",2560) to send an array D=jsonencode([1,2,3,4,5]) to port 2560.
My current implementation uses NodeJS 'dgram' to receive the data. The implementation below works:
const dgram = require('dgram');
const socket = dgram.CreateSocket('udp4');
socket.on('message', (msg,rinfo) => {
console.log(`Data Received: ${JSON.parse(msg)}`)
})
socket.bind(2560,"127.0.0.1")
BUT: This only works with NodeJS i.e. run the script above node script.js. The 'Dashboards' need to run essentially on a chrome browser, which dgram won't work (not even with browserify, it's not a supported function).
Hence, my hands are sort of tied, with Matlab I can realistically only send UDP (it's multicast) and I can't get UDP data on the JS side of things.
I was wondering, with webRTC, is it possible to get it to listen to a port? e.g. something like webRTC listen to port 2560 #127.0.0.1?
Any help would be much appreciated! I am relatively new to programming so I may be asking the wrong question here. Thanks
WebRTC requires a lot more then just a UDP port unfortunately.
I would suggest instead running a node.js server that accepts incoming multicast traffic and caches like this.
const dgram = require('dgram');
const socket = dgram.CreateSocket('udp4');
let data = [];
socket.on('message', (msg,rinfo) => {
data.push(JSON.parse(msg))
})
socket.bind(2560,"127.0.0.1")
Then I would provide a HTTP endpoint that returns the JSON. You can have the browser poll this at an interval of your choosing.
const http = require('http');
const server = http.createServer((req, res) => {
response.end(JSON.stringify(data));
})
const port = 3000
server.listen(port, () => {
console.log(`Server listening on port ${port}`);
})
You can then test this by doing curl http://localhost:3000. You are just caching the data then making it available via HTTP request.
In the future I would look into using a Websocket instead. This way the server can push the data to the browser as soon as it is available. Using a HTTP endpoint requires the browser to request on an interval.
I'm using Socket.IO to run a WebSocket server locally in NodeJS using the following code:
import express = require('express');
const path = require('path');
import http = require('http');
import { Socket } from 'socket.io';
const app = express();
const server = http.createServer(app);
const socketio = require('socket.io')(server);
app.get('/', (req, res) => {
res.send("Node Server is running");
});
server.listen(3000, function () {
console.log('Example app listening on port 3000!');
});
socketio.on("connection", (socket: Socket) => {
console.log(`connect ${socket.id}`);
console.log(`connect ${socket.handshake.url}`);
socket.on("disconnect", () => {
console.log(`disconnect ${socket.id}`);
});
});
Using a tool like Firecamp, I try to establish a connection on ws://localhost:3000, but to no avail. I eventually use the Socket.IO client to connect from a simple web page by running let socket = io(). It seems the only reason this works is because that call connects to the host serving the page by default, as stated here. Running console.log(socket) and looking at the output, I eventually find that the URL inside the engine field is ws://localhost:3000/socket.io/?EIO=4&transport=websocket&sid=qerg3iHm3IKMOjdNAAAA.
My question is why is the URL so complicated rather than simply ws://localhost:3000? And is there no easier way to get the URL instead of having to access it through dev tools?
A socket.io server does not accept generic webSocket connections. It only accepts socket.io connections as socket.io goes through an extra layer of preparation stuff (over http) before establishing the actual webSocket connection. It then also adds a layer on top of the regular webSocket packet format to support some of its features (such as message names).
When using a socket client to connect to a socket.io server in the default configuration, socket.io first makes a few regular http requests to the socket.io server and with those http requests it sends a few parameters. In your URL:
ws://localhost:3000/socket.io/?EIO=4&transport=websocket&sid=qerg3iHm3IKMOjdNAAAA
The path:
/socket.io/
Is the path that the socket.io server is looking for requests on as destined for the socket.io server. Since this is a unique path and not generally used by other requests, this allows you to share an http server between socket.io and other http requests. In fact, this is a common way to deploy a socket.io server (hooking into an http server that you are already using for http requests).
In fact, the path /socket.io/socket.io.js is also served by the socket.io server and that will return the client-side socket.io.js file. So, clients often use this in their HTML files:
<script src="/socket.io/socket.io.js"></script>
as a means of getting the socket.io client code. Again you see the use of the path prefix /socket.io on all socket.io related URLs.
In your original URL, you can see parameters for:
EIO=4 // engine.io protocol version
transport=websocket // desired transport once both sides agree
sid=qerg3iHm3IKMOjdNAAAA // client identifier so the server knows which client this
// is before the actual webSocket connection is established
Once both sides agree that the connection looks OK, then the client will make a webSocket connection to the server. In cases where webSocket connections are blocked (by network equipment that doesn't support them or blocks them), then socket.io will use a form of http polling where it repeatedly "polls" the server asking for any more data and it will attempt to simulate a continuous connection. The client configuration can avoid this http polling and go straight to a webSocket connection if you want, but you would give up the fallback behavior in case continuous webSocket connections are blocked.
And is there no easier way to get the URL instead of having to access it through dev tools?
Not really. This URL is not something you have to know at all. The socket.io client will construct this URL for you. You just specify http://localhost:3000 as the URL you want to connect to and the socket.io client will add the other parameters to it.
It must be a simple issue, but my knowledge of streams is limited.
HTTP/1 80 to HTTP/2 h2c proxy
script(not working):
const net = require('net');
const http = require('http');
const http2 = require('http2');
const socketPath = `/tmp/socket.test.${Date.now()}`;
// front http 80 server.
http.createServer((req, res) => {
const socket = net.createConnection(socketPath)
req.pipe(socket).pipe(res);
}).listen(80);
// private http2 socket server.
http2.createServer(function(socket) {
socket.write(`Echo from http2 server\r\n`);
socket.pipe(socket);
}).listen(socketPath);
HTTP/2 h2c to HTTP/2 h2c proxy
cli command to start request:
curl --http2-prior-knowledge -v http://localhost:3333/ --output -
script(not working):
const net = require('net');
const http = require('http');
const http2 = require('http2');
const socketPath = `/tmp/socket.test.${Date.now()}`;
const port = 3333;
const private = http2.createServer({allowHTTP1: true});
private.on('stream', (stream, headers) => {
console.log('private http2 request');
stream.end('HTTP/2');
});
private.listen(socketPath, () => console.log('private http2 server is listening', socketPath));
const public = http2.createServer({allowHTTP1: true});
public.on('stream', (stream, headers) => {
console.log('public http2 request');
const socket = net.connect(socketPath);
stream.pipe(socket).pipe(stream);
});
public.listen(port, () => console.log('public http2 server is listening port', port));
Finally, http2(h2c) to http2(h2c) (with unix socket) proxy works!
const net = require('net');
const http2 = require('http2');
const socketPath = `/tmp/socket.test.${Date.now()}`;
const port = 4444;
const priv = http2.createServer({});
priv.on('stream', (stream, headers) => {
console.log('private http2 request');
stream.end('HTTP/2');
});
priv.listen(socketPath, () => console.log('private http2 server is listening', socketPath));
const pub = http2.createServer({});
pub.on('stream', (stream, headers) => {
const clientSession = http2.connect('http://0.0.0.0', {
createConnection: () => net.connect({path: socketPath})
});
const req = clientSession.request({
':path': `/`,
});
req.pipe(stream).pipe(req);
});
pub.listen(port, () => console.log('public http2 server is listening port', port));
I am not a node expert, and I may have completely misunderstood what you are trying to do here, but I am really struggling to make sense of the question...
If you are trying to have Node act as a HTTP/2 proxy (so a client can connect via h2c to node and it passes on those details to another HTTP/2 aware server), then the way you are going about it seems... weird to say the least.
A proxy can be a Level 4 proxy (e.g. a TCP proxy), where it creates two TCP separate connections (one from client to proxy, and one from proxy to destination server) and sends the connects of those TCP packets between them without really inspecting or interfering with them, other then the TCP headers.
Alternative a proxy can be a Level 7 proxy (e.g. a HTTP proxy), where it creates two separate HTTP connections (one from client to proxy, and one from proxy to destination server) and sends HTTP messages between them, mapping the HTTP headers and details between them and sometimes changing details or even adding more headers (e.g. X-FORWARDED-FOR).
You seem to be trying to create some kind of hybrid between these two distinct and incompatible modes of working! You are hoping to create a HTTP or HTTP/2 server and then open a TCP socket and pass those TCP messages between them and hope this works? While this might possibly work over a simple protocol like HTTP/1 it is never going to work over HTTP/2!
For your first example a HTTP/1 instance is entirely different to HTTP/2. So to set these two up and expect them to work is flawed from the start. If one of your friends only spoke German and the other friend only spoke Spanish and you passed all the German messages, verbatim, unfiltered, and still in German to the Spanish speaker then would you expect the Spanish speaker to magically be able to understand them? Of course not! So you cannot connect HTTP/1 and HTTP/2 at a socket level - they are completely different protocols. You need your proxy to act like a translator between them.
In your second example I'm even more confused. I guess you are trying to create two HTTP/2 servers, and have the client connect to one, and then proxy requests over to the other? And presumably you would add some logic in there at some stage so only certain requests made it through otherwise this would be pointless. Regardless this will almost certainly not work. HTTP/2 is a complex protocol, in many ways more akin to TCP. So each packet needs to be given a unique stream id, and many other settings need to be negotiated between the two end points. So to assume that one HTTP/2 message will seamlessly translate to an identical HTTP/2 message on another HTTP/2 connection is extremely naive! Back to the language analogy, copying German messages verbatim to another German speaker who is perhaps hard of hearing but sitting closer to you, might work initially but as soon as one end fails to keep up, speaks a slightly different dialect, or asks you to repeat something they missed, the whole show comes tumbling down.
I would suggest you either want to make this a Level 4 proxy (so ignore HTTP and HTTP/2 and just use sockets) or you want this to be a HTTP proxy (in which case ingest each HTTP message, read it, and send a similar HTTP message on the downstream connection). You cannot have both.
I would also question why and if you need to do this? HTTP/2 is not universally supported, and gets most of it's gains between client and edge server (proxy in this case), so why do you feel the need to speak HTTP/2 all the way through? See this question for more details on that: HTTP/2 behind reverse proxy
Hope that helps and apologies if I have completely misunderstood your question or your intention!
I am trying to build a command-line chat room using Node.js and Socket.io.
This is my server-side code so far, I have tried this with both http initialisations (with express, like on the official website's tutorial, and without it):
#app = require('express')()
#http = require('http').Server(app)
http = require('http').createServer()
io = require('socket.io')(http)
io.sockets.on 'connect', (socket) ->
console.log 'a user connected'
http.listen 3000, () ->
console.log 'listening on *:3000'
I start this with nodejs server.js, the "Listening on" is showing up.
If I run lsof -i tcp:3000, the server.js process shows up.
However, when I start this client-side code:
socket = require('socket.io-client')('localhost:3000', {})
socket.on 'connect', (socket) ->
console.log "Connected"
No luck... When I run nodejs client.js, neither "connect" events, from server nor client, are fired!
My questions are :
- What am I doing wrong?
- Is it necessary to start a HTTP server to use it? Sockets are on the transport layer, right? So in theory I don't need a HTTP protocol to trade messages.
If this is a server to server connection and you're only making a socket.io connection (not also setting it up for regular HTTP connections), then this code shows the simple way for just a socket.io connection:
Listening socket.io-only server
// Load the library and initialize a server on port 3000
// This will create an underlying HTTP server, start it and bind socket.io to it
const io = require('socket.io')(3000);
// listen for incoming client connections and log connect and disconnect events
io.on('connection', function (socket) {
console.log("socket.io connect: ", socket.id);
socket.on('disconnect', function() {
console.log("socket.io disconnect: ", socket.id);
});
});
Node.js socket.io client - connects to another socket.io server
// load the client-side library
const io = require('socket.io-client');
// connect to a server and port
const socket = io('http://localhost:3000');
// listen for successful connection to the server
socket.on('connect', function() {
console.log("socket.io connection: ", socket.id);
});
This code works on my computer. I can run two separate node.js apps on the same host and they can talk to one another and both see the connect and disconnect events.
Some Explaining
The socket.io protocol is initiated by making an HTTP connection to an HTTP server. So, anytime you have a socket.io connection, there is an HTTP server listening somewhere. That HTTP connection is initially sent with some special headers that indicate to the server that this is a request to "upgrade" to the webSocket protocol and some additional security info is included.
This is pretty great reference on how a webSocket connection is initially established. It will show you step by step what happens.
Once both sides agree on the "upgrade" in protocol, then the protocol is switched to webSocket (socket.io is then an additional protocol layer on top of the base webSocket protocol, but the connection is all established at the HTTP/webSocket level). Once the upgrade is agreed upon, the exact same TCP connection that was originally the incoming HTTP connection is repurposed and becomes the webSocket/socket.io connection.
With the socket.io server-side library, you can either create the HTTP server yourself and then pass that to socket.io or you can have socket.io just create one for you. If you're only using socket.io on this server and not also sharing using http server for regular http requests, then you can go either way. The minimal code example above, just lets socket.io create the http server for you transparently and then socket.io binds to it. If you are also fielding regular web requests from the http server, then you would typically create the http server first and then pass it to socket.io so socket.io could bind to the http server you already have.
Then, keep in mind that socket.io is using the webSocket transport. It's just some additional packet structure on top of the webSocket transport. It would akin to agreeing to send JSON across an HTTP connection. HTTP is the host transport and underlying data format. Both sides then agree to format some data in JSON format and send it across HTTP. The socket.io message format sits on top of webSocket in that way.
Your Questions
Is it necessary to start a HTTP server to use it?
Yes, an HTTP server must exist somewhere because all socket.io connections start with an HTTP request to an HTTP server.
Sockets are on the transport layer, right?
The initial connection protocol stack works like this:
TCP <- HTTP protocol
Then, after the protocol upgrade:
TCP <- webSocket <- socket.io
So after the protocol upgrade from HTTP to the webSocket transport, you then have socket.io packet format sitting on top of the webSocket format sitting on top of TCP.
So in theory I don't need a HTTP protocol to trade messages.
No, that is not correct. All connections are initially established with HTTP. Once the upgrade happens to the webSocket transport, HTTP is no longer used.
I am trying to do a game in html5 with serverside logic in node.js and that uses raw websockets (not Socket.IO, I need binary data). I wish to have multiple "rooms", thus multiple websocket servers, all having separate URLs. Currently, I only found a way to have each websocket server attached to a specific port, and then proxy the upgrade requests(not entirely sure how it works) to the right port based on the url.
It works on my computer. The problem is that when I try to submit it to a PaaS provider (AppFog), the code fails because they don't permit opening any ports other than the provided http port.
Here is a pretty cleared up version of my code:
//start web server (basic static express server) on 8080
// ...
//start game server and listen to port 9000
// I use the ws module for websockets
// I plan to have a couple of these "game servers"
// ...
//open the proxy server.
var httpProxy= require('http-proxy');
var webProxyServer = httpProxy.createServer(function (req, res, proxy){
// I need http requests to be redirected to the "game servers"
if(req.url.substring(0, "/room1".length) === "/room1") // if starts with "/room1"
proxy.proxyRequest(req, res, {
host: 'localhost',
port: 9000
});
else
proxy.proxyRequest(req, res, {
host: 'localhost',
port: 8080
});
}
webProxyServer.on('upgrade', function (req, socket, head) {
//redirecting logic goes here
if(req.url=="/room1/"){
webProxyServer.proxy.proxyWebSocketRequest(req, socket, head, {
host: 'localhost',
port: 9000
})
}
});
webProxyServer.listen(8000); //the "outside port".
My question: is it somehow possible to open websocket servers without listening to any specific ports, and to manually attach sockets to them so I don't need to open any ports other than the basic http port? I know Socket.IO somehow does it. Maybe there is a way to listen to the upgrade event of a http server and pass the socket to the right websocket server?
I am pretty new to server-side stuff, so extra info here and there would be welcome.
Unfortunately, Appfog does not support websockets.
Feature Roadmap page - bottom of page: shows websockets as something coming soon (i.e. they don't support it).