I'm trying to use SSE with node + express: I intercept requests using an express route, then I initiate a SSE session by directly writing headers:
res.writeHead(200, {
"content-type": "text/event-stream",
"cache-control": "no-cache"
});
I proceed with writing intermittent payloads using "res.write()"s.
This works well with Chrome's EventSource, up until the time when I call ".close()" to end the session. Then, the connection keeps hanging: Chrome doesn't reuse the connection to initiate additional EventSource requests (or any other requests), and node never triggers a "close" event on the IncomingMessage instance.
My question is: How do I handle "eventSource.close()" properly using node's http API?
It's worth noting that:
Since I don't set a "content-length", Node automatically assumes "chunked" transfer encoding (this shouldn't be a problem AFAIK). It also defaults to "connection: keep-alive".
The session terminates OK when I'm using Firefox.
When the browser closes the event source it does let the server side know. On the server side, the response socket object (res.socket) will generate an end event followed by a close event. You can listen to this event and respond appropriately.
E.g.
res.socket.on('end', e => {
console.log('event source closed');
sseResponses = sseResponses.filter(x => x != res);
res.end();
});
If your server is trying to write to a socket closed on the browser, it should not raise an error, but will return false from res.write.
If both your server side code and client side code are hanging after you close the event source, you may have bugs on both sides.
More complete prototype, with your writeHead code from above.
var app = new (require('express'));
var responses = [];
app.get("/", (req, res) => {
res.status(200).send(`
<html>
<script>
var eventsource = null;
function connect() {
if (!eventsource) {
eventsource = new EventSource("/sse");
eventsource.onmessage = function(e) {
var logArea = window.document.getElementById('log');
logArea.value += e.data;
};
}
}
function disconnect() {
if (eventsource) {
var myeventsource = eventsource;
eventsource = null;
myeventsource.close();
}
}
</script>
<div>
<span>
Connect
Disconnect
<span>
</div>
<textarea id="log" style="width: 500px; height: 500px"></textarea>
</html>`);
});
app.get("/sse", (req, res) => {
res.writeHead(200, {
"content-type": "text/event-stream",
"cache-control": "no-cache"
});
res.socket.on('end', e => {
responses = responses.filter(x => x != res);
res.end();
});
responses.push(res);
});
app.listen(8080);
setInterval(() => {
responses.forEach(res => {
res.write('data: .\n\n');
});
}, 100);
Related
I have a mock backend in nodejs/express that I need to get working. There is an SSE setup like this:
app.get("/api/sseConnect" ,(req, res) => {
headers = {
"Content-Type": "text/event-stream",
Connection: "keep-alive",
"Access-Control-Allow-Origin": "*",
"Cache-Control": "no-transform",
};
res.writeHead(200, headers);
let intervalID = setInterval(() => {
res.write(`data: ${JSON.stringify(Math.random())}\n\n`);
}, 5000);
res.on("close", () => {
clearInterval(intervalID);
res.end();
});
});
This is working great - the client hits this route, the connection is established, and the client recieves the message every 5 seconds.
There are other standard routes, which when accessed, modify some data in the database. Once the data is modified, I need to send a server-sent-event. So for example:
app.post("/api/notifications/markasread", (req, res) => {
let { ids } = req.body;
ids.forEach(id => database.notifications[id].read = true) // or whatever
// Send a server-sent-event with some information that contains id, message status, etc
})
I know this seems really silly (why not just send a response???), but this is the way that the live api is set up - there is no response from this post route (or certain other routes). They needs to trigger an SSE, which is listened for on the front end with an EventSource instance. Based on what is heard in the eventSource.onmessage listener, a whole bunch of things happen on the front end (react-redux).
How can I 'hijack' the SSEs and trigger a response from a standard POST or GET route?
Server Sent Events are a valuable tool to open a persistent connection to a web server, where the server has the ability to push new data to the client, when available.
Using this technology in Node.js is quite straightforward and can be implemented with the following code example:
#!/usr/bin/env node
'use strict';
const http = (options, listener) => require('http').createServer(listener).listen(options.port);
http({ port: 8080 }, (req, res) => {
switch (req.url) {
case '/server-sent-events': {
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache',
});
const sendDate = () => res.write(`data: ${new Date()}\n\n`);
sendDate();
const interval = setInterval(sendDate, 1000);
req.on('close', () => clearInterval(interval));
} break;
default: {
res.writeHead(200, {
'Content-Type': 'text/html; charset=utf-8',
});
res.end(`
<!DOCTYPE html>
<html>
<head>
<title>Server Send Events</title>
<meta charset="utf-8">
<script>
const sse = new EventSource('/server-sent-events');
sse.onerror = () => document.body.innerHTML = 'Connection Error';
sse.onmessage = ({ data }) => document.body.innerHTML = data;
</script>
</head>
<body></body>
</html>
`);
}
}
});
Unfortunately I am not able to achieve the same goal with Deno, as there is no simple write method on the request object, but I guess it has to be implemented somehow using the req.w buffer. Can you help me please finish off the following example code, so the Server Sent Events can be utilised with Deno as well?
#!/usr/bin/env deno run --allow-net
import { listenAndServe as http } from 'https://deno.land/std/http/server.ts';
http({ port: 8080 }, (req) => {
switch (req.url) {
case '/server-sent-events': {
// missing steps:
// * setup the server sent event headers
// * create the interval and send the date periodically
// * clear the interval when the connection gets closed
} break;
default: {
req.respond({
headers: new Headers({
'Content-Type': 'text/html; charset=utf-8',
}),
body: `
<!DOCTYPE html>
<html>
<head>
<title>Server Send Events</title>
<meta charset="utf-8">
<script>
const sse = new EventSource('/server-sent-events');
sse.onerror = () => document.body.innerHTML = 'Connection Error';
sse.onmessage = ({ data }) => document.body.innerHTML = data;
</script>
</head>
<body></body>
</html>
`,
});
}
}
});
Thank you very much for your support!
[Update 2021-11-04]:
I have made some progress doing some research across different sources (https://deno.land/std#0.76.0/http/server.ts, https://github.com/denoland/deno/issues/4817) and got a step closer to the solution. Using the updated example below at least the setup and usage of the Server Sent Events do work now. The remaining issue (besides cleaning up and refactoring of the code) remains the safe detection when the incoming request has been closed (see comments in the source code below):
#!/usr/bin/env deno run --allow-net
import { listenAndServe as http } from 'https://deno.land/std/http/server.ts';
http({ port: 8080 }, (req) => {
switch (req.url) {
case '/server-sent-events': {
// set up a quick´n´dirty write method without error checking
req.write = (data) => {
req.w.write(new TextEncoder().encode(data));
req.w.flush();
};
// setup the server sent event headers
let headers = '';
headers += 'HTTP/1.1 200 OK\r\n';
headers += 'Connection: keep-alive\r\n';
headers += 'Cache-Control: no-cache\r\n';
headers += 'Content-Type: text/event-stream\r\n';
headers += '\r\n';
req.write(headers);
// create the interval and send the date periodically
const sendDate = () => req.write(`data: ${new Date()}\n\n`);
sendDate();
const interval = setInterval(sendDate, 1000);
// final missing step:
// * clear the interval when the connection gets closed
// currently dropping the connection from the client will
// result in the error: Uncaught (in promise) BrokenPipe:
// Broken pipe (os error 32)
// this error also does not seem to be catchable in the
// req.write method above, so there needs to be another safe
// way to prevent this error from occurring.
} break;
default: {
req.respond({
headers: new Headers({
'Content-Type': 'text/html; charset=utf-8',
}),
body: `
<!DOCTYPE html>
<html>
<head>
<title>Server Send Events</title>
<meta charset="utf-8">
<script>
const sse = new EventSource('/server-sent-events');
sse.onerror = () => document.body.innerHTML = 'Connection Error';
sse.onmessage = ({ data }) => document.body.innerHTML = data;
</script>
</head>
<body></body>
</html>
`,
});
}
}
});
[Update 2021-04-16]
All issues have been resolved and are posted in my accepted answer below.
Deno's http library doesn't support SSE, but you can use Oak Framework, or implement it yourself.
import { Application, Router } from "https://deno.land/x/oak/mod.ts";
const app = new Application();
const router = new Router();
router.get('/', ctx => {
ctx.response.body = `
<!DOCTYPE html>
<html>
<head>
<title>Server Send Events</title>
<meta charset="utf-8">
<script>
const sse = new EventSource('/server-sent-events');
sse.onerror = () => document.body.innerHTML = 'Connection Error';
sse.onmessage = ({ data }) => document.body.innerHTML = data;
</script>
</head>
<body></body>
</html>
`;
})
router.get("/server-sent-events", (ctx) => {
const target = ctx.sendEvents();
const sendDate = () => target.dispatchMessage(`${new Date()}`);
sendDate();
const interval = setInterval(sendDate, 1000);
});
app.use(router.routes());
await app.listen({ port: 8080 });
In the end I have found an answer to my question and so the full answer with plenty of comments follows, so you get a working version of server sent events in Deno. The solution below also solves the os error 32, which gets caused by not catching the connection writer flash method:
#!/usr/bin/env deno run --allow-net
// imports
import { ServerRequest, listenAndServe as http } from 'https://deno.land/std/http/server.ts';
// commodity
const encoder = new TextEncoder();
const print = console.log;
// start the web-server
// this one allows the endpoint `/server-sent-events`, which hosts a clock that
// will be refreshed every second (the efficiency of the clock solution could of
// course be optimised, as every client gets its own clock interval, but this
// this does not matter as this example wants to show how to setup and clean a
// task for every connecting client)
// all other requests will be answered with a simple html page that subscribes
// to the sse-based clock
http({ port: 8080 }, async (req) => {
// ip address of the client (formatted as `ip:port`, so we cut the `:port` part
// of it)
const ip = req.headers.get('host').split(':').slice(0, -1).join(':');
// determine the endpoint to access
switch (req.url) {
// host the server sent event based clock
case '/server-sent-events': {
// logging
print(`+ Client ${ip} connected`);
// prepare the disconnect promise. we will use this one later on to await
// the clients disconnect, so we can properly clean up. so the promise will
// be resolved manually by us when we detect a disconnect from the client
// on an attempt to send new data to him (unfortunately there seems to be
// no other way to detect when the client actually closed the connection)
let resolver;
const disconnect = new Promise((resolve) => resolver = resolve);
// write helper
req.write = async (data) => {
// send the current data to the client
req.w.write(encoder.encode(data));
// to actually send the data we need to flush the writer first. we need
// to try/catch this part, as not handling errors on flush will lead to
// the `Broken pipe (os error 32)` error
try {
await req.w.flush();
} catch(err) {
// throw any errors but the broken pipe, which gets thrown when the
// client has already disconnected and we try to send him new data
// later on
if (err.name !== 'BrokenPipe') {
throw err;
}
// close the connection from our side as well
req.conn.close();
// resolve our `disconnect` promise, so we can clean up
resolver();
}
};
// date writer (interval method which pushes the current date to the client)
const sendDate = async () => await req.write(`data: ${new Date()}\n\n`);
// prepare and send the headers
let headers = '';
headers += `HTTP/1.1 200 OK\r\n`;
headers += `Connection: keep-alive\r\n`;
headers += `Cache-Control: no-cache\r\n`;
headers += `Content-Type: text/event-stream\r\n`;
headers += `\r\n`;
await req.write(headers);
// send the date now for the first time and then every second
sendDate();
const interval = setInterval(sendDate, 1000);
// await until the clients disconnects to clean up. so we will be "stuck"
// here until a disconnect gets detected as we use a promise based approach
// to detect the disconnect
await disconnect;
clearInterval(interval);
// logging
print(`- Client ${ip} disconnected`);
} break;
// all other requests host a simple html page which subscribes to the clock
default: {
print(`* Serve website to ${ip}`);
req.respond({
headers: new Headers({
'Content-Type': 'text/html; charset=utf-8',
}),
body: `
<!DOCTYPE html>
<html>
<head>
<title>Server Sent Events</title>
<meta charset="utf-8">
<script>
const sse = new EventSource('/server-sent-events');
sse.onerror = () => document.body.innerHTML = 'Connection Error';
sse.onmessage = ({ data }) => document.body.innerHTML = data;
</script>
</head>
<body></body>
</html>
`,
});
}
}
});
This example seems more idiomatic.
import { serve } from "https://deno.land/std#0.116.0/http/server.ts";
const msg = new TextEncoder().encode("data: hello\r\n");
serve(async (_) => {
let timerId: number | undefined;
const body = new ReadableStream({
start(controller) {
timerId = setInterval(() => {
controller.enqueue(msg);
}, 1000);
},
cancel() {
if (typeof timerId === "number") {
clearInterval(timerId);
}
},
});
return new Response(body, {
headers: {
"Content-Type": "text/event-stream",
},
});
});
console.log("Listening on http://localhost:8000");
I have a server which looks like this:
const http2 = require('http2');
const {
HTTP2_HEADER_METHOD,
HTTP2_HEADER_PATH,
HTTP2_HEADER_STATUS,
HTTP2_HEADER_CONTENT_TYPE
} = http2.constants;
const fs = require('fs');
const server = http2.createSecureServer({
key: fs.readFileSync('./ssl/localhost-privkey.pem'),
cert: fs.readFileSync('./ssl/localhost-cert.pem')
});
server.on('error', (err) => {
console.error(err);
});
server.on('stream', (stream, headers,flags) => {
stream.respond({
'content-type': 'text/html',
[HTTP2_HEADER_STATUS]: 200,
[HTTP2_HEADER_CONTENT_TYPE]: 'text/plain'
});
stream.end('<h1>Hello World 2</h1>');
});
server.on('request', (msg) => {
/* THIS IS BEING FIRED TWICE*/
console.log('request:' + JSON.stringify(msg) );
});
server.on('session', (msg) => {
/* THIS IS ALSO BEING FIRED TWICE*/
console.log('session:' + JSON.stringify(msg) );
});
server.listen(8443);
From my browser I type into the url https://myserver:8443. On my server I can see the session event is consoled log twice. Why is this happening since I am only making one request? As well everytime I refresh the page the request event is being fired twice instead of only once. I am using nodejs 11.0.0
You should log the URL that is being requested with console.log(msg.url). You will likely find that one of the requests is for the favicon.ico as this is something that a browser will request when it doesn't already have a favicon cached for a particular domain.
All requests of a web server have to look at the actual resource being requested and respond appropriately based on the exact resource being requested.
I'm piping to a file an HTTPS request, it works ok 99.9% of calls, but occasionally (maybe when server or network are not available) hangs indefinitely...
This obviously cause my application to stop working and requiring a manual restart...
I have other https connections that used to occasionally hang that always complete now using the following error code on the request object, as suggested on node documentation:
request.on('socket', function(socket) {
socket.setTimeout(10000);
socket.on('timeout', function() { request.abort(); });
});
request.on('error', function(e) {
// Handle the error...
console.error("FAILED!");
});
... but it seems that timeouts on the request are ignored if the destination is piped to a file stream, maybe I should handle an error with a timeout on the filesystem object, but the documentation is not clear if there is an event I have to wait for except for 'finish'...
Here is the sample code, I hope someone can help me:
var https = require('https'),
fs = require('fs');
var opts = {
host: 'www.google.com',
path: '/',
method: 'GET',
port: 443
};
var file = fs.createWriteStream('test.html');
var request = https.request(opts, function(response) {
response.pipe(file);
file.on('finish', function() {
file.close(function(){
console.log("OK!");
});
});
});
request.on('socket', function(socket) {
socket.setTimeout(10000);
socket.on('timeout', function() { request.abort(); });
});
request.on('error', function(e) {
console.error("FAILED!");
});
request.end();
If you wanna try the hang, change host and path with a huge file and disconnect the network cable during the transfer, it should time out after 10 seconds, but it doesn't...
I set up a demo node.js http server that sends a very slow answer and a client similar to your sample code.
When I start the client and then stop the server while sending the response then I also don't get a timeout event on the socket but I get a end event on the response within the client:
var request = https.request(opts, function(response) {
response.pipe(file);
file.on('finish', function() {
file.close(function(){
console.log("OK!");
});
});
response.on('end', function() {
// this is printed when I stop the server
console.log("response ended");
});
});
```
Maybe you could listen to that event?
I spawned the following child: var spw = spawn('ping', ['-n','10', '127.0.0.1']) and I would like to receive the ping results on the client side (browser) one by one, not as a whole.
So far I tried this:
app.get('/path', function(req, res) {
...
spw.stdout.on('data', function (data) {
var str = data.toString();
res.write(str + "\n");
});
...
}
and that:
...
spw.stdout.pipe(res);
...
In both cases browser waits 10 of the pings to complete, and then prints the result as a whole. I would like to have them one by one, how to accomplish that?
(Client is just making a call to .../path and console.logs the result)
EDIT: Although I do believe that websockets are necessary to implement this, I just want to know whether there are any other ways. I saw several confusing SO answers, and blog posts (in this post, at step one OP streams the logs to the browser) which didn't help, therefore I decided to go for a bounty for some attention.
Here's a complete example using SSE (Server sent events). This works in Firefox and probably Chrome too:
var cp = require("child_process"),
express = require("express"),
app = express();
app.configure(function(){
app.use(express.static(__dirname));
});
app.get('/msg', function(req, res){
res.writeHead(200, { "Content-Type": "text/event-stream",
"Cache-control": "no-cache" });
var spw = cp.spawn('ping', ['-c', '100', '127.0.0.1']),
str = "";
spw.stdout.on('data', function (data) {
str += data.toString();
// just so we can see the server is doing something
console.log("data");
// Flush out line by line.
var lines = str.split("\n");
for(var i in lines) {
if(i == lines.length - 1) {
str = lines[i];
} else{
// Note: The double-newline is *required*
res.write('data: ' + lines[i] + "\n\n");
}
}
});
spw.on('close', function (code) {
res.end(str);
});
spw.stderr.on('data', function (data) {
res.end('stderr: ' + data);
});
});
app.listen(4000);
And the client HTML:
<!DOCTYPE Html>
<html>
<body>
<ul id="eventlist"> </ul>
<script>
var eventList = document.getElementById("eventlist");
var evtSource = new EventSource("http://localhost:4000/msg");
var newElement = document.createElement("li");
newElement.innerHTML = "Messages:";
eventList.appendChild(newElement);
evtSource.onmessage = function(e) {
console.log("received event");
console.log(e);
var newElement = document.createElement("li");
newElement.innerHTML = "message: " + e.data;
eventList.appendChild(newElement);
};
evtSource.onerror = function(e) {
console.log("EventSource failed.");
};
console.log(evtSource);
</script>
</body>
</html>
Run node index.js and point your browser at http://localhost:4000/client.html.
Note that I had to use the "-c" option rather than "-n" since I'm running OS X.
If you are using Google Chrome, changing the content-type to "text/event-stream" does what your looking for.
res.writeHead(200, { "Content-Type": "text/event-stream" });
See my gist for complete example: https://gist.github.com/sfarthin/9139500
This cannot be achieved with the standard HTTP request/response cycle. Basically what you are trying to do is make a "push" or "realtime" server. This can only be achieved with xhr-polling or websockets.
Code Example 1:
app.get('/path', function(req, res) {
...
spw.stdout.on('data', function (data) {
var str = data.toString();
res.write(str + "\n");
});
...
}
This code never sends an end signal and therefore will never respond. If you were to add a call to res.end() within that event handler, you will only get the first ping – which is the expected behavior because you are ending the response stream after the first chunk of data from stdout.
Code Sample 2:
spw.stdout.pipe(res);
Here stdout is flushing the packets to the browser, but the browser will not render the data chunks until all packets are received. Thus the reason why it waits 10 seconds and then renders the entirety of stdout. The major benefit to this method is not buffering the response in memory before sending — keeping your memory footprint lightweight.