Is there an API in the browser (outside of websockets) which allows us to stream data from a file to the browser? something like this:
const reader = new FileReader();
reader.readAsText(file, "UTF-8");
reader.on('data', d => { // imaginary api
// new line of data d
});
what could happen is the user selects the local file, and some process on the local OS writes to it. If this doesn't work, then websockets is an option.
Browsers can consume streaming data using the Streams API, here how to use it, from those links:
The basic usage of Streams hinges around making responses available as streams. For example, the response body returned by a successful fetch request can be exposed as a ReadableStream, and you can then read it using a reader created with ReadableStream.getReader(), cancel it with ReadableStream.cancel()
// Fetch the original image
fetch('./tortoise.png')
// Retrieve its body as ReadableStream
.then((response) => {
const reader = response.body.getReader();
// …
});
A good post about the Streams API
Another option could be using server sent events implementing the "streaming" as a sequence of reactions to events (new lines from the file?), still from mdn links EventSource Interface:
Unlike WebSockets, server-sent events are unidirectional; that is, data messages are delivered in one direction, from the server to the client (such as a user's web browser). That makes them an excellent choice when there's no need to send data from the client to the server in message form.
Here a link to another question with a lot of cool info and links
These solutions involve some Server side work of course
Background: Stream Consumption
This is how you can consume and read a stream of data-bytes that is received in the client:
Get a Response object (example a fetch response.)
Retrieve the ReadableStream from the body field of it i.e response.body
As body is an instance of ReadableStream, we can do body.getReader()
And use the reader as reader.read()
That is simple enough, we can consume the stream of bytes coming from the server and we can consume it the same way in NodeJS and Browser through the Node Web Stream API.
Stream from path in NodeJS
To create a stream from URLPath in Node JS it is quite easy (you just pass the path to a stream.) You can also switch from NodeStream to WebStream with ease (Stream.toWebStream())
Problem: Stream from user upload action
Can we get a stream right from user file uploads in the browser ? Can you give a simple example ?
The point of it would be to process the files as the user is uploading them, not to use the Blob stream.
Is it possible analyze user data as it is being uploaded i.e is it possible for it to be a stream ?
Is node http request actually fired after req.end or after http.request ?
Context: I am using node js http module and wanted to understand what happens between:
var req = http.request(options)
// Register handler events
req.end();
Can node open socket and start dns look up before req.end() Or req.end() is to just suggest that no more data needs to be sent ?
From documentation "With http.request() one must always call req.end() to signify the end of the request - even if there is no data being written to the request body." I am not sure what to make out of this ?
Is node http request actually fired after req.end or after http.request ?
http.request() returns an instance of the http.ClientRequest class and the ClientRequest instance is a writable stream. Therefore, the request will be fired after req.end()
Can node open socket and start dns look up before req.end() Or req.end() is to just suggest that no more data needs to be sent ?
The answer for your question is no, the socket will be created after you sent the request, as mentioned before when you use req.end().
From documentation "With http.request() one must always call req.end() to signify the end of the request - even if there is no data being written to the request body." I am not sure what to make out of this ?
I think you should try to understand a little more about node streams. When you invoke the request method from http, it returns a writable stream, that means the request is "available" to be written. When you do the .end() you are telling to that stream that no more writing is needed, in other words, you don't need to build the request anymore and so the request is sent.
I am using smartapi provided by angelbroking.
I want to make a stock ticker which can display realtime price of stocks like this one
https://www.tickertape.in/screener?utm_source=gads&utm_medium=search&utm_campaign=screener&gclid=Cj0KCQiA8ICOBhDmARIsAEGI6o1xfYgsbvDEB6c2OFTEYRp9e5UDnJxgCyBJJphdKTduZ_EOHCAchpoaAp-WEALw_wcB
I am able to connect to websocket using the sdk provided in documentation but I don't know how to display that data in my html page.
Please suggest if you know how to get the json data from nodejs console to html.
The nodejs code is
let { SmartAPI, WebSocket } = require("smartapi-javascript");
let web_socket = new WebSocket({
client_code: "P529774",
feed_token: "0973308957"
});
web_socket.connect()
.then(() => {
web_socket.runScript("nse_cm|2885", "cn") // SCRIPT: nse_cm|2885, mcx_fo|222900 TASK: mw|sfi|dp
web_socket.runScript("nse_cm|2885", "mw")
/*setTimeout(function() {
web_socket.close()
}, 60000)*/
})
web_socket.on('tick', receiveTick)
function receiveTick(data) {
console.log("receiveTick:::::", data)
}
The response I get is similar to this :
[{"ak":"ok","task":"mw","msg":"mw"}]
[{"lo":"1797.55","ts":"ACC-EQ","tp":null,"ltp":"1800.05","ltq":"27","bs":"16","tk":"22","ltt":"31\/08\/2017 11:32:01",
"lcl":null,"tsq":"76435","cng":"-11.15","bp":"1800.00","bq":"510","mc":"34012.01277(Crs)","isdc":"18.77872
(Crs)","name":"sf","tbq":"76497","oi":null,"yh":"1801.25","e":"nse_cm","sp":"1800.90","op":"1814.00","c": "1811.20",
"to":"145093696.35","ut":"31-Aug-2017 11:32:01","h":"1817.55","v":"80391","nc":"- 00.62","ap":"1804.85","yl":"1800.00","ucl":null,"toi":"16654000" }]
The github repo for smartapi nodejs
https://github.com/angelbroking-github/smartapi-javascript
The API Docs
https://smartapi.angelbroking.com/docs/Introduction
There are many ways, here's two:
Cache the last message + HTTP polling
This is not the most efficient solution, but perhaps the simplest. Each time your recieveTick() callback hits, you could save the response message in a global object / collection (cache it). Better yet, you could pre-process the message and therefore just cache whatever info you actually care about in that global collection and save bandwidth on the connection between your frontend HTML and backend.
Then, add an HTTP endpoint to your backend that serves up the last info relevant to a given ticker. You could use Express.js or some other simple HTTP server library. That way when your frontend calls
http://<backend_host>:<backend_port>/tickers/<ticker>
Your backend will read from the cached data and serve up the needed data.
Create your own websocket and forward the data
This is a better solution, specially if your data providers API has a quick (subsecond) refresh rate. Create your own websocket server that will make a websocket connection with your frontend. Then, when you get a message from the data providers websocket, simply processes it in whatever way you would like (to get it into the format your frontend wants) then forward it to the frontend by using your websocket server. This will also be done within the recieveTick() function.
There are many websocket tools for nodejs. For help with the websocket stuff check this out https://ably.com/blog/web-app-websockets-nodejs
Also just a quick note, in your question you said "...how to get the json data from nodejs console to html". This kind of suggests that you would like to write the data to the console, and then read it from the console to html. This isn't the way you should think about it. The console was one destination, and the html is another, both originating from the websocket callback.
I'm using puppeteer-stream to get a stream of a browser controlled by Node, running on a server. I am able to write this stream out to a file with no issues.
I wanted to stream this stream via WebRTC to a browser (basically to see what the browser is running in realtime). For webrtc, I'm trying to use simple-peer since it has ready bindings for Node as well as the browser-side.
However, when I try to pass this stream to simple-peer, I get the following error:
/Users/my_user/my_project/node_modules/simple-peer/index.js:286
stream.getTracks().forEach(track => {
^
TypeError: stream.getTracks is not a function
at Peer.addStream (/Users/my_user/my_project/node_modules/simple-peer/index.js:286:12)
This is because the Stream I have is a ReadableStream, but simple-peer (or most webrtc libs) expects a MediaStream.
How can I convert a realtime ReadableStream into a MediaStream that can be used with WebRTC? I found examples to convert MediaStreams into ReadableStreams such as here but not vice versa.
Am I missing something here?