I'm working on a project that uses the binance api to create an interface to make day trading cryptos easier.
The call to their api looks like this:
binance.websockets.candlesticks(['BNBBTC'], "1m", function(candlesticks) {
let { e:eventType, E:eventTime, s:symbol, k:ticks } = candlesticks;
let { o:open, h:high, l:low, c:close, v:volume, n:trades, i:interval, x:isFinal, q:quoteVolume, V:buyVolume, Q:quoteBuyVolume } = ticks;
console.log(symbol+" "+interval+" candlestick update");
console.log("open: "+open);
console.log("high: "+high);
console.log("low: "+low);
console.log("close: "+close);
console.log("volume: "+volume);
console.log("isFinal: "+isFinal);
});
It seems to be returning data at a fixed interval, so I'm skeptical as to whether it's actually real time, but regardless, I'm wondering how to send this data to the front end as it comes in.
Currently, I'm doing this with the static data:
router.get('/interface', function(req,res) {
binance.candlesticks("BNBBTC", "5m", function(ticks, symbol) {
console.log("candlesticks()", ticks);
let last_tick = ticks[ticks.length - 1];
let [time, open, high, low, close, volume, closeTime, assetVolume, trades, buyBaseVolume, buyAssetVolume, ignored] = last_tick;
console.log(symbol+" last close: "+close);
res.render('interface', {ticks:ticks});
});
});
I've messed with socket.io in the past, but am unsure how to utilize it. Any help would be much appreciated! And please hmu if you're interested in cryptos. We are putting together a group in discord to share our research, and trading strategies.
To initiate the data sending process from the backend, (instead of frontend requesting data), you should use websockets (socketIO as you have mentioned).
To do that, first, you should start a socketio server in your express app, by wrapping the http/https server or express app.
Then, from the frontend, you should initiate a socketio-client.
Next, your frontend client should establish a connection with the server using the connect method of the socketio-client. It will fire an event in the server, with the socket connection.
Finally, the server can use that socket connection, to send any amount of data to the client. (You might need to save the connection for latter use).
i'm trying to do basically the same thing, what discord group you talking about?
Related
I would like to implement a system that allows users to add each other as friends and share data between them. I have gotten the authentication done and currently researching ways to do this real time. This project of mine is purely a learning experience so I am looking for many ways to perform this task to grow my knowledge.
I have experience using Websockets on a previous project and it was easy to use. Websockets seems like the best solution to my problem as it allows the user to send and receive invites through the open socket. However I have also learnt that the downside would be a long open socket connection that might be potentially performance taxing(?) Since I'm only sending/receiving information only when an invite is sent/received, websockets might be overutilized for a simple function.
At the same time I would like to learn about new technologies and I found out about Server Sent Events that would be less performance heavy(?) Using SSE would be much efficient as it only sends HTTP requests to the clients/server whenever the user send the invite.
Please correct me if I'm wrong for what I typed out above as this is what I gathered through my reading online. So now I'm having a hard time understanding whether SSE is better than websocket for my project. If there are other technologies please do let me know too! Thank you
how you doing ?
The best advise would be always to use websocket in this context, cuz your project can grow and need some feature that would be better using websocket
But you got another options, one of the is Firebase, Yes, FIREBASE!
You can do a nice reactive application with firebase, becouse the its observers update data in realtime, just like the websockets do.
But here go some cons and pros.
Websocket: Can make your project escalable, its more complete, you can use it in any context, BUT: is hard to implement and takes more time to be learned and understood.
Firebase, Easy and fast to implement, you can do a chat in 20 minuts, and surelly would help you with your problem, There is Firestore and Reatime database.. even the firestore updates in realtime.. BUT: Firebase costs in a big project can be expensive, i dont think is a good option for a big project.
Thats it.. the better options to do a real time data application to me.
A little bit more about. Firebase vs Websocket
https://ably.com/compare/firebase-vs-socketio
to send a friend invitation, you just send an API request. WebSocket is used for real time communication. From react.js, get the email and send the email to the server
export const sendFriendInvitation = async (data) => {
try {
return axios.post("/friend-invitation", data);
} catch (exception) {
console.error(error)
}
};
On node.js side, write a controller to control this request:
const invitationRequest = async (req, res) => {
// get the email
const { targetMail } = req.body;
// write code to handle that same person is not sending req to himself
// get the details of user who sent the email
const targetUser = await User.findOne({
mail: targetMail.toLowerCase(),
});
if (!targetUser) {
return res
.status(404)
.send("send error message");
}
// you should have Invitations model
// check if invitation already sent.
// check if the user we would like to invite is our friend
// now create a new invitation
// if invitation has been successfully created, update the user's friend
return res.status(201).send("Invitation has been sent");
};
I am using smartapi provided by angelbroking.
I want to make a stock ticker which can display realtime price of stocks like this one
https://www.tickertape.in/screener?utm_source=gads&utm_medium=search&utm_campaign=screener&gclid=Cj0KCQiA8ICOBhDmARIsAEGI6o1xfYgsbvDEB6c2OFTEYRp9e5UDnJxgCyBJJphdKTduZ_EOHCAchpoaAp-WEALw_wcB
I am able to connect to websocket using the sdk provided in documentation but I don't know how to display that data in my html page.
Please suggest if you know how to get the json data from nodejs console to html.
The nodejs code is
let { SmartAPI, WebSocket } = require("smartapi-javascript");
let web_socket = new WebSocket({
client_code: "P529774",
feed_token: "0973308957"
});
web_socket.connect()
.then(() => {
web_socket.runScript("nse_cm|2885", "cn") // SCRIPT: nse_cm|2885, mcx_fo|222900 TASK: mw|sfi|dp
web_socket.runScript("nse_cm|2885", "mw")
/*setTimeout(function() {
web_socket.close()
}, 60000)*/
})
web_socket.on('tick', receiveTick)
function receiveTick(data) {
console.log("receiveTick:::::", data)
}
The response I get is similar to this :
[{"ak":"ok","task":"mw","msg":"mw"}]
[{"lo":"1797.55","ts":"ACC-EQ","tp":null,"ltp":"1800.05","ltq":"27","bs":"16","tk":"22","ltt":"31\/08\/2017 11:32:01",
"lcl":null,"tsq":"76435","cng":"-11.15","bp":"1800.00","bq":"510","mc":"34012.01277(Crs)","isdc":"18.77872
(Crs)","name":"sf","tbq":"76497","oi":null,"yh":"1801.25","e":"nse_cm","sp":"1800.90","op":"1814.00","c": "1811.20",
"to":"145093696.35","ut":"31-Aug-2017 11:32:01","h":"1817.55","v":"80391","nc":"- 00.62","ap":"1804.85","yl":"1800.00","ucl":null,"toi":"16654000" }]
The github repo for smartapi nodejs
https://github.com/angelbroking-github/smartapi-javascript
The API Docs
https://smartapi.angelbroking.com/docs/Introduction
There are many ways, here's two:
Cache the last message + HTTP polling
This is not the most efficient solution, but perhaps the simplest. Each time your recieveTick() callback hits, you could save the response message in a global object / collection (cache it). Better yet, you could pre-process the message and therefore just cache whatever info you actually care about in that global collection and save bandwidth on the connection between your frontend HTML and backend.
Then, add an HTTP endpoint to your backend that serves up the last info relevant to a given ticker. You could use Express.js or some other simple HTTP server library. That way when your frontend calls
http://<backend_host>:<backend_port>/tickers/<ticker>
Your backend will read from the cached data and serve up the needed data.
Create your own websocket and forward the data
This is a better solution, specially if your data providers API has a quick (subsecond) refresh rate. Create your own websocket server that will make a websocket connection with your frontend. Then, when you get a message from the data providers websocket, simply processes it in whatever way you would like (to get it into the format your frontend wants) then forward it to the frontend by using your websocket server. This will also be done within the recieveTick() function.
There are many websocket tools for nodejs. For help with the websocket stuff check this out https://ably.com/blog/web-app-websockets-nodejs
Also just a quick note, in your question you said "...how to get the json data from nodejs console to html". This kind of suggests that you would like to write the data to the console, and then read it from the console to html. This isn't the way you should think about it. The console was one destination, and the html is another, both originating from the websocket callback.
I am creating a browser game with node.js (backend api) and angular (frontend). My goal is to implement an in-game chat to allow communication between players on the same map. The chat is not an essential part of the game, so messages don't need to be instant (few seconds of latency should be ok). It is just a cool feature to talk some times together.
A good solution should be to implement socket.io to have real-time communication. But as chat is not an essential component and is the only thing which will require websockets, i'm wondering if there is not an alternative to avoid server overload with sockets handling.
I thinked about polling every 2 or 3 seconds my REST API to ask for new messages, but it may overload server the same way... What are your recommandations?
Thank you for your advices
There's a pretty cool package called signalhub. It has a nodejs server component and stuff you can use in your users' browsers. It uses a not-so-well-known application of the http (https) protocol called EventSource. EventSource basically opens persistent http (https) connections to a web server.
It's a reliable and lightweight setup. (The README talks about WebRTC signalling, but it's useful for much more than that.)
On the server side, a simple but effective server setup might look like this:
module.exports = function makeHubServer (port) {
const signalhubServer = require('signalhub/server')
const hub = signalhubServer({ maxBroadcasts: 0 })
hub.on('subscribe', function (channel) {
/* you can, but don't have to, keep track of subscriptions here. */
})
hub.on('publish', function (channel, message) {
/* you can, but don't have to, keep track of messages here. */
})
hub.listen(port, null, function () {
const addr = hub.address()
})
return hub
}
In a browser you can do this sort of thing. It user GET to open a persistent EventSource to receive messages. And, when it's time to send a message, it POSTs it.
And, Chromium's devtools Network tab knows all about EventSource connections.
const hub = signalhub('appname', [hubUrl])
...
/* to receive */
hub.subscribe('a-channel-name')
.on('data', message => {
/* Here's a payload */
console.log (message)
})
...
/* to send */
hub.broadcast('a-channel-name', message)
As far as I know Web Sockets allows bi-directional communication. and web sockets (for example: Socket.io) connections are always open. so, whenever new data has arrived data should be automatically pushed to the view via socket.
but in below code I am using set_interval to make a http.get call. and set_interval is called once every 1 second.
now, doing these does not give a real-time feel that is, the new data is pulled once every 1 second. which is statically defined.
in-short, I want to automate what set_interval does in below code. I don't want a static fetch interval value. This is because at-times stock price could change within 100ms and at times it would change once in few seconds.
Now, if I set interval to 1 sec, that is make a call every 1 second. the real feel of high fluctuation in market move would not be seen.
I am not sure how usually developers fetch data in IOT applications. for example car is monitored in real-time and let's say speed of the car is fetched in real time and graphed on a web or mobile application.
How do I achieve something similar like that in Stock Ticker? I want to simply plugin the application to an API and when new data arrives instantly push it to all the viewers (subscribers) in real-time.
Code below
////
// CONFIGURATION SETTINGS
////
var FETCH_INTERVAL = 1000;
var PRETTY_PRINT_JSON = true;
////
// START
////
var express = require('express');
var http = require('http');
var https = require('https');
var io = require('socket.io');
var cors = require('cors');
function getQuote(socket, ticker) {
https.get({
port: 443,
method: 'GET',
hostname: 'www.google.com',
path: '/finance/info?client=ig&q=' + ticker,
timeout: 1000
}, function(response) {
response.setEncoding('utf8');
var data = '';
response.on('data', function(chunk) {
data += chunk;
});
response.on('end', function() {
if(data.length > 0) {
var dataObj;
try {
dataObj = JSON.parse(data.substring(3));
} catch(e) {
return false;
}
socket.emit(ticker, dataObj[0].l_cur);
}
});
});
}
I am making a call to method getQuote depending on FETCH_INTERVAL set above
function trackTicker(socket, ticker) {
// run the first time immediately
getQuote(socket, ticker);
// every N seconds
var timer = setInterval(function() {
getQuote(socket, ticker);
}, FETCH_INTERVAL);
socket.on('disconnect', function () {
clearInterval(timer);
});
}
var app = express();
app.use(cors());
var server = http.createServer(app);
var io = io.listen(server);
io.set('origins', '*:*');
app.get('/', function(req, res) {
res.sendfile(__dirname + '/index.html');
});
io.sockets.on('connection', function(socket) {
socket.on('ticker', function(ticker) {
trackTicker(socket, ticker);
});
});
server.listen(process.env.PORT || 4000);
Edits - Update
Okay, so I would need real-time feed. (this bit is sorted)
As far as I know, Real-time feeds are quite expensive and buying 10,000+ end points for each online client is quite expensive.
1) How do I make use of real-time feed to serve 1000s of end users? Can I use web sockets, Redis, publish/subscribe, broadcasting or some technology that copies real-time feed to tonnes of users? I want a efficient solution because I want to keep the expense of real-time data feed as low as possible.
How do I tackle that issue?
2) Yes, I understand polling needs to be done on server side and not on a client-side (to avoid doing polling for each client). but then what tech do I need to use? websockets, redis, pub/sub etc..
I have API URL and a token to access the API.
3) I am not just in need to fetch the data and push it to end users. But I would need to do some computation on the fetched data, will need to pull data from Redis or database as well and do calculations on it then push it to the view.
for example:
1) data I get in real-time market feed {"a":10, "b":20}
2) get data from DB or Redis {"x":2, "y":4}
3) do computation : z = a * x + b * y
4) finally push value of z in the view.
How do I do all these in real-time at the same-time push it to multiple clients?
Can you share a roadmap with me? I got the first piece of the puzzle getting real-time datafeed.
1) How do I make use of real-time feed to serve 1000s of end users? Can I use web sockets, Redis, publish/subscribe, broadcasting or some technology that copies real-time feed to tonnes of users? I want a efficient solution because I want to keep the expense of real-time data feed as low as possible.
How do I tackle that issue?
To "push" data to browser clients, you would want to use a webSocket or socket.io (built on top of webSockets). Then, anytime your server knows there's an update, it can immediately send that update to any currently connected client that is interested in that info. The basic idea is that the client connects to your server as soon as the web page is loaded and keeps that connection open for as long as the web page(s) are open.
2) Yes, I understand polling needs to be done on server side and not on a client-side (to avoid doing polling for each client). but then what tech do I need to use? websockets, redis, pub/sub etc..
It isn't clear to me what exactly you're asking about here. You will get updated prices using whatever the most efficient technology is that is offered by your provider. If all they provide is http calls, then you have to poll regularly using http requests. If they provide a webSocket interface to get updates, then that would be preferable.
There are lots of choices for how to keep track of which clients are interested in which pieces of information and how to distribute the updates. For a single server, you could easily build your own with just a Map of stock prices where the stock symbol is the key and an array of client identifiers is the value in the Map. Then, any time you get an update for a given stock, you just fetch the list of client IDs that are interested in that stock and send the update to them (over their webSocket/socket.io connection).
This is also a natural pub/sub type of application so anyone of the backends that support pub/sub would work just fine too. You could even use an EventEmitter where you .emit(stock, price) and each separate connection adds a listener for the stock symbols they are interested in.
For multiple servers at scale, you'd probably want to use some external process that manages the pub/sub process. Redis is a candidate for that.
3) I am not just in need to fetch the data and push it to end users. But I would need to do some computation on the fetched data, will need to pull data from Redis or database as well and do calculations on it then push it to the view.
I don't really see what question there is here. Pick your favorite database to store the info you need to fetch so you can get it upon demand.
How do I do all these in real-time at the same-time push it to multiple clients? Can you share a roadmap with me? I got the first piece of the puzzle getting real-time datafeed.
Real-time data feed.
Database to store your meta data used for calculations.
Some pub/sub system, either home built or from a pre-built package.
Then, follow this sequence of events.
Client signs in, connects a webSocket or socket.io connection.
Server accepts client connection and assigns a clientID and keeps track of the connection in some sort of Map between clientID and webSocket/socket.io connection. FYI, socket.io does this automatically for you.
Client tells server which items it wants to monitor (probably message sent over webSocket/socket.io connection.
Server registers that interest in pub/sub system (essentially subscribing the client to each item it wants to monitor.
Other clients do the same thing.
Each time client requests data on a specific item, the server makes sure that it is getting updates for that item (however the server gets its updates).
Server gets new info for some item that one or more clients is interested in.
New data is sent to pub/sub system and pub/sub system broadcasts that information to those clients that were interested in info on that particular item. The details of how that works depend upon what pub/sub system you choose and how it notifies subscribers of a change, but eventually a message is sent over webSocket/socket.io for the item that has changed.
When a client disconnects, their pub/sub subscriptions are "unsubscribed".
I am working with a nodejs express server which uses socket.io to communicate an iOS client, and am having a little trouble trying to test how many clients can connect and exchange data at any one time.
My goal is to be able to run a script which connects to socket.io with thousands of different sessions, as well as send and receive data to understand our system's scale. Currently we are using a single dyno on Heroku but will likely be considering other options on AWS soon.
I have found code which should do what I am trying to do for earlier versions of socket.io, such as this, but have had issues since it seems v1.x has a very different handshake protocol. I tried out using the socket.io-client package, but trying to connect multiple times only simulates use of one session, I need to simulate many in independent users.
I have been picking apart the socket.io-client code, but have only gotten so far as creating a connection - I am stuck on the sending data part. If anyone has any knowledge or could point to some written resources on how data is sent between a client and a socket.io server, it would help me out a lot.
Here's what I have so far:
var needle = require('needle'),
WebSocket = require('ws'),
BASE_URL = 'url-to-socket-host:5002';
var connectionNo = 0;
needle.get('http://' + BASE_URL + '/socket.io/?EIO=3&transport=polling&t=1416506501335-0', function (err, resp) {
// parse the sid
var resp = JSON.parse(resp.body.toString().substring(5, resp.body.toString().length));
// use the sid to connect using websockets
var url = 'ws://' + BASE_URL + '/socket.io/?EIO=3&transport=websocket&sid=' + resp.sid;
console.log(connectionNo + ' with sid: ' + resp.sid);
var socket = new WebSocket(url, void(0), {
agent: false
});
socket.on('open', function () {
console.log('Websocket connected: ' + connectionNo);
// I don't understand how to send data to the server here,
// from looking at the source code it should use some kind
// of binary encoding, any ideas?
socket.on('message', function (msg) {
console.log(msg);
});
});
});
I will continue deconstructing the socket.io-client code but if anyone has any clues or recourses that may help, let me know. Thanks.
I ended up setting for using the socket.io-client npm package which has the ability to connect to a new session on every connection. I found an example benchmark in this issue.
There is not so much need for me to manually connect to socket.io using pure websockets and HTTP, but thanks to Yannik for pointing out the parser in use. The spec of the inner workings of v1.x can be found here.
Thanks!
The problem my reside in the fact that you are not using socket.io in your client code. You have imported ('ws') which is another module whose docs are here: https://www.npmjs.org/package/ws.
You probably want to ws.send('something');. When you receive a message in ws, it also comes with an object with a property indicating whether it is binary data or not. If it is, you will need to concatenate the chunks incrementally. There is a canonical way to do this which you can find via google. But it looks a little like this:
var message;
socketConnection.on('data', function(chunk){ message += chunk});