Understanding basic websocket API with node.js and "ws"-package (API: bitfinex) - node.js

I'm trying to figure out basic websocket communication using node.js, the "ws"-package (which seems to be a very popular websocket package from npmjs.com) and the bitfinex.com (a cryptocurrency exchange) websocket API.
I want to read the public Ticker for a certain currency-pair, the docs are here: https://docs.bitfinex.com/v2/reference#ws-public-ticker
My result so far is working but is still much different from what I am supposed to get according to the docs.
I am working with this code snippet taken from the documentation linked above:
const ws = require('ws')
const w = new ws('wss://api.bitfinex.com/ws/2')
w.on('message', (msg) => {
console.log(msg)
})
let msg = JSON.stringify({
event: 'subscribe',
channel: 'ticker',
symbol: 'tBTCUSD'
})
w.on('open', () => {
w.send(msg)
})
Which works so far by outputting to the console the message from the subscribed channel:
[1,[14873,23.49464465,14874,61.09031263,1087,0.0789,14872,56895.20497085,15500,13891]]
But now, and here is the issue, in the docs the response looks different. How would I determine which number is what? I should be able to get all kinds of more information from the response, no?
The given example response looks like this:
// response - trading
{
event: "subscribed",
channel: "ticker",
chanId: CHANNEL_ID,
pair: "BTCUSD"
}
How does this relate to that array of numbers I get? How would I for example read the "pair:" field ("BTCUSD") or any of the other listed fields, like (BID, BID_PERIOD, VOLUME, HIGH, LOW etc.)? Am I missing something obvious?
I know this is a lot to ask at once but maybe someone knows one or two good examples or hints to enlighten me. Thanks in advance!
Kind regards,
s

The overall websocket scheme for this API is described in https://bitfinex.readme.io/v2/docs/ws-general If you haven't already read that page, now would be a good time to do it.
For your example program you should have seen info and subscribed events as the first two messages from the websocket. info should have been sent as soon as the websocket connection was established, and subscribed should have been sent in response to your subscribe request.
After that, you should see a ticker snapshot message followed by periodic ticker update messages for the channel that you subscribed to. These are the JSON arrays that you're seeing. The format of these messages for a public ticker channel is is described at https://bitfinex.readme.io/v2/reference#ws-public-ticker -- click the Snapshot and Update headings in the dark green 'details' bar to see the definitions. In this case snapshots and updates use the same format:
[ CHANNEL_ID,
[ FRR, BID, BID_PERIOD, BID_SIZE, ASK, ASK_PERIOD, ASK_SIZE,
DAILY_CHANGE, DAILY_CHANGE_PERC, LAST_PRICE, VOLUME, HIGH, LOW
]
]
with meanings as described in the 'Stream Fields' table at the above URL.
You can parse these messages as JSON strings and access the field values just as you would for any array.
It's a little strange that the API sends these as arrays rather than as objects with named attributes. I imagine they're looking to keep these messages compact because they make up the bulk of the traffic.

Related

Socket.io offline behaviour server-side

I have run into an unforeseen problem with my socket.io setup.
I use socket.io to live load data from my database (mongoDB, nodejs, react).
To accomplish this, I use mongoDB's changestream to detect changes and then push them to the front-end via socket.io.
Now this works perfectly as long as the user is connected. And right now, when the user reconnects, it just reloads all data. While this is fine for most users, there is a small group with very bad network connection and thus the front-end is reloading data all the time. Which causes the front-end to be unresponsive for some time.
So, I am looking for a way to only send events that occurred during the front-end being offline. While the front-end can do this quite easily: https://socket.io/docs/v4/client-offline-behavior/
It doesn't seem possible to do this at the server side. Since socket.io (server side) immediately forgets sockets that have disconnected and thus cant buffer events.
So, I was wondering if there is a good way do this? Or would this need a full "wrapper" around socket.io that caches disconnected sockets?
Any help or advice would be appreciated!
I find it is a really interesting and painful problem ! ^^'
If you can give more variables, it may help people to give you a better answer
For instance
How many data are stored in database, how much a typical user will receive, and how many events are triggered on a time frame ?
How long should an event take to be visible ? I mean, if users receive an event with a 10s,30s,... delay, is it harmfull for the service they provide.
How your data is structured ? is it a simple json array with the same field, custom field, dynamic json object, etc..
How your react app is structured, do you put heavy logic when your data is update, etc..
I think you should put more controls in your front end code and update only when new datas.
Some paths to explore
1. Put more controls in your front end
As you stated, for the users with bad connection, the react client seems to update his state too quickly, when they reload data after the websocket is connected, again and again. Ui may freeze in this case, yes.
For this, I think of two approaches :
Before updating the state, check if react current state is the same as the data you receive from websocket connection. If the reconnection is quick enough and no new data arrived, it should be the same. So in this case do not update react state.
If too many events are triggered and after each reconnection new data arrived, you can buffer the datas from the websocket and display it only once per time frame. What i mean by time frame, is you can use functions like setInterval or requestAnimationFrame to trigger react update. A pseudo react code to illustrate this.
function App() {
const [events, setEvents] = useState({ datas: [] });
const bufferedEvents = useRef([]);
useEffect(() => {
websocket.on("connected", (newEvents) => {
bufferedEvents.current = bufferedEvents.current.concat(newEvents);
})
websocket.on("data", (newEvent) => {
bufferedEvents.current = bufferedEvents.current.concat(newEvent);
})
// In the setInterval function you take all the events receive at the connection + new events. to update the react state. You clean the bufferedEvents at the same time.
const intervalId=setInterval(() => {
const events = bufferedEvents.current;
bufferedEvents.current = [];
//update if new datas
if (events.length > 0) {
setEvents((prevState) => { return { datas: prevState.datas.concat(events) } });
}
// console.log(events)
}, 1000) // trigger data update every second. You could replace this approach with a requestAnimationFrame. You can adapt the time refresh as you need.
//Do not forget to clear the interval when the component is unmount
return ()=>{
clearInterval(intervalId)
}
}, []);
return (
<div>
<span>Total events : {events.datas.length}</span>
<br />
{
events.datas.map(event => {
return <div>{event.data}</div>
})
}
</div>
)
}
You can look at this article for details on using requestAnimation frame.
I think that modifying the front end is needed in all case, but still alone, not really good on performance.
2. Fetch only new data in your back end
For this approach, it really depends how your data is structured in the database.
If the data have some timestamp in it, I can think of a naive but simple cookie with a timestamp in it.
When user connects the first time, this cookie is null.
When they fetch the data, on the websocket connection, they receive all the datas. When datas arrived, you update the cookie timestamp with the most recent date in the data.
Websocket is disconnected, you open a new websocket with the cookie timestamp on it. With this information you can query all the datas more recent than the timestamp on the cookie.
Like this, you don't have to download the entirity of data, but only fresh ones.
Other approaches may be more helpfull but without more informations on your datas and more precise requirements, it is hard to say.
If you have a lot of data, I will personally check some pagination mechanism and maybe combine some classic http request for fetching the data, and websocket, sse, or long polling for live events.
You can put a comment if needed and I will update my response !
Cheers

Get extended/full text tweets in Twitter API v2

The new Twitter v2 API was just released a couple of weeks ago, so this may just be an issue of the documentation not being done quite yet.
What I am trying to do is search recent tweets for "puppies" and return all that have some kind of media attached. However, when I run this search in Postman, not all of the returned tweets have attachments.media_keys. I noticed that the ones that do not have attachments.media_keys are tweets whose text ends in ellipses .... I understand that in the v1.1 API, this issue is solved by specifying tweet_mode=extended in the query params or tweet.fields=extended_tweet. However, these do not seem to work in the v2 API and I have not seen any documentation about getting the full text of tweets (and the associated attachments). Does anyone know how to do this in v2?
My Postman query url: "https://api.twitter.com/2/tweets/search/recent?query=has:media puppies&tweet.fields=attachments&expansions=attachments.media_keys&media.fields=duration_ms,height,media_key,preview_image_url,public_metrics,type,url,width"
In my app, I am using Node.js Axios to perform the query:
var axios = require('axios');
var config = {
method: 'get',
url: 'https://api.twitter.com/2/tweets/search/recent?query=has:media puppies&tweet.fields=attachments&expansions=attachments.media_keys&media.fields=duration_ms,height,media_key,preview_image_url,public_metrics,type,url,width',
headers: {
'Authorization': 'Bearer {{my berarer token}}',
}
};
axios(config)
.then(function (response) {
console.log(JSON.stringify(response.data));
})
.catch(function (error) {
console.log(error);
});
As of July, 2021, for sure this "problem" or strange behavior concerns retweets.
To get full text of a retweet while getting recent tweets for a user I did the following trick:
First I get recent tweets for a user following docs:
curl "https://api.twitter.com/2/users/2244994945/tweets?expansions=attachments.poll_ids,attachments.media_keys,author_id,entities.mentions.username,geo.place_id,in_reply_to_user_id,referenced_tweets.id,referenced_tweets.id.author_id&tweet.fields=attachments,author_id,context_annotations,conversation_id,created_at,entities,geo,id,in_reply_to_user_id,lang,possibly_sensitive,public_metrics,referenced_tweets,reply_settings,source,text,withheld&user.fields=created_at,description,entities,id,location,name,pinned_tweet_id,profile_image_url,protected,public_metrics,url,username,verified,withheld&place.fields=contained_within,country,country_code,full_name,geo,id,name,place_type&poll.fields=duration_minutes,end_datetime,id,options,voting_status&media.fields=duration_ms,height,media_key,preview_image_url,type,url,width,public_metrics,non_public_metrics,organic_metrics,promoted_metrics&max_results=5" -H "Authorization: Bearer $BEARER_TOKEN"
This is an all fields query (not all fields are necessary) but it is necessary to get ['includes']['tweets'] within the structure of the returned JSON data. This is the place where you have to look for the full text of a retweet - it is at: ['includes']['tweets'][0..n]['text] while all the recent tweets (and retweets) are found at ['data'][0..n]['text'].
Then you have to match the shortened retweets from the ['data'] with those from the ['includes']['tweets']. I do it using ['data'][n]['referenced_tweets'][0]['id'] which should match ['includes']['tweets'][m]['id]. where n and m are some indexes.
To be 100% safe you can check if ['data'][n]['referenced_tweets'][0]['id'] has a matching key/value pair: type: retweet (suggesting that this is really a retweet reference), but for me the 0 index works in all checked cases so not to complicate things more I left it this way for now :)
If that sounds complicated just dump the whole parsed JSON with all tweets and check the structure of the data.
Great question, thank you. We’re discussing this on the Twitter Developer forums as well.
In v2 of the API we have eliminated the notion of an “extended Tweet” since we assume that all new apps understand the concept of 280 characters, so the complete text is in the Tweet text field.
The difference you’re finding is in retweets or quoted Tweets where the embedded text is truncated. This is (perhaps surprisingly) the same as v1.1 and the former premium and enterprise APIs as well. We are investigating whether to modify this, and the implications in doing so.
I don’t for any means want to take traffic away from Stack, but you might find more ongoing updates and information on our developer forums. Thanks!

How do I manage groups/rooms with node WebSockets?

TL;DR below.
I am currently developing a React/Redux SPA that is driven by real-time data. I've decided to use ws, instead of socket.io since socket.io feels a bit high level for what I'm doing, I'd rather manage sockets myself.
In saying that, I'm struggling to find a way to manage the separation of updates/messages per view/route. Since I'm using client-side routing it's per express route won't really work...
Messages between the server and client via WebSockets are JSON with actions like GET_ITEMS then a response of GET_ITEMS_SUCCESS with an array of 'items' and for errors: ..._ERROR etc. This is all fine, since it's just 1 to 1 transaction. Though the problem arises when broadcasting (1 to all) to all relevant clients when the server receives an update.
So, I assume it best practice to limit these broadcasts to the clients that are viewing/want the data. So when viewing, for example, the Item page, there is no point in broadcasting updates to the User data since that is only used on the User page.
I haven't been able to find any common practices when dealing with this sort of situation, just a few small outdated/barely used wrappers for ws that just add a few basic functions to leave/join but don't offer much flexibility with implementation.
What I think MIGHT work is to have an object/array for each 'group'/'room', which stores the clients that are currently listening to updates from a given section. So a user would send an action to INIT_LISTEN (& ``) with a param of category, e.g. ITEM for updates and other actions related to items.
TL;DR
What my question really boils down to is: How do I store a reference to a single socket? (ws client object? ws client ID?) Then, can I store this in an object/array to iterate through like below.
const ClientRooms = {
Items: {
{
...ws
}
/* ...rest of the client */
}
}
or
const ClientRooms = {
Items: [ "xyz" ] /* Array of ws ids */
}
I have a "ping--pong" heartbeat function to keep clients active and prevent silent connection failures/disconnections. I can't find if ws.terminate() still fires the ws close event so I can iterate 'group'/'room' the object/array to find and remove instances of that client.

socket.io - io.sockets.adapter object?

I'm experimenting with socket.io and trying to build a multi-room chat app. The guide I'm following is out of date using pre 1.0.0 socket.io.
I'm trying to find a list of connected clients in a given room. Googling around shows that I have to use the adapter. However, I cannot find the documentation for it anywhere. I searched for it in the git-hub doc but search didn't return any information on adapter. https://github.com/socketio/socket.io-client/blob/master/docs/API.md
Can someone point me in the right direction and where I can read more about adapter and associated methods on it? Also if you can provide the most up to date documentation for socket.io I'd greatly appreciate it. Thank you.
You can get a map of all rooms in the top level namespace like this:
io.nsps['/'].adapter.rooms
You can list the sockets in one of those rooms like this:
function getSocketsInRoom(room, namespace = '/') {
let room = io.nsps[namespace].adapter.rooms[room];
return room.sockets;
}
As best I can tell, this kind of stuff is simply not documented. I've only discovered things like this by examining how things are stored in the debugger. That may or may not mean it's subject to change in the future - I really have no idea.
sockets:
{ '2v8OmIS4qTGX61-YAAAC': true, '3YnScxOgpmAGhZWsAAAG': true },
length: 2 }
it gives u this output. So it basically gives you the clientId and whether it is connected or not and total number of clients connected to a specific room. When the code below is executed (in server side written in node.js) gives u the above output.There is currently two clients connected to the same room named "hello".
var clientsInRoom = io.sockets.adapter.rooms[room];
but when u write this code below and console log it
var clientsInRoom = io.sockets.adapter.rooms
when single client is connected it will console log this
{ '9mVAHSDwcwnqsF4aAAAA':
Room { sockets: { '9mVAHSDwcwnqsF4aAAAA': true }, length: 1 } }
this crazy '9mVAHSDwcwnqsF4aAAAA' literals is client id which is unique for each client

Resolve MongoDB reference

I am currently building a chatting app with nodejs and mongoDB.
Basically I have two collections to maintain in the db.
user = {
_id: ObjectId("1234"),
account: "stan123"
}
thread = {
_user: ObjectId("1234"),
messages: [
{
body:"hi"
_user:ObjectId("1234")
},
{
body:"second msg"
_user:ObjectId("1234")
}
]
}
I am planning to pass the thread model with all resolved info (user) to the client side, so that I can construct my widget with it.
I searched for solutions for this.Some suggests to make extra calls from client side to get the data.
However, I am worried that when the amount of message grows, there will be considerable http calls that might hurt site speed.
I know some drivers can resolve DBRefs automatically and make the code clean.
However, according to
http://docs.mongodb.org/manual/applications/database-references/
I decided to just use id to maintain reference that make it's as simple as possible.
My plan is resolving all references on server side. Current approach is getting the length of message array first.
Then loop through the message array and make a second query to resolve user info separately.
In each query callback, do a messageToResolve++ and if(messageToResolve >= thread.messages.length)
If the condition meets, send the resolved model to client and end the response.
This is not a case I would consider embedded because it would be painful when you need to update user data.
(message is embedded because it exists only when thread exists)
I am not sure if it's a good way to do it.
Does anyone has a better solution?
Sorry if I didn't explain my problem and solution clear enough.
And thanks in advance.

Resources