Right now I have a Discord bot that is in approximately 575 servers, and on the website I made it lists the current server count of the bot. Right now, my method is to log the bot in every 5 minutes on the express app for the webpage and save the current server count to be served to the client. This causes memory usage spikes whenever I have to log in though, and using a whole discord.js application for one function seems inefficient.
I tried using the Discord API endpoint, but that was extremely laggy because there is only an endpoint for listing all the servers, not just the count. The endpoint also can only send info on 100 servers at a time, so I'd have to make a lot of different requests.
I'm hoping that there's a way to do this that would use less memory but still be fast. I tried looking into discord.js's source code to see if I could just isolate the functionality I needed, but I wasn't able to even find where in the code the data is requested from Discord. If anyone is able to figure how I could do this, it would be greatly appreciated.
You can try using free online database as a way to "communicate" data between your bot and your express app.
For example, you can use Cloud Firestore. Every 15 minutes (or whatever frequency you want) you can have your bot save server count information (and update time too if you want) into Cloud Firestore. Every time a client loads up your webpage, it'll retrieve the data from Cloud Firestore and be able to display server count and last updated time. (Alternatively you could have your express app retrieve that data every 15 minutes and cache it to send to the client)
You can use this method to share other data from your bot to your express app too.
The solution I ended up needing was a Discord websocket connection. That keeps everything updated live without having to deal with the memory and caching issues that come with discord.js. I've had a few other questions after this one on that topic, check those out if you want to see more on Discord websocket connections.
Related
I have a app which lets yoy keep your notes at a single place its realtime bw all the devices you are logged in I am using a nodejs wesocket it was working fine but a recently i found out someone was sending a huge amount of requests to my websocket server. He sent a large amount of data through websockets to my mongodb and the data was sent just for the purpose of taking the app down (useless crap data just had 'aaaaa')
What i want is prevent those clients from using the websockets who are making more than 10requests per minute.
As mentioned in the comments its better to go with services like CloudFlare, but for your specific use case (to implement directly on server) you should look at ways to rate limit the requests.
Here is an example of an library to rate limit web-sockets in node
https://www.npmjs.com/package/ws-rate-limit
I have created a bot for slack and deployed to Azure, I am making some API calls to another server from this BOT, for this bot I have a client requirement, My client wants to measure the time taken by request to reach to server and time taken by response to come back to Bot. (only time taken by request/response to reach to either side . I have been exploring Azure application insight from three days , but could not find any helpful service. I can not change my bot code , Is there any way in azure service by that I can monitor latency?
Here is simple diagram:-
Bot ----t1----> Server
<---t2-----
I don't want response time taken to process at Server side (No calculation time ) Just request/response travelling time.
Ganesh,
What you seem to be asking is how long it takes the API to process a request and return a response. Nothing to do with your bot.
My suggestion would be to create performance tests against the API directly using a tool such as Jmeter. This will give you average response times for say 10,000 requests and plot out on nice graphs etc.
If you need a help doing this, I could write it up for you in step by step instructions.
Let me know.
Thanks,
Tim
I'm working on a chat application and using socket.io / node for that. Basically I came up with the following strategies:
Send message from the client which is received by the socket server which then sends it to the receiving client. On the background I store that to the message on the DB to be retrieved later if the user wishes to seee his old conversations.
The pros of this approach is that the user gets the message almost instantly since we don't wait for the DB operation to complete, but the con is that if the DB operation failed and exactly that time the client refreshed its page to fetch the message, it won't get that.
Send message form the client to the server, the server then stores it on the DB first and then only sends it to the receiving client.
The pros is that we make sure that the message will be received to the client only if its stored in the DB. The con is that it will be no way close to real time since we'll be doing a DB operation in between slowing down the message passing.
Send message to the client which then is stored on a cache layer(redis for example) and then instantly broadcast it to the receiving client. On background keep fetching records from redis and updating DB. If the client refreshes the page, we first look into the DB and then the redis layer.
The pros is that we make the communication faster and also make sure messages are presented correctly on demand. The con is that this is quite complex as compared to above implementations, and I'm wondering if there's any easier way to achieve this?
My question is whats the way to go if you're building a serious chat application that ensures both - faster communication and data persistence. What are some strategies that app like facebook, whatsapp etc. use for the same? I'm not looking for exact example, but a few pointers will help.
Thanks.
I would go for the option number 2. I've been doing myself Chat apps in node and I found out that this is the best option. Saving in a database takes few milliseconds, which includes the 0.x milliseconds to write in the databse and the few milliseconds of latency in communication ( https://blog.serverdensity.com/mongodb-benchmarks/ ).
SO I would consider this approach realtime. The good thing with this is that if it fails, you can display a message to the sender that it failed, for whatever reason.
Facebook, whatsapp and many other big messaging apps are based on XMPP (jabber) which is a very, very big protocol for instant messaging and everything is very well documented on how to do things but it is based in XML, so you still have to parse everything etc but luckily there are very good libraries to handle with xmpp. So if you want to go the common way, using XMPP you can, but most of the big players in this area are not following anymore all the standards, since does not have all the features we are used to use today.
I would go with doing my own version, actually, I already something made (similar to slack), if you want I could give you access to it in private.
So to end this, number 2 is the way to go (for me). XMPP is cool but brings also a lot of complexity.
I'm new to NodeJS, however I have been able to make progress to a certain extent.
As it Is, I have created a nodeJS chat server with socket.io while the client app is a Java desktop application. Every time a user logs out the server emits a broadcast notifying all sockets of the log out event. My fear is that as time goes on, if hundreds of thousand of users are connected it might consume too much of the server resources to continually broadcast to every body each time a user logs out (which could be at a rate of more that one log out per second).
In view, of the above mentioned I'm considering refactoring the code such that each time a user logs out, a database will be queried for the friends of the logged out user so that ONLY the users friends will be notified every time he/she logs out rather that the whole world. But I'm not sure if it's a better way to go about it (considering the database queries per log out), so I am throwing the question to the public.
Which of the stated approaches is a better practice ?
I'm developing a web scraping website to find available delivery restaurants. The website searches on the most popular delivery portals and shows the result aggregated in a single page.
The site is hosted on Heroku with 4 dynos.
http://deliveria.net/#05409-002
When a user makes a request on the website, it makes around 30 HTTP requests to retrieve the result.
The problem is the performance, the requests aren't fast and each search can make 30 of them, locking the app while the search is being performed for a single user.
I tried to increase Heroku dynos:
heroku scale web=10
And I didn't feel a perceptible gain.
What is the best approach to scale this kind of application?
(I can't use cache, as the searches need to be in real time)
Current stack:
Heroku
Node.js
express
request module
EJS
Pusher
Redis
The important thing here is to have workers, because you must avoid blocking the event loop in your main app.
Try to delegate the 30 http requests between the available workers. Maybe Kue can help you with this aspect (you push new jobs to the queue and they get executed one by one by the workers). So for example if you have 10 dynos on Heroku, use 9 for workers (that make those 30 http searches).
From the user's point of view it's important to know that the application is reacting fast to his search (and doesn't give him the 'freeze' impression), so maybe you would like to update him as soon as you have preliminary results (for example 10 pages get searched out of 30). You could do that via WebSockets (Socket.IO) and even show a nice graphical progress bar or something similar.