im asking this question because i dont know what to look for right now and my googling wasnt great so far.
I am making nodejs,express,sql app that scrape website. It takes 30 to 120 seconds to scrape whole category. How to make that function run in the background without blocking website. Frontend template engine is eJS. If its not possible to do with eJS which framework,library should i use then? I imagine it work like that
User go to /scrape
Choose category and send to server by clicking button
Some container on /scrape gets greyed out with circle rotating or
other % or smth
User can freely leave /scrape and click around webiste or just stay
on /scrape waiting for result
When user cames back to /scrape the results are there or when he
stayed results shows up with or without reloading the page
Getting full respond to these questions will be very helpfull. But just keywords for me to look up also are very helpfull
Sorry for bad english
For your case here you could use redis, or just store the data you scrape on an data structrue that you like (in my opinion, because of the category, hashmaps (js objects) are the best here) directly in nodejs. The process would then look like this:
User goes to /scrape and selects a category
Backend checks if that category was already scraped (e.g. checks for the data in the hashmap with the category name as key)
If the data exists (just check if the key is defined), then send the data to the user, else (if the data isn't stored, e.g. key == undefined), send the user a message that the data is beign scraped and just run the scrape funtion in the backround. The scrape function than scrapes the data, and if it is done, it pushes the data with the category key to the hashmap. To avoid the same categorys beign scraped at the same time, you could add a "pending" property to the hashmap. So if the user accesses the /scrape route, you check in the hashmap if the category key exsists, if yes and pending is false, send data, if yes and pending is true, send wait alert, if the key doesn't exists, start the scrape function and send a wait alter.
Additionally, to make the whole thing "live", you could use socket.io (https://socket.io/) to implement websockets. You could then send the scraped data to the user without the user having to reload the page to check if the scrape process is done.
I made a little exmaple, that doesn't implement scraping, but should make the whole logic here a little bit easier to understand. I also added some explenation to the code in form of comments.
const express = require("express");
const app = express()
// the data hashmap
const data = {};
// scrape function
const scrape = async (id) => {
// set pending to true to prevent multiple scraped on same category
data[id] = { pending: true, data: {} }
// this would be your scrape functio, I used a promise here that
// resolves after 5 seconds with an random number just for
// simplicity
const a = await new Promise((res, rej) => {
setTimeout(() => { res(Math.floor(Math.random()*1000)) }, 5000)
})
// if the data was scraped, set pending to false and add the data
data[id].pending = false;
data[id].data = { id: a }
}
// "scrape" route
app.get("/:id", async (req, res) => {
const { id } = req.params; // if would represent category
// check if id (category) is not in hashmap, if not, then
// start the scrape process and send a wait alert
if (data[id] == undefined) {
scrape(id);
res.send("scraping...")
// if the data is already beign scraped, send a wait alert
// the pending property prvents that multiple people trigger
// the scrape of the same category
} else if (data[id].pending == true) {
res.send("still scraping...")
// lastly, if the data is defined, and is not pending, then
// you could just send it
} else {
res.send(data[id].data)
}
})
// to test this, go to the root with any id, could be string, number,
// whatever (e.g. /1337 or /helloworld), wait for 5 seceonds (or
// leave and come back after 5 seconds), refresh the page and you can
// see the random number. If you now go to an other route (e.g /test)
// and go back to the last one, you still can see the data, if you again
// wait for 5 seconds and then go back to /test, you can see the data.
// You can also open multiple tabs at the same time, which means the
// scraping is asynchronous, so you don't have to wait for the
// one category to be scraped to scrape the next
app.listen(5000)
Related
Okay so I have a Nodejs/Express app that has an endpoint which allows users to receive notifications by opening up a connection to said endpoint:
var practitionerStreams = [] // this is a list of all the streams opened by pract users to the
backend
async function notificationEventsHandler(req, res){
const headers ={
'Content-Type': 'text/event-stream',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache'
}
const practEmail = req.headers.practemail
console.log("PRACT EMAIL", practEmail)
const data = await ApptNotificationData.findAll({
where: {
practEmail: practEmail
}
})
//console.log("DATA", data)
res.writeHead(200, headers)
await res.write(`data:${JSON.stringify(data)}\n\n`)
// create a new stream
const newPractStream = {
practEmail: practEmail,
res
}
// add the new stream to list of streams
practitionerStreams.push(newPractStream)
req.on('close', () => {
console.log(`${practEmail} Connection closed`);
practitionerStreams = practitionerStreams.filter(pract => pract.practEmail !== pract.practEmail);
});
return res
}
async function sendApptNotification(newNotification, practEmail){
var updatedPractitionerStream = practitionerStreams.map((stream) =>
// iterate through the array and find the stream that contains the pract email we want
// then write the new notification to that stream
{
if (stream["practEmail"]==practEmail){
console.log("IF")
stream.res.write(`data:${JSON.stringify(newNotification)}\n\n`)
return stream
}
else {
// if it doesnt contain the stream we want leave it unchanged
console.log("ELSE")
return stream
}
}
)
practitionerStreams = updatedPractitionerStream
}
Basically when the user connects it takes the response object (that will stay open), will put that in an Object along with a unique email, and write to it in the future in sendApptNotification
But obviously this is slow for a full app, how exactly do I replace this with Redis? Would I still have a Response object that I write to? Or would that be replaced with a redis stream that I can subscribe to on the frontend? I also assume I would store all my streams on redis as well
edit: from what examples I've seen people are writing events from redis to the response object
Thank you in advance
If you want to use Redis Stream as notification system, you can follow this official guide:
https://redis.com/blog/how-to-create-notification-services-with-redis-websockets-and-vue-js/ .
To get this data as real time you need to create a websocket connection. I prefer to send to you an official guide instead of create it for you it's because the quality of this guide. It's perfect to anyone understand how to create it, but you need to adapt for your reality, of course.
However like I've said to you in the comments, I just believe that it's more simple to do requests in your api endpoint like /api/v1/notifications with setInterval in your frontend code and do requests each 5 seconds for example. If you prefer to use a notification system as real time I think you need to understand why do you need it, so in the future you can change your system if you think you need it. Basically it's a trade-off you must to do!
For my example imagine two tables in a relational database, one as Users and the second as Notifications.
The tables of this example:
UsersTable
id name
1 andrew
2 mark
NotificationTable
id message userId isRead
1 message1 1 true
2 message2 1 false
3 message3 2 false
The endpoint of this example will return all cached notifications that isn't read by the user. If the cache doesn't exists, it will return the data from the database, put it on the cache and return to the user. In the next call from API, you'll get the result from cache. There some points to complete in this example, for example the query on the database to get the notifications, the configuration of time expiration from cache and the another important thing is: if you want to update all the time the notifications in the cache, you need to create a middleware and trigger it in the parts of your code that needs to notify the notifications user. In this case you'll only update the database and cache. But I think you can complete these points.
const redis = require('redis');
const redisClient = redis.createClient();
app.get('/notifications', async (request, response) => {
const userId = request.user.id;
const cacheResult = await redisClient.get(`user:${userId}:notifications`)
if (cacheResult) return response.send(cacheResult);
const notifications = getUserNotificationsFromDatabase(userId);
redisClient.set(`user:${userId}:notifications`, notifications);
response.send(notifications);
})
Besides that there's another way, you can simple use only the redis or only the database to manage this notification. Your relational database with the correct index will send to your the results as faster as you expect. You'll only think about how much notifications you'll have been.
I'm trying to create a chat icon in a navigation bar with a message count (ala FB, Twitter, etc). I'm using ReactJS. The menu component is separate from the chat components. I'm kind of getting what I want, but not completely. I'm hoping someone can show me the correct way to listen for real time changes to total_unread_count in react.
Right now, if user a sends a new message to user b and user b is not on the chat screen, they will get badge with a message count. I can navigate all around and it'll keep that count. If I click on the chat view to navigate to the chat and click on the channel, then go back to any other screen, the notification count does not change. But, if I refresh the page manually, it'll clear the badge.
So, my expectation is that when a user has viewed the channel, that the state of the chat icon's badge count be updated to reflect that I've seen the message. Having to manually refresh the screen means I'm doing something wrong.
Here's my code so far:
useEffect(() => {
async function init() {
// set state initially because it appers that client.on events only happen a new event.
if (user != null) {
const response: any = await chatClient.connectUser({ id: user.uid }, token)
setUnreadCount(response.me.total_unread_count)
}
// update state when a new event occures.
chatClient.on((event) => {
if (event.unread_channels !== undefined) {
console.log("COUNT: ", event.total_unread_count)
setUnreadCount(event.total_unread_count)
}
})
}
init()
}, [])
I hope I'm communicating this well enough to make sense. I appreciate any help I can get.
I am struggling to make my own matchmaking in a multiplayer game of two players. I have a button that says "play",and i want if the user presses it to change in database his schema property "inQueue to true" .
So after that i want to search the database to check who also is online and in queue(and meets requirement range for elo eg. player1:1200 ELO player2:1288 ) . But i dont want to send multiple requests until it finds a match.
iI was wondering if is there a way to send the request one time and when mongodb finds a match then return the response.
( i am using the mern stack with the help of sockets too)*
The following middleware attempts to find a match on the database. If that is unsuccessful, another attempt is scheduled for a second later. A JSON response is sent only after a successful attempt, but spaces are sent while waiting in the hope that this prevents a browser timeout.
.get("/path", function(req, res) {
res.type("json");
async function attempt() {
var match = await User.find(...);
if (match)
res.end(JSON.stringify({match: match.username}));
else {
res.write(" ");
setTimeout(attempt, 1000);
}
}
attempt();
});
If the backend times out requests after, say, 30 seconds, this cannot be used. In this case, consider repeating the requests in the frontend instead:
async function attempt() {
var match = await axios.get("/path"); // returns 0 or more matches
if (match.length > 0)
// Fill results of match into the HTML page
else
setTimeout(attempt, 1000);
}
attempt();
But I think this is only a poor substitute for the real solution based on web sockets as suggested by balexandre.
I am building a web app and one of the functions is adding songs to a song queue. Basically there are several different rooms and each of the rooms properties(including the song queue) are stored on the server in a array. I use app.post to send a new song url to the server which then uses that url and adds it to the appropriate room.
Right now, when one person adds a song to the queue only their page is updated. Other people who are in the same room do not receive those updates unless they add their own song and the server returns them the new song queue(the server returns an updated song queue everytime someone adds one).
I believe there is a way to do the real time updates using socket.io but I am pressed for time and am wondering if there is a way to achieve the desired outcome using pure express node.js so I won't need to change everything.
Backend:
let rooms = [];
//other stuff that makes these things functional
/***ADD TO QUEUE***/
app.post('/api/addToQueue', async (req,res) => {
console.log('Add To Queue Request Recieved');
const activeRoom = req.body.activeRoom;
const enteredURL = req.body.enteredURL;
let newQueue;
for(let i = 0; i < rooms.length; i++){
//locates the room to update
if(rooms[i].roomCode === activeRoom){
rooms[i].songQueue.push(enteredURL);
console.log('Updated Queue');
newQueue = rooms[i].songQueue;
break;
}
}
res.json(newQueue);
console.log('Sent New Queue');
console.log(rooms);
});
FrontEnd
//inside the add to queue button component which contains the input bar as well
addToQueue = async () => {
const activeRoom = this.props.getRoom();
await fetch('/api/addToQueue', {
method: "post",
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({activeRoom: activeRoom, enteredURL: this.state.enteredURL})
})
.then(res => res.json())
.then(newQueue => this.props.onAddToQueue(newQueue));
//onAddToQueue changes the state of another component which then renders in the added song as a
paragraph element.
}
My file pathing has my frontend inside the backend.
Would love some help :) TIA
You can add setInterval() built-in function that running every few seconds to check a value update or not.
so that when one person adds a song to the queue the other people who are in the same room can receive those updates.
There are 3 options for real-time web apps
sockets: if you need two-way communication
server-sent events: only the server can send info to frontend
polling: Make a request every certain time to the server
The first two need server changes (the second is easier), the third is just frontend change. All of them have their pros and cons, but I am sure you can find them and make a decision.
I am quite new to web development itself. And I have a problem which I have no idea how to solve or even what to google. The web app is developed with Node JS and MySQL.
So let me explain the situation.
On my website you can create and delete posts.
On "/create", the user enters contents for the post.
Then the user clicks the "submit" button, which is routered to "/create_process"(where the data is actually saved in database)
Occasionally there is some delay in loading "/create_process". So the user keeps refreshing while loading. -> Here is the problem. Everytime the user refreshes at this stage same inputs are sent again and again. The result is multiple posts with exactly same contents.
I am sure that there must be a way to block such trivial inputs.
You can have a throttle function on whatever the use is clicking. Example:
const throttle = (func, limit) => {
let inThrottle
return function() {
const args = arguments
const context = this
if (!inThrottle) {
func.apply(context, args)
inThrottle = true
setTimeout(() => inThrottle = false, limit)
}
}
}
from medium article