I am building a hosting server with Node and MongoDB. Process of working look something like this:
User open page with form which contains 4 inputs:
sender email
receiver email
message from sender to receiver
files (multiple)
User fills all inputs properly and sends POST request on server.
Server handles form with multer and saves files, then in callback the object with fields data where is stored info received from form is prepared and sent to database on MongoLab.
In callback of saving doc in database, server sends mails to sender and receiver with generated link from where they can download uploaded files.
Now I would like to implement additional input to form, where user can set date when his files should be deleted from the server.
So there are two things to do: delete files and delete doc in database on time set by the user.
Do you have some ideas how to implement such thing?
To delete a file, you can simply use fs.unlink()
const fs = require('fs');
const deleteFile = (file) => {
fs.unlink("path/to/file/folder/"+file, (err) => {
if (err) throw err;
}
}
You want to create a setTimeout(), but you need to find how much time is remaining before the date provided by the user, you should do something like this:
const time_remaining = (date_provided) => new Date(date_provided) - new Date();
Then just use setTimeout():
let timeOuts = []; // We create an array of timeouts in case we want to cancel one later
// I assume you use express and body-parser
app.post('/upload', (req, res) => {
const timer = setTimeout( () => deleteFile(req.body.file), time_remaining (req.body.date));
timeOuts.push(timer);
}
Related
Okay so I have a Nodejs/Express app that has an endpoint which allows users to receive notifications by opening up a connection to said endpoint:
var practitionerStreams = [] // this is a list of all the streams opened by pract users to the
backend
async function notificationEventsHandler(req, res){
const headers ={
'Content-Type': 'text/event-stream',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache'
}
const practEmail = req.headers.practemail
console.log("PRACT EMAIL", practEmail)
const data = await ApptNotificationData.findAll({
where: {
practEmail: practEmail
}
})
//console.log("DATA", data)
res.writeHead(200, headers)
await res.write(`data:${JSON.stringify(data)}\n\n`)
// create a new stream
const newPractStream = {
practEmail: practEmail,
res
}
// add the new stream to list of streams
practitionerStreams.push(newPractStream)
req.on('close', () => {
console.log(`${practEmail} Connection closed`);
practitionerStreams = practitionerStreams.filter(pract => pract.practEmail !== pract.practEmail);
});
return res
}
async function sendApptNotification(newNotification, practEmail){
var updatedPractitionerStream = practitionerStreams.map((stream) =>
// iterate through the array and find the stream that contains the pract email we want
// then write the new notification to that stream
{
if (stream["practEmail"]==practEmail){
console.log("IF")
stream.res.write(`data:${JSON.stringify(newNotification)}\n\n`)
return stream
}
else {
// if it doesnt contain the stream we want leave it unchanged
console.log("ELSE")
return stream
}
}
)
practitionerStreams = updatedPractitionerStream
}
Basically when the user connects it takes the response object (that will stay open), will put that in an Object along with a unique email, and write to it in the future in sendApptNotification
But obviously this is slow for a full app, how exactly do I replace this with Redis? Would I still have a Response object that I write to? Or would that be replaced with a redis stream that I can subscribe to on the frontend? I also assume I would store all my streams on redis as well
edit: from what examples I've seen people are writing events from redis to the response object
Thank you in advance
If you want to use Redis Stream as notification system, you can follow this official guide:
https://redis.com/blog/how-to-create-notification-services-with-redis-websockets-and-vue-js/ .
To get this data as real time you need to create a websocket connection. I prefer to send to you an official guide instead of create it for you it's because the quality of this guide. It's perfect to anyone understand how to create it, but you need to adapt for your reality, of course.
However like I've said to you in the comments, I just believe that it's more simple to do requests in your api endpoint like /api/v1/notifications with setInterval in your frontend code and do requests each 5 seconds for example. If you prefer to use a notification system as real time I think you need to understand why do you need it, so in the future you can change your system if you think you need it. Basically it's a trade-off you must to do!
For my example imagine two tables in a relational database, one as Users and the second as Notifications.
The tables of this example:
UsersTable
id name
1 andrew
2 mark
NotificationTable
id message userId isRead
1 message1 1 true
2 message2 1 false
3 message3 2 false
The endpoint of this example will return all cached notifications that isn't read by the user. If the cache doesn't exists, it will return the data from the database, put it on the cache and return to the user. In the next call from API, you'll get the result from cache. There some points to complete in this example, for example the query on the database to get the notifications, the configuration of time expiration from cache and the another important thing is: if you want to update all the time the notifications in the cache, you need to create a middleware and trigger it in the parts of your code that needs to notify the notifications user. In this case you'll only update the database and cache. But I think you can complete these points.
const redis = require('redis');
const redisClient = redis.createClient();
app.get('/notifications', async (request, response) => {
const userId = request.user.id;
const cacheResult = await redisClient.get(`user:${userId}:notifications`)
if (cacheResult) return response.send(cacheResult);
const notifications = getUserNotificationsFromDatabase(userId);
redisClient.set(`user:${userId}:notifications`, notifications);
response.send(notifications);
})
Besides that there's another way, you can simple use only the redis or only the database to manage this notification. Your relational database with the correct index will send to your the results as faster as you expect. You'll only think about how much notifications you'll have been.
I have a database with a bunch of dates and an online overview where you can view them, now I know I can copy a URL from my Google Agenda and import this in other calendar clients so I can view the events there.
I want to generate an Express endpoint where I fetch every event every time the endpoint is called and return it in a format that can be imported by other calendar clients. Now with packages like iCal-generator I could generate, read, and return the file whenever a user requests the URL. but it feels redudent to write a file to my storage to then read it, return it and delete it every time it's requested.
What is the most effiecent way to go about this?
Instead of generating the file/calendar data on every request, you could implement a simple caching mechanism. That is, upon start of your node app you generate the calendar data and put it in your cache with corresponding time to live value. Once the data has expired or new entries are inserted into your DB you invalidate the cache, re-generate the data and cache it again.
Here's a very simple example for an in-memory cache that uses the node-cache library:
const NodeCache = require('node-cache');
const cacheService = new NodeCache();
// ...
const calendarDataCacheKey = 'calender-data';
// at the start of your app, generate the calendar data and cache it with a ttl of 30 min
cacheCalendarData(generateCalendarData());
function cacheCalendarData (calendarData) {
cacheService.set(calendarDataCacheKey, calendarData, 1800);
}
// in your express handler first try to get the value from the cache
// if not - generate it and cache it
app.get('/calendar-data', (req, res) => {
let calendarData = cacheService.get(calendarDataCacheKey);
if (calendarData === undefined) {
calendarData = generateCalendarData();
cacheCalendarData(calendarData);
}
res.send(calendarData);
});
If your app is scaled horizontally you should consider using redis.
100% untested, but I have code similar to this that exports to a .csv from a db query, and it might get you close:
const { Readable } = require('stream');
async function getCalendar(req, res) {
const events = await db.getCalendarEvents();
const filename = 'some_file.ics';
res.set({
'Content-Type': 'text/calendar',
'Content-Disposition': `attachment; filename=${filename}`,
});
const input = new Readable({ objectMode: true });
input.pipe(res)
.on('error', (err) => {
console.error('SOME ERROR', err);
res.status(500).end();
});
events.forEach(e => input.push(e));
input.push(null);
}
if you were going to use the iCal generator package, you would do your transforms within the forEach method before pushing to the stream.
I have been staring at this for hours and can't find a solution and that is even though by all suggestions it SHOULD be quite easy - https://learn.microsoft.com/en-us/bot-framework/nodejs/bot-builder-nodejs-proactive-messages.
I have created a simple code which will "register" the user and save their data in my cosmosDatabse on Azure. That works perfectly.
//ON "register" SAVE USER DATA AND SAY REGISTERED MESSAGE
bot.dialog('adhocDialog', function(session, args) {
var savedAddress = session.message.address;
session.userData.savedAddress = savedAddress;
//REGISTERED MESSAGE
session.endDialog("*Congratulations! You are now registered in our network! (goldmedal)*");
})
.triggerAction({
matches: /^register$/i
})
But how can I then access that specific user and send him a message if, say, a condition is met? (in fact on HTTP request)
I am fairly certain we have to write the conversation ID or user ID somewhere. The question is where?
function startProactiveDialog(address) {
bot.beginDialog(address, "A notification!");
}
This is how simple I think it should be. But where do you specify the user then?
You've saved the address of the user inside of your database by saving it to session.userData.savedAddress. When the event triggers, perform a query to your database that checks for the users that meet two criteria.
They're registered to listen for the event
Their address has been saved inside of the database.
In your case, you can save a property to the session.userData object, a property that lists which events they're listening for. If you just need to send a message to the user, then you can simply use bot.loadSession(savedAddress) to ping the user.
Edit:
So instead of looking specifically by user ID, you should send a query to your CosmosDB that looks for entries that have a "listen-to" Boolean-type flag corresponding to the event.
You're not worrying about the user ID at first, you're just retrieving all entries with a query that would (broadly speaking) look like this:
SELECT * FROM BotState WHERE data LIKE 'listenForEvent=1.
So to setup your session.userData so that the above theoretical query would work, you would need to modify that snippet of code in your question to something like the following:
bot.dialog('adhocDialog', function(session, args) {
var savedAddress = session.message.address;
session.userData.savedAddress = savedAddress;
session.userData.listenForEvent = 1 // Our property we're going to look for.
session.endDialog("*Congratulations! You are now registered in our network! (goldmedal)*");
})
.triggerAction({
matches: /^register$/i
})
Actually, the savedAddress should be an instance of IAddress, and also, the function loadSession(address: IAddress, callback: (err: Error, session: Session) => void): void; and address(adr: IAddress): Message; under Message class all require IAddress as the parameter.
So first of all, you should save the entire address json object in cosmosDB for later using.
As botbuilder for Node.js is built on Restify or Express, you can build an addition route for your user to trigger and send proactive messages. The work flow could be following:
Guide user to register & Save the user's address object with the account mapping in your DB
Create a Route in Restify or Expressjs for trigger the proactive message:
server.get('/api/CustomWebApi', (req, res, next) => {
//find the user's address in your DB as `savedAddress`
var msg = new builder.Message().address(savedAddress);
msg.text('Hello, this is a notification');
bot.send(msg);
res.send('triggered');
next();
}
);
or if you want to leverage loadSession
server.get('/api/CustomWebApi', function (req, res, next) {
bot.loadSession(savedAddress, (err, session) => {
if (!err) {
session.send('Hello, this is a notification')
session.endConversation();
}
})
res.send('triggered');
next();
});
I created a users.json file, to which I save all the users. It works the way I need it to. I guess database would be better, but I don't really have a clue where to begin with that. Database is a whole new chapter I have not encountered yet, so it doesn't make sense to work on it when the project needs are resolved.
I built a simple API endpoint with NodeJS using Sails.js.
When someone access my API endpoint, the server starts to wait for data and whenever a new data appears, he broadcasts it using sockets. Each client should receive his own stream of data based on his user input.
var Cap = require('cap').Cap;
collect: function (req, res) {
var iface = req.param("ip");
var c = new Cap(),
device = Cap.findDevice(ip);
c.on('data', function(myData) {
sails.sockets.blast('message', {"host": myData});
});
});
The response do not complete (I never send a res.json() - what actually happens is that the browser keep loading - but the above functionality works).
2 Problems:
I'm trying to subscribe and unsubscribe to to this API endpoint from my client (using RxJS). When I subscribe, I start to receive data via sockets - but I can't unsubscribe to the API endpoint (the browser expect the request to be completed).
Each client should subscribe to his own socket room based on the request IP parameter ( see updated code ). Currently it blasts the message to everyone.
How I can create a stream/service-like API endpoint with Sails.js that will emit new data to each user based on his input?
My goal is to be able to subscribe / unsubscribe to this API endpoint from each client.
Revised Answer
Let's assume your API endpoint is defined in config/routes.js like this:
...
'get /collect': 'SomeController.collectSubscribe',
'delete /collect': 'SomeController.collectUnsubscribe',
Since each Cap instance is tied to one device, we need one instance for each subscription. Instead of using the sails join/leave methods, we keep track of Cap instances in memory and just broadcast to the request socket's id. This works because Sails sockets are subscribed to their own ids by default.
In api/controllers/SomeController.js:
// In order for the `Cap` instances to persist after `collectSubscribe` finishes, we store them all in an Object, associated with which socket the were created for.
var caps = {/* req.socket.id: <instance of Cap>, */};
module.exports = {
...
collectSubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest("I need a websocket! Help!");
if (!!caps[req.socket.id]) return res.badRequest("Dude, you are already subscribed.");
caps[req.socket.id] = new Cap();
var c = caps[req.socket.id]; // remember that `c` is a reference to our new `Cap`, not a copy.
var device = c.findDevice(req.param('ip'));
c.open(device, ...);
c.on('data', function(myData) {
sails.sockets.broadcast(req.socket.id, 'message', {host: myData});
});
return res.ok();
},
collectUnsubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest("I need a websocket! Help!");
if (!caps[req.socket.id]) return res.badRequest("I can't unsubscribe you unless you actually subscribe first.");
caps[req.socket.id].removeAllListeners('data');
delete caps[req.socket.id];
return res.ok();
}
}
Basically, it goes like this: when a browser request triggers collectSubscribe, a new Cap instance listens to the provided IP. When the browser triggers collectUnsubscribe, the server retreives that Cap instance, tells it to stop listening, and then deletes it.
Production Considerations: please be aware that the list of Caps is NOT PERSISTENT (since it is stored in memory and not a DB)! So if your server is turned off and rebooted (due to lightning storm, etc), the list will be cleared, but considering that all websocket connections will be dropped anyway, I don't see any need to worry about this.
Old Answer, Kept for Reference
You can use sails.sockets.join(req, room) and sails.sockets.leave(req, room) to manage socket rooms. Essentially you have a room called "collect", and only sockets joined in that room will receive a sails.sockets.broadcast(room, eventName, data).
More info on how to user sails.sockets here.
In api/controllers/SomeController.js:
collectSubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest();
sails.sockets.join(req, 'collect');
return res.ok();
},
collectUnsubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest();
sails.sockets.leave(req, 'collect');
return res.ok();
}
Finally, we need to tell the server to broadcast messages to our 'collect' room.
Note that this only need to happen once, so you can do this in a file under the config/ directory.
For this example, I'll put this in config/sockets.js
module.exports = {
// ...
};
c.on('data', function(myData) {
var eventName = 'message';
var data = {host: myData};
sails.sockets.broadcast('collect', eventName, data);
});
I am assuming that c is accessible here; If not, you could define it as sails.c = ... to make it globally accessible.
I've got a form submission that accepts an image. I'm creating thumbnails (resizing and pushing to S3) with the image upon submission, and it's taking awhile and blocking so I've decided I want to push it to a message queue and have that handle it.
I've decided to go with node-redis, since I'm already using redis in my stack. What I'm unclear on is how exactly the implementation would look (in its most basic form).
Consider some pseudocode below as:
var redis = require('redis'),
client = redis.createClient();
function listenForJob() {
client.on('message', function(msg) {
// msg is our temporary path name
// Kick of resize and push to s3 job
});
}
// Attached to my route where I POST to (e.g. /submit)
exports.form = function(req, res, next) {
// Input comes in, and image comes in as req.files
// and a temporary image is saved in /uploads
// which I'll save as the image submission for now
// until the process to resize and push to s3 happens.
listenForJob();
// Save to db
var data = {
title: req.body.title,
img: req.files.path // save temp file
}
connection.query('INSERT INTO submissions SET ?', data, function(err, rows) {
// Publish to our redis client?
client.publish('message', req.files.path);
connection.release();
res.redirect('/submissions');
});
};
Is this implementation even remotely the correct way to approach this? I'm new to taskworkers/message queues so I'm just wondering how to do implement it properly (given my use case).