I am creating an application which would watch a file and fetch the contents from that file (similar to tail but with the possibility of paging in previous data as well). I read up on quite a few solutions ranging from spawning a new process to getting only the updated bytes of the file but I am still a little confused on a few parts.
What I want to do exactly is the following:
Watch a file and trigger an event/callback whenever new data comes into the file
Read this new data from the file and efficiently send it to a client. Using a websocket or something else. (suggest a good way to do this please)
At the client end, take this data and display it to user and keep updating it with new data as it comes
If the user requests older data a way to fetch that data from the file we are watching
I am looking for efficient solutions for the above sub problems and any suggestions for a better approach are also welcome.
FYI I am new to nodejs so verbosity in your solutions would be highly appreciated.
Watch for changes
Suggest you look at chokidar, it is an optimized implementation of fs.watch, fs.events, the native node.js libraries.
// Initialize watcher.
const watcher = chokidar.watch('some/directory/**/*.xml', config);
// Add event listeners.
watcher
.on('add', path => log(`File ${path} has been added`))
.on('change', path => log(`File ${path} has been changed`))
.on('unlink', path => log(`File ${path} has been removed`));
To get the changed value
Here you can look at diff module. And you will need to store the state of the previous and current files. In order to build the changes.
To notify the client
You will need to create a websocket server, recommend you to use socket.io and then in your application you will create the diff and send a websocket message to the server. The server will notify/broadcast the message to the needed clients.
Related
It's been a while since I used node and express and I was sure that this was possible, but i'm having an issue of figuring it out now.
I have a simple postgres database with sequelize. I am building a back end and don't have a populated database yet. I want to be able to provide fake data to use to build the front end and to test with. Is there a way to populate a database when the node server is started? Maybe by reading a json file into the database?
I know that I could point to this fake data using a setting in the environment file, but I don't see how to read in the data on startup. Is there a way to create a local database, read in the data, and point to that?
You can use fake factory package, I think it can solve your problem.
https://www.npmjs.com/package/faker-factory
FakerJs provides that solution.
import { faker } from '#faker-js/faker';
const randomName = faker.name.findName();
const randomEmail = faker.internet.email();
with the above, you can run a loop for loop to be specific to create
the desired data you may need for your project.
Also, check on the free web-API that provides fake or real data to workon
So being novice in Express and Node, I fear I must be not understanding something fundamental about middleware. I am trying to upload a file using express-formidable, and I've got it to work (as far as taking the file and uploading it to the directory of my choice). However, I would love a progress bar or to be able to do something at the start of the upload such as choose the file name. I see a lot of examples of regular formidable using the 'progress' or 'file' event. But for the life of me I can't figure out how to access this with express-formidable. Since I can't create an IncomingForm, I don't know where the events are or how to bind them.
What am I missing about how express-formidable works? Does it cut out everything about formidable and just stuff a file where you tell it? Or do you have access to everything formidable?
Here's an example of a route I have called ingest, which is where I receive an uploaded file and process it.
app.post('/ingest', function(req, res) {
/*I want to be able to show progress here, and do other things
like set the file name before it's saved, but by the time I get here
the file is already processed and saved, and I can't figure out how
to access events using '.on' if there's no form object*/
});
From this question, Sails js using models outside web server I learned how to run a command from the terminal to update records. However, when I do this the changes don't show up until I restart the server. I'm using the sails-disk adapter and v0.9
According to the source code, the application using sails-disk adapter loads the data from file only once, when the corresponding Waterline collection is being created. After that all the updates and destroys happen in the memory, then the data is being dumped to the file, but not being re-read.
That said, what's happening in your case is that once your server is running, it doesn't matter if you are changing the DB file (.tmp/disk.db) using your CLI instance, 'cause lifted Sails server won't know about the changes until it's restarted.
Long story short, the solution is simple: use another adapter. I would suggest you to check out sails-mongo or sails-redis (though the latter is still in development), for both Mongo and Redis have data auto expiry functionality (http://docs.mongodb.org/manual/tutorial/expire-data/, http://redis.io/commands/expire). Besides, sails-disk is not production-suitable anyways, so sooner or later you would need something else.
One way to accomplish deleting "expired records" over time is by rolling your own "cron-like job" in /config/bootstrap.js. In psuedo code it would look something like this:
module.exports.bootstrap = function (cb) {
setInterval(function() { < Insert Model Delete Code here> }, 300000);
cb();
};
The downside to this approach is if it throws it will stop the server. You might also take a look at kue.
Is there a service that creates basically a one-time download of a file, preferably something I can use from NodeJS?
I've done some research on FilePicker, and haven't found anything about regenerating the link it gives you for a file. There may be a way to do this with NodeJS, but I'm using Meteor at the same time so many Node things probably will conflict.
You could build it with meteor. Using meteor-router with meteorite & use server side routing to deliver the files.
You need a collection to keep track of downloaded files:
Server JS
var downloads = new Meteor.Collection("downloads");
//create a link
downloads.insert({url:"/mydownload.zip",downloaded:false})
Meteor.Router.add('/file/:id', 'GET', function(id) {
download = downloads.findOne(id);
if( download) {
if(dowload.downloaded) {
this.response.send("You've already downloaded me")
}
else
{
//I guess you could just redirect or stream the file for an extra layer of surety
this.response.redirect(download.url);
}
}
});
On the client you can use /files/{{_id}} with _id of the file from downloads the person has as the link
My recommendation would also be to add custom server-side logic to count # of uploads (or just flag a file as downloaded/not downloaded) and respond accordingly. The closest you could do with Filepicker.io would be using the security policies to restrict downloading the file to a specific time interval.
in addition to using the router package
in Meteor.startup you can add
var require = __meteor_bootstrap__.require;
fs = require( 'fs' );
the fs variable should be declared on the server only. the fs package is used by Meteor and does not need to be added separately.
once you have done this, you can create files with Meteor.uuid() as their name which makes them unique and very difficult to guess. It is also possible to delete the file after a certain amount of time by using Meteor.setTimeout
the question is: where do the files to be downloaded come from?
Solution using Heroku Cloud and NodeJS Meteor Hooks
Heroku in particular is actually great for temporary file download links: they offer a "temporary scratchpad" filesystem that is reset every time the program restarts, and each running Node server cannot see the files other instances have created.
Each dyno gets its own ephemeral filesystem, with a fresh copy of the
most recently deployed code. During the dyno’s lifetime its running
processes can use the filesystem as a temporary scratchpad, but no
files that are written are visible to processes in any other dyno and
any files written will be discarded the moment the dyno is stopped or
restarted.
Taken from the Heroku documentation: https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
Thus, any files written to the "filesystem" will be temporary.
This allows for a very easy solution to this problem: you can simply use NodeJS filesystem manipulation to create temporary files on the server, serve them once (or for a limited time), and then remove them so they cannot be downloaded again.
This in combination with something like $.download() will make a seamless experience which in turn prevents unauthorized downloads.
Last night I dump windows 7 and formatted my hard driver to port to a Linux based operating system, Purely for the reasons that I wanted to start working with Node.JS
So I have installed Node.JS and have done a few test stuff, the http server and sockets etc.
What I would like to do is build a HTTP Server that is tightly intergrated with an MVC Framework, but before I get started on all that I need to learn how to build efficiently in Node.
For example within PHP as my framework I would create a bootloading system to load all base classes etc, then i would fire my events system ready to start attaching callbacks.
I would continue to process the request etc until the output is generated which then gets sent of to an output handler that would process headers etc etc
But Node s a totally new environment for this and im wondering on the best practises to build an system in Node.
The information im looking for is more to do with the design structure rather then the actual coding of the application, how to load the lib where to load the libs, etc etc
Any help is appreciated.
So far my WebApplication is coming along nicely, I have built my application pretty traditionally and a little procedural.
What i have started out is creating a directory structure like so:
<root>
startup.js
/public/
favicon.ico
/images/
/stylesheets/
/javascripts/
/system/
init.js
config.js
/libs/
/exceptions/
http.js
server.js
/application/
/views/
/_override/
/errors/
generic.view
/partials/
sidebar.voew
index.view
/controllers/
index.js
/models/
users.js
This directory structure is like most MVC Based Web Applications out there so using this method I feel comfortable.
The startup file is whats executed by node as the entry point, node startup & and looks like so:
/*
* Header of t he file, Copyright etc
*/
var _Intitialize = require("./system/init.js");
//Displays the command line header, title, copyright etc
_Intitialize.DisplayCommandLineHeader();
//Check the enviroment, Permissions, Ports etc
_Intitialize.CheckEnviroment();
//Start the server and listen the port.
_Initialize.StartServer();
the init file is the main work, its what tells all other areas of the system to run, stop etc.
I have a file in libs called serverhandler.js, and this is required into init.js, I then create a server and assign the callback to the ServerHandler.Listener. Who then listens for requests, checks to see if the file exists in public directory, if so it then reads in chunks and sends back.
if no file was found in public it would then create a route with Route.Create("/path?params"); which deters 3 elements, Controller, Method, Params from the uri, and then the controller files are loaded if exists.
I've taken on the approach of throwing error pages like so:
if(!FileSystem.exists(RequiredPath))
{
throw new HTTPExceptions.FileNotFound();
}
Hope this helps some people getting started in Node.
Have a look at
http://dailyjs.com/2010/11/01/node-tutorial/ , it's pretty relevant.
I would suggest looking at the current modules too
https://github.com/joyent/node/wiki/modules
and reading the code of any of the projects in the areas you are interested in, esp. the middleware, routing and module loaders.