Is there a service that creates basically a one-time download of a file, preferably something I can use from NodeJS?
I've done some research on FilePicker, and haven't found anything about regenerating the link it gives you for a file. There may be a way to do this with NodeJS, but I'm using Meteor at the same time so many Node things probably will conflict.
You could build it with meteor. Using meteor-router with meteorite & use server side routing to deliver the files.
You need a collection to keep track of downloaded files:
Server JS
var downloads = new Meteor.Collection("downloads");
//create a link
downloads.insert({url:"/mydownload.zip",downloaded:false})
Meteor.Router.add('/file/:id', 'GET', function(id) {
download = downloads.findOne(id);
if( download) {
if(dowload.downloaded) {
this.response.send("You've already downloaded me")
}
else
{
//I guess you could just redirect or stream the file for an extra layer of surety
this.response.redirect(download.url);
}
}
});
On the client you can use /files/{{_id}} with _id of the file from downloads the person has as the link
My recommendation would also be to add custom server-side logic to count # of uploads (or just flag a file as downloaded/not downloaded) and respond accordingly. The closest you could do with Filepicker.io would be using the security policies to restrict downloading the file to a specific time interval.
in addition to using the router package
in Meteor.startup you can add
var require = __meteor_bootstrap__.require;
fs = require( 'fs' );
the fs variable should be declared on the server only. the fs package is used by Meteor and does not need to be added separately.
once you have done this, you can create files with Meteor.uuid() as their name which makes them unique and very difficult to guess. It is also possible to delete the file after a certain amount of time by using Meteor.setTimeout
the question is: where do the files to be downloaded come from?
Solution using Heroku Cloud and NodeJS Meteor Hooks
Heroku in particular is actually great for temporary file download links: they offer a "temporary scratchpad" filesystem that is reset every time the program restarts, and each running Node server cannot see the files other instances have created.
Each dyno gets its own ephemeral filesystem, with a fresh copy of the
most recently deployed code. During the dyno’s lifetime its running
processes can use the filesystem as a temporary scratchpad, but no
files that are written are visible to processes in any other dyno and
any files written will be discarded the moment the dyno is stopped or
restarted.
Taken from the Heroku documentation: https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
Thus, any files written to the "filesystem" will be temporary.
This allows for a very easy solution to this problem: you can simply use NodeJS filesystem manipulation to create temporary files on the server, serve them once (or for a limited time), and then remove them so they cannot be downloaded again.
This in combination with something like $.download() will make a seamless experience which in turn prevents unauthorized downloads.
Related
I’m using Electron Builder to compile my Electron app to an .AppImage file, and I’m using the fs module to write to an .json file, but it’s not working in the appimage format (it’s working fine when I have the normal version not made with Electron Builder). I can still read from the file.
The code (preload):
setSettings: (value) => {fs.writeFileSync(path.join(__dirname, "settings.json"), JSON.stringify(value), "utf8")}
The code (on the website):
api.setSettings(settings);
The project: https://github.com/Nils75owo/crazyshit
That's not a problem with AppImage or Electron Builder but with the way you're packaging your app. Since you didn't post your package.json*, I can only guess what's wrong, but probably you haven't changed Electron Builder's default behaviour regarding packing your application.
By default, Electron Builder compiles your application, including all resources, into a single archive file in the ASAR format (think of it like the TAR format). Electron includes a patched version of the fs module to be able to read from the ASAR file, but writing to it is obviously not supported.
You have two options to mitigate this problem: Either you store your settings somewhere in the user's directory (which is the way I'd go, see below) or you refrain from packing your application to an ASAR file, but that will leave all your JavaScript code outside the executable in a simple folder. (Note that ASAR is not capable of keeping your code confidential, because there are applications which can extract such archives, but it makes it at least a little harder for attackers or curious eyes to get a copy of your code.)
To disable packing to ASAR, simply tell Electron Builder that you don't want it to compile an archive. Thus, in your package.json, include the following:
{
// ... other options
"build": {
// ... other build options
"asar": false
}
}
However, as I mentioned above, it's probably wiser to store settings in a common place where advanced users can actually find (and probably edit, mostly for troubleshooting) them. On Linux, one such folder would be ~/.config, where you could create a subdirectory for your application.
To get the specific application data path on a cross-platform basis, you can query Electron's app module from within the main process. You can do so like this:
const { app } = require ("electron"),
path = require ("path");
var configPath;
try {
configPath = path.join (app.getPath ("appData"), "your-app-name");
} catch (error) {
console.error (error);
app.quit ();
}
If you however have correctly set your application's name (by using app.setName ("...");), you can instead simply use app.getPath ("userData"); and omit the path joining. Take a look at the documentation!
For my Electron Applications, I typically choose to store settings in a common hidden directory (one example for a name could be the brand under which you plan to market the application) in the user's home directory and I then have separate directories for each application. But how you organise this is up to you completely.
* For the future, please refrain from directing us to a GitHub repository and instead include all information (and code is information too) needed to replicate/understand your problem in your question. That'd save us a lot of time and could potentially get you answers faster. Thanks!
I am creating an application which would watch a file and fetch the contents from that file (similar to tail but with the possibility of paging in previous data as well). I read up on quite a few solutions ranging from spawning a new process to getting only the updated bytes of the file but I am still a little confused on a few parts.
What I want to do exactly is the following:
Watch a file and trigger an event/callback whenever new data comes into the file
Read this new data from the file and efficiently send it to a client. Using a websocket or something else. (suggest a good way to do this please)
At the client end, take this data and display it to user and keep updating it with new data as it comes
If the user requests older data a way to fetch that data from the file we are watching
I am looking for efficient solutions for the above sub problems and any suggestions for a better approach are also welcome.
FYI I am new to nodejs so verbosity in your solutions would be highly appreciated.
Watch for changes
Suggest you look at chokidar, it is an optimized implementation of fs.watch, fs.events, the native node.js libraries.
// Initialize watcher.
const watcher = chokidar.watch('some/directory/**/*.xml', config);
// Add event listeners.
watcher
.on('add', path => log(`File ${path} has been added`))
.on('change', path => log(`File ${path} has been changed`))
.on('unlink', path => log(`File ${path} has been removed`));
To get the changed value
Here you can look at diff module. And you will need to store the state of the previous and current files. In order to build the changes.
To notify the client
You will need to create a websocket server, recommend you to use socket.io and then in your application you will create the diff and send a websocket message to the server. The server will notify/broadcast the message to the needed clients.
I'm trying out gun.js I have it installed as a node.js project, I have configured the amazon S3 bucket through the dotenv and I have tried adding a data.json file and still I cant get gun.js to save the file locally or to he S3 bucket.
I know its early days for gun, but I get the feeling I'm missing something obvious.
I'm expecting to find a .json file in he local file system and or in the S3 bucket but I get neither.
require('dotenv').config();
var Gun = require('gun');
var gun = Gun({
file: 'data.json', // local testing and development
s3: {
key: process.env.AWS_KEY, // AWS Access Key
secret: process.env.AWS_SECRET, // AWS Secret Token
bucket: process.env.AWS_BUCKET // The bucket you want to save into
}
});
gun.put({ hello: 'world' }).key('my/first/data');
#bill Just noticed this now, sorry for the late answer. Thanks to #paul-w for notifying me of this and his response earlier today.
This question and answer assumes you are running a version EARLIER than v0.4.x!
If you are in NodeJS and are getting the error “You have no persistence layer to save to”, it means the default storage drivers (S3, file.js) didn't get installed or were deactivated - which is unusual as this happens automatically.
Try installing gun (again?) via npm install gun in your local NodeJS project directory, not a git clone or a copy&paste.
I can only guess, given the context you explain, that you might have copied/moved gun (like the gun.js file) into your project. The browser will work with just the single file, but NodeJS needs more - it needs the S3/file.js modules, which will be included if installed with npm or properly git cloned.
Also unlikely (since your code doesn't show this), if you happen to (this is bad) Gun({wire: {put: null, get: null}}) (or something similar) it would intentionally break the persistence drivers.
If you are in the browser and getting the error (and assuming your not overwriting the persistence drivers like in the previous paragraph), it could be because of some weird situation like you are using an old version of IE or a browser that doesn't have JSON support. Again, all these things are unlikely but I'm just wanting to be comprehensive.
Note: The above applies to the question in your title. However your actual question doesn't ask about the error, it asks about not seeing data in data.json or in S3. Answering that below.
To which #paul-w is more on track. If you are using S3 then the file.js module (data.json) automatically deactivates itself. If you are using the file.js module (data.json) then S3 does not get activated. As #paul-w mentioned, v0.4.x will support easily having multiple storage engines simultaneously. However, you should see your data in at least one or the other - unless you are getting the "no persistence layer" error, in which case you won't see your data anywhere because there isn't any persistence! But again, default persistence layers are included with gun by default (unless installation was incorrect, or you explicitly overwrite them - both unusual things).
I hope this answers your question. Sorry I didn't see it till now. Please let me know if this works, and also join the conversation at https://gitter.im/amark/gun . Thank you for helping start the stackoverflow questions! We need more of these!
I think Mark is going to answer this more officially, but the quick answer is that in gun.js 0.3 (current) there is a single gun server peer or storage target, and when you run gun as a server (e.g. from node.js rather than a browser), S3 is preferred, if S3 credentials are specified. But gun is also saving your data changes in browser memory, or localStorage (up to the browser limit of 5MB), and S3 is there for a more permanent storage.
So in the example above, I think the problem is that the file entry will only be used if there is a problem saving changes to S3, and that's why you don't see the new data going there. Maybe try putting an error in the S3 credentials (e.g. add an 'x' for now) and see if it starts using the file path instead.
In gun.js 0.4 there are plans to make use of all peers specified in the constructor or dynamically, but that feature isn't here yet.
(And I probably butchered that answer, but hopefully Mark can correct any inaccuracies in this. I'm new to gun.js but had the same question.)
I am developing an npm module, where user can interact with it through a terminal by executing commands:
> mymodule init
> mymodule do stuff
When executing certain commands user is being asked for some data, which will be used by the module. Since this data won't really change while using the module and since these commands can be executed pretty frequently, it is not the best option to ask user for the data any time he runs a command. So I've decided to cache this data, and as soon as it should live through multiple module calls, the easiest way to store it that I see is a file (the data structure allows to store it in a simple JSON). But I am to quite sure where should this file go on a user's machine.
Where in the file system should I store a cache in a form of a file or multiple files for a custom npm module, considering that the module itself can be installed globally, on multiple operation systems and can be used in multiple projects at the same time?
I was thinking about storing it in a module's folder, but it might be tricky in case of global installation + multi-project use. The second idea was to store it in OS specific tmp storage, but I am not quite sure about it too. I am also wondering if there are some alternatives to file storage in this case.
I would create a hidden folder in the user's home directory (starting with a dot). For instance, /home/user/.mymodule/config.cfg. The user's home directory isn't going anywhere, and the dot will make sure it's out of the user's way unless they go looking for it.
This is the standard way that most software stores user configs, including SSH, Bash, Nano, Wine, Ruby, Gimp, and even NPM itself.
On some systems you can cache to ~/.cache by create a sub-directory to store your cache data, though its much more common for applications to create a hidden directory in the users home directory. On modern windows machines you can use create a directory in C:/Users/someUser/AppData. In Windows using a . suffix will not hide a file. I'd recommend you do something platform agnostic like so:
var path = require('path');
function getAppDir(appName, cb) {
var plat = process.platform;
var homeDir = process.env[(plat == 'win32') ? 'USERPROFILE' : 'HOME'];
var appDir;
if(plat == 'win32') {
appDir = path.join(homeDir, 'AppData', appName);
}
else {
appDir = path.join(homeDir, '.' + appName);
}
fs.mkdir(appDir, function(err) {
if(err) return cb(err);
cb(null, appDir);
})
}
Just declare a function to get the app directory. This should handle most systems, but if you run into a case where it does not it should be easy to fix because you can just create some kind of alternate logic here. Lets say you want to allow a user to specify a custom location for app data in a config file later on, you could easily add that logic. For now this should suite most of your cases for most all Unix/Linux systems and Windows Vista and up.
Storing in system temp folder, depending on the system, your cache could be lost on an interval(cron) or on reboot. Using the global install path would lead to some issues. If you need this data to be unique per project, then you can extend that functionality to allow you to store this data in the project root, but not the module root. It would be best not to store it in the module root, even if its just installed as a local/project module, because then the user doesn't have the ability to include this folder in their repositories without including the entire module.
So in the event that you need to store this cached data relevant to a project, then you should do so in the project root not the node_modules. Otherwise store it in the users home directory in a system agnostic way.
First you need to know in what kind of SO you are running:
Your original idea is not bad, because global modules are not really global in all SO and in all virtual enviroments.
Using /home/user may not work in Windows. In windows you have to check process.ENV.HOMEPAHT
I recommend you a chain of checks to determine the best place.
Let the user take the control. Chose your own env variable. Supouse MYMOD_HOME. You firts check if process.ENV.MYMOD_HOME exists, and use it
Check if windows standard process.ENV.LOCALAPPDATA
Check if windows standard process.ENV.HOMEPATH
Check if exists '/home/user' or '~'
Otherwise use __dirname
In all cases create a directory ./mymodule
Last night I dump windows 7 and formatted my hard driver to port to a Linux based operating system, Purely for the reasons that I wanted to start working with Node.JS
So I have installed Node.JS and have done a few test stuff, the http server and sockets etc.
What I would like to do is build a HTTP Server that is tightly intergrated with an MVC Framework, but before I get started on all that I need to learn how to build efficiently in Node.
For example within PHP as my framework I would create a bootloading system to load all base classes etc, then i would fire my events system ready to start attaching callbacks.
I would continue to process the request etc until the output is generated which then gets sent of to an output handler that would process headers etc etc
But Node s a totally new environment for this and im wondering on the best practises to build an system in Node.
The information im looking for is more to do with the design structure rather then the actual coding of the application, how to load the lib where to load the libs, etc etc
Any help is appreciated.
So far my WebApplication is coming along nicely, I have built my application pretty traditionally and a little procedural.
What i have started out is creating a directory structure like so:
<root>
startup.js
/public/
favicon.ico
/images/
/stylesheets/
/javascripts/
/system/
init.js
config.js
/libs/
/exceptions/
http.js
server.js
/application/
/views/
/_override/
/errors/
generic.view
/partials/
sidebar.voew
index.view
/controllers/
index.js
/models/
users.js
This directory structure is like most MVC Based Web Applications out there so using this method I feel comfortable.
The startup file is whats executed by node as the entry point, node startup & and looks like so:
/*
* Header of t he file, Copyright etc
*/
var _Intitialize = require("./system/init.js");
//Displays the command line header, title, copyright etc
_Intitialize.DisplayCommandLineHeader();
//Check the enviroment, Permissions, Ports etc
_Intitialize.CheckEnviroment();
//Start the server and listen the port.
_Initialize.StartServer();
the init file is the main work, its what tells all other areas of the system to run, stop etc.
I have a file in libs called serverhandler.js, and this is required into init.js, I then create a server and assign the callback to the ServerHandler.Listener. Who then listens for requests, checks to see if the file exists in public directory, if so it then reads in chunks and sends back.
if no file was found in public it would then create a route with Route.Create("/path?params"); which deters 3 elements, Controller, Method, Params from the uri, and then the controller files are loaded if exists.
I've taken on the approach of throwing error pages like so:
if(!FileSystem.exists(RequiredPath))
{
throw new HTTPExceptions.FileNotFound();
}
Hope this helps some people getting started in Node.
Have a look at
http://dailyjs.com/2010/11/01/node-tutorial/ , it's pretty relevant.
I would suggest looking at the current modules too
https://github.com/joyent/node/wiki/modules
and reading the code of any of the projects in the areas you are interested in, esp. the middleware, routing and module loaders.