I'm using a file for storing JSON data. My module makes CRUD actions on the file and I'm using require() to load the json, instead of fs.readFile(). The issue is, if the file is deleted, using fs.unlink(), then calling the file again using require still loads the file... which has just been deleted. I'm a bit lost how to get around this, possibly #garbage-collection?
Example:
fs.writeFile('foo.json', JSON.stringify({foo:"bar"}), function(){
var j = require('./foo.json')
fs.unlink('./foo.json', function(){
console.log('File deleted')
var j = require('./foo.json')
console.log(j)
})
})
When loading a module using require, Node.js caches the loaded module internally so that subsequent calls to require do not need to access the drive again. The same is true for .json files, when loaded using require.
That's why your file still is "loaded", although you deleted it.
The solution to this issue is to use the function for loading a file that is appropriate for it, which you already mentioned: fs.readFile(). Once you use that, everything will work as expected.
Related
So, i am trying to download the contents of a directory via sftp using nodejs, and so far I am getting stuck with an error.
I am using the ssh2-sftp-client npm package and for the most part it works pretty well as i am able to connect to the server and list the files in a particular remote directory.
Using the fastGet method to download a file also works without any hassles, and since all the methods are promise based i assumed i could easily download all the files in the directory simply enough, by doing something like:
let main = async () => {
await sftp.connect(config.sftp);
let data = await sftp.list(config.remote_dir);
if (data.length) data.map(async x => {
await sftp.fastGet(`${config.remote_dir}/${x.name}`, config.base_path + x.name);
});
}
So it turns out the code above successfully downloads the first file, but then crashes with the following error message:
Error: Failed to get sandbox/demo2.txt: The requested operation cannot be performed because there is a file transfer in progress.
This seems to indicate that the promise from fastGet is resolving too early as the file transfer is supposed to be over when the next element of the file list is processed.
I tried to use the more traditional get() instead but it is using streams, and it fails with a different error. After researching it seems there's been a breaking change regarding streams in node 10.x. well in my case calling get simply fails (not even downloading the first file).
Does anyone know a workaround to this? or else, another package that can download several files by sftp?
Thanks!
I figured out, since the issue was concurrent download attempts on one client connection, i could try to manage it with one client per file download. I ended up with the following recursive function.
let getFromFtp = async (arr) => {
if (arr.length == 0) return (processFiles());
let x = arr.shift();
conns.push(new Client());
let idx = conns.length - 1;
await conns[idx].connect(config.sftp.auth);
await conns[idx]
.fastGet(`${config.sftp.remote_dir}/${x.name}`, `${config.dl_dir}${x.name}`);
await connections[idx].end();
getFromFtp(arr);
};
Notes about this function:
The array parameter is a list of files to download, presumably fetched using list() beforehand
conns was declared as an empty array and is used to contain our clients.
using array.prototype.shift(), to gradually deplete the array as we go through the file list
the processFiles() method is fired once all the files were downloaded.
this is just the POC version. of couse we need to add the error management to that.
I have a UDP client that grabs some data from another source and writes it to a file on the server. Since this is large amount of data, I dont want the end user to wait until they its full written to the server so that they can download it. So I made a NodeJS server that grabs the latest data from the file and sends it to the user.
Here is the code:
var stream = fs.readFileSync(filename)
.on("data", function(data) {
response.write(data)
});
The problem here is, if the download starts when the file was only for example 10mb.. the fs.readFileSync will only read my file up to 10mb. Even if 2 mins later the file increased to 100mb. fs.readFileSync will never know about the new updated data. How can I do this in Node? I would like somehow refresh the fs state or maybe perpaps wait for new data using fs file system. Or is there some kind of fs fileContent watcher?
EDIT:
I think the code below describes better what I would like to achieve, however in this code it keeps reading forever and I dont have any variable from fs.read that can help me stop it:
fs.open(filename, 'r', function(err, fd) {
var bufferSize=1000,
chunkSize=512,
buffer=new Buffer(bufferSize),
bytesRead = 0;
while(true){ //check if file has new content inside
fs.read(fd, buffer, 0, chunkSize, bytesRead);
bytesRead+= buffer.length;
}
});
Node has built-in methods in the fs module. It is tagged as unstable, so it can change in the future.
Its called: fs.watchFile(filename[, options], listener)
You can read more about it here: https://nodejs.org/api/fs.html#fs_fs_watchfile_filename_options_listener
But i highly suggest you to use one of the good modules mantained actively like
watchr:
From his readme:
Better file system watching for Node.js. Provides a normalised API the
file watching APIs of different node versions, nested/recursive file
and directory watching, and accurate detailed events for
file/directory changes, deletions and creations.
The module page is here: https://github.com/bevry/watchr
(Used the module in a couple of proyects and working great, im not related to it in other way)
you need store in some data base last size of file.
read filesize first.
load your file.
then make a script to check if file was change.
you can consult the size with jquery.post to obtain your result and decide if need to reload in javascript
I'm trying to delete the file with nodejs fs and I notice that file has been generated then trying to delete (failed to delete the file) while the file is not even uploaded on browser with protractor. Generate and delete file functions are created using nodejs fs.
So how can I put them in a way then wait until file is uploaded then delete the file?
helper.generateFile(filePath);
helper.uploadFile(UploadButtonElement, filePath);
uploadButtonElm.click();
helper.deleteFile(filePath);
Is there a way to execute deleteFile only when below two actions are completed.
helper.uploadFile(UploadButtonElement, filePath);
uploadButtonElm.click();
Thanks.
Protractor operations schedule promises to do things. They do not actually do them. Thus, your helper functions will end up running well before any of the protractor code actually accomplishes what you asked. Use then to chain your dependencies explicitly. Like so:
helper.generateFile(filePath);
helper.uploadFile(UploadButtonElement, filePath);
uploadButtonElm.click().then(function() {
helper.deleteFile(filePath);
});
Please read https://github.com/angular/protractor/blob/master/docs/control-flow.md and https://code.google.com/p/selenium/wiki/WebDriverJs#Understanding_the_API
function deleteAlreadyDownloadedFiles(extension,username) {
let os = require('os');
console.log('USERNAME HERE IS'+require("os").userInfo().username);
var downloadDirectory = '/Users/'+require("os").userInfo().username+'/Downloads/';
const fs = require('fs');
const path = require('path');
const directory = '/Users/'+require("os").userInfo().username+'/Downloads/';
fs.readdir(directory, (err, files) => {
if (err) throw err;
for (const file of files) {
fs.unlink(path.join(directory, file), err => {
if (err
&& (err.code === "EACCES" || err.code === "EPERM")) {
console.log("Retrying rename file: ")
return;
}
});
}
});
}
I'm attempting to return generated files to the front end through Express' res.download function. I'm using chrome, but whenever I call that API that executes the following code all that is returned is the same values returned from the Express res.sendFile() function.
I know that res.download uses res.sendFile, but I would like the download function to actually save to the file system instead of just returning the file in the body of the response.
This is my code.
exports.download = function(req,res) {
var filePath = //somefile that I want to download
res.download(filePath, 'response.txt', function(err) {
throw err;
}
}
I know that the above code at least partly works because I'm getting back, in the response, the contents of the file. However, I want it to be saved onto the file system.
Am I misunderstanding what the download function is supposed to do? Do I just need to take the response data and write it to the file system manually?
res.download adds headers that suggest to the browser that the file should be downloaded rather than opened. However, there's no way to force the browser to do this; it's ultimately the user's choice whether to download a particular file, typically.
If you're triggering this request with AJAX, well, that's not going to cause it to be downloaded, because your JavaScript is requesting that it get the data.
Do I just need to take the response data and write it to the file system manually?
You don't have file system access in browser-side JavaScript. I'm not sure how you intend to do this.
Please find below a sample code in nodejs:
var hello_file = require.resolve('hello')
var hello = require('hello')
console.log(m.hello()); // there is a method hello in module hello.js
delete require.cache[hello_file]
console.log(m.hello()); // it still works
I thought the delete would remove the reference to module and hence the last line should throw an error. But it does not. What could be the reason and what does delete cache really mean?
The cache doesn't know about it anymore but your var hello still has a reference to what was previously loaded.
The next time you call require('hello') it will load the module from the file. But, until you update the reference that var hello is holding, it will continue to point to the originally loaded module.
As you know, node would load a module once even if you require many times, Modules are cached after the first time they are loaded. If you delete it from cache, it will reload the module from filesystem to the cache the next time you require.