I'm trying to determine if a file is already in my system before I download it.
I get the file name from the server through
chrome.downloads.onDeterminingFilename.addListener(function (item, suggest) {
var message = { "fileName": item.filename, "directory": directory };
suggest({
filename: item.filename,
conflict_action: 'prompt',
conflictAction: 'prompt'
});
// conflict_action was renamed to conflictAction in
// https://chromium.googlesource.com/chromium/src/+/f1d784d6938b8fe8e0d257e41b26341992c2552c
// which was first picked up in branch 1580.
});
which works great.
I am checking the filename through native messaging.
chrome.runtime.sendNativeMessage("testcsharp", message, function (response) {
if (chrome.runtime.lastError) {
alert("ERROR: " + chrome.runtime.lastError.message);
} else {
isDownloaded = JSON.parse(JSON.stringify(response)).data;
}
});
The problem is they are both asynchronous but I need to avoid the save as dialog if my file already exists and the only way I've found to do this is call chrome.downloads.cancel(item.id) but there is no guarantee that a response will come in time from native messaging to cancel in ondeterminingfilename. Is there a way to do this or am I just stuck notifying that this file already exists and closing the save dialog manually?
Related
I wrote a function that work like this
onNewZipFileRequested
{get all the necessary data}
.then{download all the files}
.then{create a zipfile with all those file}
.then{upload that zipfile} (*here is the problem)
.than{update the database with the signedUrl of the file}
Here is the relevant code
[***CREATION OF ZIP FILE WORKING****]
}).then(() =>{
zip.generateNodeStream({type:'nodebuffer',streamFiles:true})
.pipe(fs.createWriteStream(tempPath))
.on('finish', function () {
console.log("zip written.");
return bucket.upload(tempPath, { //**** problem****
destination: destinazionePath
});
});
}).then(()=>{
const config = {
action:'read',
expires:'03-09-2391'
}
return bucket.file(destinazionePath).getSignedUrl(config)
}).then(risultato=>{
const daSalvare ={
signedUrl: risultato[0],
status : 'fatto',
dataInserimento : zipball.dataInserimento
}
return event.data.ref.set(daSalvare)
})
On the client side, as soon as the app see the status change and the new Url, a download button (pointing to the new url) appears
Everything is working, but if I try to download the file immediately... there is no file yet!!!
If I wait same time and retry the file is there.
I noted that the time I have to wait depend on the size of the zipfile.
The bucket.upload promise should resolve on the end of the upload, but apparently fires too early.
Is there a way to know exactly when the file is ready?
I may have to make same very big file, it's not a problem if the process takes several minutes, but I need to know when it's over.
* EDIT *
there was a unnecessary nesting in the code. While it was not the error (results are the same before and after refactoring) it was causing some confusion in the answers, so i edited it out.
Id' like to point out that i update the database only after getting the signed url, and i get that only after the upload (i could not otherwise), so to get any result at all the promise chain MUST work, and in fact it does. When on the client side the download button appears (happens when 'status' become 'fatto') it is already linked to the correct signed url, but if i press it too early the file is not there (Failed - No file). If i wait some second (the bigger the file the longer i have to wait) then the file is there.
(English is not my mother language, if i have been unclear ask and i will try to explain myself better)
It looks like the problem could be that the braces are not aligned properly, causing a then statement to be embedded within another. Here is the code with the then statements separated:
[***CREATION OF ZIP FILE WORKING****]}).then(() => {
zip.generateNodeStream({type: 'nodebuffer', streamFiles: true})
.pipe(fs.createWriteStream(tempPath))
.on('finish', function () {
console.log('zip written.')
return bucket.upload(tempPath, {
destination: destinazionePath
})
})
}).then(() => {
const config = {
action: 'read',
expires: '03-09-2391'
}
return bucket.file(destinazionePath).getSignedUrl(config)
}).then(risultato => {
const daSalvare = {
signedUrl: risultato[0],
status : 'fatto',
dataInserimento : zipball.dataInserimento
}
return event.data.ref.set(daSalvare)
})
I'm building an app for windows using Electron. To package and distribute it I'm using electron-builder. Electron-builder relies on many packages, and for auto-updates it uses Squirrel-windows.
I've been battling with auto-update on Windows for 3 days and at the end I've come up with a working solution that seems to give no problems.
I wont' go into the details of what I've tried, and failed. Instead, I'll post here the solution with which I've come up.
I'm sharing it with you guys, to see if you may point out to me any flaws that will make my system fail, or, if it truly is a solid solution, to help those who are struggling as I was. For this latter reason, I'm posting some more code than it would be necessary, hoping it will help others.
The logic is as follows:
if the sub-folder fullupdate inside the path of the current executable does not exists (see later, it will be clarified), we connect with an online server and check if there is an update by sending the current app version;
if there is no update, do nothing.
if there is an update, we instruct the server to return a json string that contains the url from which we can download of the .exe installer produced by electron-builder. NB: not the .nupkg (server code not provided :-)).
we download the file and save it inside a sub folder fullupdate in the local folder in which our app is currently saved. This should be "safe" as electron-builder saves the app in the current user folder AppData, so we should not have permissions issues.
at the end of the download, we create a new file update inside the folder fullupdate to be sure the download has finished successfully. We could also rename the file, but I prefer this way.
next time the app opens:
if the folder fullupdate exists we check if the file update exists. If it does not exists, the download was not finished, so we delete the folder fullupdate and call the remote server again to start all over again.
else, if the file update exists, we launch the .exe file we have downloaded, and return true. This will prevent the app from opening the main window. The cool thing is that the updater will delete the whole old version of the app saved in AppData (while leaving local user data) and replace it with the new version. In this way we will get rid also of the folder fullupdate.
Now the code:
// we want to run this only on windows
var handleStartupEvent = function() {
if (process.platform !== 'win32') {
return false;
}
/////////////////
// MANUAL UPDATER
/////////////////
var appFolder = 'app-' + appVersion;
var pathApp = path.dirname(process.execPath);
var pathUpdate = pathApp + '\\fullupdate';
var checkupdateurl = 'https://api.mysite.com/getjson/' + appVersion.split('.').join('-');
function checkIfDownloaded(){
if (!fs.existsSync(pathUpdate)) checkUpdate();
else return checkIfInstallLocal();
}
function checkIfInstallLocal(){
if(fileExists('fullupdate\\update')) return installLocal();
else {
deleteFolderRecursive(pathUpdate);
checkUpdate();
}
}
function installLocal(){
cp.exec('fullupdate\\Update.exe', function( error, stdout, stderr){
if ( error != null ) {
console.log(stderr);
}
});
return true;
}
// from http://www.geedew.com/remove-a-directory-that-is-not-empty-in-nodejs/
var deleteFolderRecursive = function(path) {
if( fs.existsSync(path) ) {
fs.readdirSync(path).forEach(function(file,index){
var curPath = path + "/" + file;
if(fs.lstatSync(curPath).isDirectory()) deleteFolderRecursive(curPath);
else fs.unlinkSync(curPath);
});
fs.rmdirSync(path);
}
};
// from http://stackoverflow.com/questions/4482686/check-synchronously-if-file-directory-exists-in-node-js
function fileExists(path) {
try {
return fs.statSync(path).isFile();
}
catch (e) {
if (e.code == 'ENOENT') { // no such file or directory. File really does not exist
return false;
}
throw e; // something else went wrong, we don't have rights, ...
}
}
function checkUpdate(){
https.get('https://api.mysite.com/getjson/' + app.getVersion().split('.').join('-'), (res) => {
res.setEncoding('utf8');
res.on('data', function(chunk) {
if(chunk) thereIsUpdate(chunk);
});
}).on('error', (e) => {
console.log(e);
});
}
function thereIsUpdate(chunk){
var data = JSON.parse(chunk);
if(data && data.url) getNewUpdate(data.urlsetup);
}
function getNewUpdate(url){
fs.mkdirSync(pathUpdate);
var file = fs.createWriteStream(pathUpdate + '/Update.exe');
var responseSent = false; // flag to make sure that response is sent only once.
var request = https.get(url, function(response) {
response.pipe(file);
file.on('finish', () =>{
file.close(() => {
if(responseSent) return;
responseSent = true;
});
fs.closeSync(fs.openSync(pathUpdate + '/update', 'w'));
});
});
}
if(checkIfDownloaded()) return true;
/////////////////////////
// SQUIRREL EVENTS HANDLER
//////////////////////////
// see http://stackoverflow.com/questions/30105150/handle-squirrels-event-on-an-electron-app
};
// here we call the function. It is before the opening of the window, so that we prevent the opening if we are updating, or if there is a Squirrel event going on (see SO question, link above)
if (handleStartupEvent()) {
return;
}
My nodejs application is built around a "project file".
Several modules (by "module", I mean a simple javascript file of my project) of this application need to load, modify and save this file often, via streams (fs.createReadStream, fs.createWriteStream), and since those modules are executed independently from each other, with sometimes an origin from an websocket events (for instance), I need to make the save/load operations of the project file atomic.
It means the following scenario:
moduleA writes the project file
in the same time, and before moduleA has finished to write the file, moduleB wants to read it => ideally, it should wait for the write operation of moduleA (currently, it reads a partially written file and detect an error) before really read the file
Is nodejs able to do this natively or do I have to build a sort of atomic wrapper over my read/write stream system?
There is to my knowledge nothing built in. There are modules such as redis-lock though, that implement a lock mechanism.
If you run on a single non-clustered server you could probably cope with implementing a simple local lock though.
This might give you an idea:
var Fs = require("fs"),
LOCK = require ("os").tmpdir () + '/foo-lock.';
function transLock(id, cb) {
Fs.open(LOCK + id, "wx", function(err, fd) {
if (err) {
// someone else has created the file
// or something went wrong
cb(err);
} else {
Fs.close(fd, function(err) {
// there should be no error here except weird stuff
// like EINTR which must be ignored on Linux
cb();
});
}
});
}
function transUnlock(id) {
Fs.unlink(LOCK + id, function(err) {
if (err) {
// something is wrong and nothing we can do except
// perhaps log something or do some background cleanup
}
});
}
function main() {
var id = "some-unique-name";
transLock(id, function(err) {
if (err)
console.log(err);
else {
// ... do your stuffs ...
transUnlock(id);
}
});
}
main();
which is the most elegant way or technology to let a node.js server know if a file is created on a server?
The idea is: a new image has been created (from a webcam or so) -> dispatch an event!
UPDATE: The name of the new file in the directory is not known a priori and the file is generated by an external software.
You should take a look at fs.watch(). It allows you to "watch" a file or directory and receive events when things change.
Note: The documentation states that fs.watch is not consistent across platforms, so you should take that in to account before using it.
fs.watch(fileOrDirectoryPath, function(event, filename) {
// Something changed with filename, trigger event appropriately
});
Also something to be aware of from the docs:
Providing filename argument in the callback is not supported on every
platform (currently it's only supported on Linux and Windows). Even on
supported platforms filename is not always guaranteed to be provided.
Therefore, don't assume that filename argument is always provided in
the callback, and have some fallback logic if it is null.
If filename is not available on your platform and you're watching a directory you may need to do something where you initially read the directory and cache the list of files in it. Then, when you get an event from fs.watch, read the directory again and compare it to the cached list of files to see what was added (if anything).
Update 1: There's a good module called watch, on github, which makes it easy to watch a directory for new files.
Update 2: I threw together an example of how to use fs.watch to get notified when new files are added to a directory. I think the module I linked to above is probably the better way to go, but I thought it would be nice to have a basic example of how it might work if you were to do it yourself.
Note: This is a fairly simplistic example just to show how it could work in general. It could almost certainly be done more efficiently and it's far from throughly tested.
function watchForNewFiles(directory, callback) {
// Get a list of all the files in the directory
fs.readdir(directory, function(err, files) {
if (err) {
callback(err);
} else {
var originalFiles = files;
// Start watching the directory for new events
var watcher = fs.watch(directory, function(event, filename) {
// Get the updated list of all the files in the directory
fs.readdir(directory, function(err, files) {
if (err) {
callback(err);
} else {
// Filter out any files we already knew about
var newFiles = files.filter(function(f) {
return (originalFiles.indexOf(f) < 0);
});
// Reset our list of "original" files
originalFiles = files;
// If there are new files detected, call the callback
if (newFiles.length) {
callback(null, newFiles);
}
}
})
});
}
});
}
Then, to watch a directory you'd call it with:
watchForNewFiles(someDirectoryPath, function(err, files) {
if (err) {
// handle error
} else {
// handle any newly added files
// "files" is an array of filenames that have been added to the directory
}
});
I came up with my own solution using this code here:
var fs = require('fs');
var intID = setInterval(check,1000);
function check() {
fs.exists('file.txt', function check(exists) {
if (exists) {
console.log("Created!");
clearInterval(intID);
}
});
}
You could add a parameter to the check function with the name of the file and call it in the path.
I did some tests on fs.watch() and it does not work if the file is not created. fs.watch() has multiple issues anyways and I would never suggest using it... It does work to check if the file was deleted though...
We have a buffer we'd like to write to a file. If the file already exists, we need to increment an index on it, and try again. Is there a way to create a file only if it doesn't exist, or should I just stat files until I get an error to find one that doesn't exist already?
For example, I have files a_1.jpg and a_2.jpg. I'd like my method to try creating a_1.jpg and a_2.jpg, and fail, and finally successfully create a_3.jpg.
The ideal method would look something like this:
fs.writeFile(path, data, { overwrite: false }, function (err) {
if (err) throw err;
console.log('It\'s saved!');
});
or like this:
fs.createWriteStream(path, { overwrite: false });
Does anything like this exist in node's fs library?
EDIT: My question isn't if there's a separate function that checks for existence. It's this: is there a way to create a file if it doesn't exist, in a single file system call?
As your intuition correctly guessed, the naive solution with a pair of exists / writeFile calls is wrong. Asynchronous code runs in unpredictable ways. And in given case it is
Is there a file a.txt? — No.
(File a.txt gets created by another program)
Write to a.txt if it's possible. — Okay.
But yes, we can do that in a single call. We're working with file system so it's a good idea to read developer manual on fs. And hey, here's an interesting part.
'w' - Open file for writing. The file is created (if it does not
exist) or truncated (if it exists).
'wx' - Like 'w' but fails if path exists.
So all we have to do is just add wx to the fs.open call. But hey, we don't like fopen-like IO. Let's read on fs.writeFile a bit more.
fs.readFile(filename[, options], callback)#
filename String
options Object
encoding String | Null default = null
flag String default = 'r'
callback Function
That options.flag looks promising. So we try
fs.writeFile(path, data, { flag: 'wx' }, function (err) {
if (err) throw err;
console.log("It's saved!");
});
And it works perfectly for a single write. I guess this code will fail in some more bizarre ways yet if you try to solve your task with it. You have an atomary "check for a_#.jpg existence, and write there if it's empty" operation, but all the other fs state is not locked, and a_1.jpg file may spontaneously disappear while you're already checking a_5.jpg. Most* file systems are no ACID databases, and the fact that you're able to do at least some atomic operations is miraculous. It's very likely that wx code won't work on some platform. So for the sake of your sanity, use database, finally.
Some more info for the suffering
Imagine we're writing something like memoize-fs that caches results of function calls to the file system to save us some network/cpu time. Could we open the file for reading if it exists, and for writing if it doesn't, all in the single call? Let's take a funny look on those flags. After a while of mental exercises we can see that a+ does what we want: if the file doesn't exist, it creates one and opens it both for reading and writing, and if the file exists it does so without clearing the file (as w+ would). But now we cannot use it neither in (smth)File, nor in create(Smth)Stream functions. And that seems like a missing feature.
So feel free to file it as a feature request (or even a bug) to Node.js github, as lack of atomic asynchronous file system API is a drawback of Node. Though don't expect changes any time soon.
Edit. I would like to link to articles by Linus and by Dan Luu on why exactly you don't want to do anything smart with your fs calls, because the claim was left mostly not based on anything.
What about using the a option?
According to the docs:
'a+' - Open file for reading and appending. The file is created if it does not exist.
It seems to work perfectly with createWriteStream
This method is no longer recommended. fs.exists is deprecated. See comments.
Here are some options:
1) Have 2 "fs" calls. The first one is the "fs.exists" call, and the second is "fs.write / read, etc"
//checks if the file exists.
//If it does, it just calls back.
//If it doesn't, then the file is created.
function checkForFile(fileName,callback)
{
fs.exists(fileName, function (exists) {
if(exists)
{
callback();
}else
{
fs.writeFile(fileName, {flag: 'wx'}, function (err, data)
{
callback();
})
}
});
}
function writeToFile()
{
checkForFile("file.dat",function()
{
//It is now safe to write/read to file.dat
fs.readFile("file.dat", function (err,data)
{
//do stuff
});
});
}
2) Or Create an empty file first:
--- Sync:
//If you want to force the file to be empty then you want to use the 'w' flag:
var fd = fs.openSync(filepath, 'w');
//That will truncate the file if it exists and create it if it doesn't.
//Wrap it in an fs.closeSync call if you don't need the file descriptor it returns.
fs.closeSync(fs.openSync(filepath, 'w'));
--- ASync:
var fs = require("fs");
fs.open(path, "wx", function (err, fd) {
// handle error
fs.close(fd, function (err) {
// handle error
});
});
3) Or use "touch": https://github.com/isaacs/node-touch
Todo this in a single system call you can use the fs-extra npm module.
After this the file will have been created as well as the directory it is to be placed in.
const fs = require('fs-extra');
const file = '/tmp/this/path/does/not/exist/file.txt'
fs.ensureFile(file, err => {
console.log(err) // => null
});
Another way is to use ensureFileSync which will do the same thing but synchronous.
const fs = require('fs-extra');
const file = '/tmp/this/path/does/not/exist/file.txt'
fs.ensureFileSync(file)
With async / await and Typescript I would do:
import * as fs from 'fs'
async function upsertFile(name: string) {
try {
// try to read file
await fs.promises.readFile(name)
} catch (error) {
// create empty file, because it wasn't found
await fs.promises.writeFile(name, '')
}
}
Here's a synchronous way of doing it:
try {
await fs.truncateSync(filepath, 0);
} catch (err) {
await fs.writeFileSync(filepath, "", { flag: "wx" });
}
If the file exists it will get truncated, otherwise it gets created if an error is raised.
This works for me.
// Use the file system fs promises
const {access} = require('fs/promises');
// File Exist returns true
// dont use exists which is no more!
const fexists =async (path)=> {
try {
await access(path);
return true;
} catch {
return false;
}
}
// Wrapper for your main program
async function mainapp(){
if( await fexists("./users.json")){
console.log("File is here");
} else {
console.log("File not here -so make one");
}
}
// run your program
mainapp();
Just keep eye on your async - awaits so everthing plays nice.
hope this helps.
You can do something like this:
function writeFile(i){
var i = i || 0;
var fileName = 'a_' + i + '.jpg';
fs.exists(fileName, function (exists) {
if(exists){
writeFile(++i);
} else {
fs.writeFile(fileName);
}
});
}