I am using node https/http module to download file. The code looks like this:
https.get(url)
.on('response', (response) => {
response.pipe(fs.createWriteStream(dataDir + filename))
}
In the next step I use unzip module to unzip the file.
fs.createReadStream(dataDir + filename)
.pipe(unzip.Extract({path: dataDir}))
.on('close', () => { something }
.on('error', () => { catch error}
For most cases, Everything works well. However, sometimes unzip module returns this error:
Error: invalid signature: 0x6d74683c
The only issue I can notice is the file name. I grab filename from url. It is the last parameter of a url. However, in some cases, the exact filename is different from the one shown in url. For instance:
https://github.com/request/request/archive/master.zip
The name shown in url is master.zip but when I download it the name is request-master.zip.
Is this the actual problem? If so, how can I solve it?
The filename is most likely a symptom rather than the root cause. The unzip module is unmaintained and has many open issues. It is likely to blame and should not be used, regardless.
Try out decompress or yauzl instead. Also, since you are attempting to extract the archive in a stream, you may want to read about how that is not truly possible.
Related
I'm currently building a Node.js application that will eventually be used to convert certain file formats into other formats. Most of the work is being done by the libreoffice-convert library.
I am able to do file conversions without any issues when passing a local file path to the library but it doesn't seem to be working when I grab the contents of a remote file via request() and pass the received body to libreoffice-convert.
This is the relevant code I have right now:
request(fileUrl, {encoding: 'binary'}, function(error, response, body) {
const ext = '.html';
libre.convert(body, ext, undefined, (err, done) => {
if (err) {
console.log(`Error converting file: ${err}`);
res.sendStatus(500);
} else {
console.log(done);
}
});
});
I can see that when I run this, libreoffice starts the conversion but eventually, I'm getting this error:
Error: ENOENT: no such file or directory, open '/var/folders/j9/z_z85kh5501dbslrg53mpjsw0000gn/T/libreofficeConvert_-6529-x08o2o3peLMh/source..html
The example libreoffice-convert code gets the local file using fs.readFileSync() but given that I want to get my contents from a remote file, I'm passing the body received in the request() call.
To be sure that body has the correct contents, I compared the result I receive from fs.readFileSync() to the result I receive from request() when calling for the same exact file locally and remotely. There didn't seem to be any differences at all.
Am I missing something or it's a matter that the libreoffice-convert library or libreoffice itself doesn't support this?
libreoffice-convert is dependent on some linux package, i.e. libreoffice-writer. apt install libreoffice-writer will solve your problem.
I'm using a library called opn. This makes it easier to spawn a child process to open a file in the main process of electron. I could use shell but I need to be able to detect when the file is closed which shell API doesn't provide.
Whenever I open an excel file and there are no other excel files opened, it will work fine and it will wait until I close the file before it resolve the promise. However, if there are other excel files open then it won't wait for the file to be closed and it will resolve the promise immediately.
I did some digging in the source code of the library and the command used to open the file is this for windows: start "" /wait name_of_file where name_of_file is a place holder for the file. The actual command the library executes is different but this one is enough for my use case. I ran into the same problem even when doing it manually without using the library. Maybe I should post this under super user instead of stackoverflow.
Here is the code that opens the file, wait for the file then executes callback.
opn(file, { app: 'EXCEL', wait: true }).then(() => {
const stream = fs.createReadStream(file);
papa.parse(stream, {
delimiter: ',',
complete: (results) => {
mainWindow.webContents.send('uploadNewIndexFileRen', key,
results.data);
},
error: (err) => {
throw err;
}
});
}).catch((error) => { throw error; });
UPDATE:
Here is a quick nodejs repo. All you need to do is clone/fork to check the problem or make a pull request then do npm install to download needed packages. To reproduce the exact problem that I have you might need to be in the following software versions:
Window 10
npm 5.5.1
node v8.9.1
excel 2013
opn 5.2.0 as seen in package.json
This is probably due to the way Excel handles files. I'm guessing Excel handles files with a single process so subsequent opening of files simply signals to the existing process that another files needs opening before closing.
You may be able to force Excel to open a new instance for each file by following these instructions.
Apparently using the /x command line switch will also start a new process:
excel.exe /x
Is there any way to access the Linux root ('/') directory through node? Right now, I have something like this:
multipart = require('connect-multiparty')
app.use(multipart({
uploadDir: config.tmp
}));
...
var file = req.files.file;
fs.renameSync(file.path, ".", function(err) {
if(err) console.error(err.stack);
})
but the problem is that file.path is refering to a folder inside the Linux root rather than the project's root.
The most literal answer to your question
Is there any way to access the Linux root ('/') directory through node?
Is yes, by using /. Node.js shouldn't be giving any special treatment to it. Root is a directory just like any other.
On to your code...
fs.renameSync takes a source first and destination second.
You are renaming a file to ., which represents the current working directory. I'm not even sure if you can rename something to .. I would extract the filename from the path, then set the destination to the root directory plus that file name.
How to access the root directory, which as you said, is /, well, use /.
By the way, why are you using renameSync with a callback and nothing after it? According to the documentation, this is not valid. It's either async with a callback, or sync without a callback. So your callback is probably not firing.
var file = req.files.file;
fs.rename(file.path, '/' + path.basename(file.path), function(err) {
if(err) console.error(err.stack);
});
By the way, I have to advocate strongly against an application writing files to the Linux root directory, for a number of reasons:
Requires root privileges, which opens a can of worms.
Nobody would ever think to look there
The slightest bug could cause irrevocable damage. What if the desired file name is "etc" or "boot"?
There are a million better locations for storing files uploaded from a web server.
I'm trying to upload a file in my node/express app, but everytime I've got the ENOENT error when renaming the file. My code is that:
var tmp_path = req.files.file.path;
fs.rename(tmp_path, target_path, function (err) {
if(err) throw err;
...
});
where target_path will be the destination path. If I do:
console.log('exists ' + fs.existsSync(tmp_path));
then my server logs:
exists true
Also, listing the contents of tmp directory shows that the file is there. What's the problem?
FS methods like fs.rename which create, move or rename files expect that any directories in the path already exist. When they do not, you'll get an ENOENT. Since very often what you mean is "make this file -- and any directories in the path I specify for it" you may want to consider using an NPM library that abstracts access to fs with methods that take care of such things.
There are quite a few options. For example fs-extra is one of the better-tested libraries. Using fs-extra you can use ensureDir in that operation to make the directory structure first if it does not yet exist.
Supervisor is a package for Node.js that monitors files in your app directory for modifications and reloads the app when a modification occurs.
This script interprets symbolic links as regular files and logs out a warning. I would like to fork Supervisor so that either this can be fixed entirely or that a more descriptive warning is produced.
How can I use the File System module of Node.js to determine if a given file is really an symbolic link?
You can use fs.lstat and then call statis.isSymbolicLink() on the fs.Stats object that's passed into your lstat callback.
fs.lstat('myfilename', function(err, stats) {
console.log(stats.isSymbolicLink());
});
Seems like you can use isSymbolicLink()
const files = fs.readdirSync(dir, {encoding: 'utf8', withFileTypes: true});
files.forEach((file) => {
if (file.isSymbolicLink()) {
console.log('found symlink!');
}
}