I am working on upload module of my server and I set file uploads with multiparty. I am currently trying to limit the upload size simply i a doing something like this
req.on("data", function(dt) {
bytes += dt.length;
if (bytes > 2048) {
req.connection.destroy();
console.log("connection destroyed due to huge file size");
}
console.log(bytes);
});
I thought this length is in bytes and tried to limit it with 2mb
but i noticed this unit is a bit strange for testing i uploaded a 148 kb file but the length of the variable i created so far is 421 it is neither in bits nor bytes why it is so strange number? where do this extra ~300k come from?
Did you try filesystem module for checking size of the file?
E.g.
var fs = require("fs");
var stats = fs.statSync("myfile.txt");
var fileSizeInBytes = stats.size;
Related
I'm reading from a fs readStream (from graphql upload file) and piping it through SharpJS transform function and piping it through a writeStream to write to a file.
In my system (windows) it works just fine but in my Host (linux) it creates a core dump file with 500mb size and the images that it creates are with size of 0 kb.
const transform = (dimen) => sharp().resize(dimen, dimen)
const fs512 = fs.createWriteStream(addSuffix(filePath))
const fs256 = fs.createWriteStream(addSuffix(filePath))
const fs128 = fs.createWriteStream(addSuffix(filePath))
await stream.pipe(transform(512)).pipe(fs512)
await stream.pipe(transform(256)).pipe(fs256)
await stream.pipe(transform(128)).pipe(fs128)
I have tried listening on finish event and closing them but it didn't work.
I think the issue is because of SharpJS, if I remove the first pipe, it works (without resizing).
What are the effective ways to work Node js with a large JSON file of 600 MB and more?
My partner gives me from his REST API wery large JSON file. 600mb, 1000mb
Its structure is as follows
{ nameid1:[list id,....], nameid2:[list id,....], }
[list id,....] - An array with ID can be up to hundreds of millions of records.
Now to work with such files I use the following sequence of actions.
I save it to hard drive
With the sed command, from a single-line file, I make it multi-line
Example
exec (`sed -i 's /', '/', '\ n / g' file.json)
I work directly with the file using readline
I tried to use JSONStream but it causes FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
function getStream() {
let jsonData = __dirname + '/jsonlarge/file.json',
stream = fs.createReadStream(jsonData, {
encoding: 'utf8'
})
parser = JSONStream.parse('*');
stream.pipe(parser)
parser.on('data', (data) => {
console.log('received:', data);
});
}
Example structure json file
{"Work":"12122001","name":"Regist","world":[{"name":"000000","point":"rfg","Content":["3202b9a3fba","121323","2343454","45345543","354534534"]}, {"name":"000000","point":"rfg","Content":["3202b","121323","2343454","45345543","354534534"]}, {"name":"000000","point":"rfg","Content":["320","121323","2343454","45345543","354534534"]}]}
Maybe someone knows a faster way to work with such files.
Thanks
I have a very large, binary file (>25 GB), and I need to very quickly read a small range of bytes from it at a specific offset. How can I accomplish this in Node.js in an efficient way?
A fairly minimal example of what you want, refer to https://nodejs.org/api/all.html#fs_fs_createreadstream_path_options for more details
const fs = require("fs");
const stream = fs.createReadStream("test.txt", { start: 1, end: 5 });
stream.on("data", chunk => console.log(chunk.toString()));
Provided you have a file called test.txt of course...
I want to read some stdout of some process with nodejs, in Windows.
So far i got this:
I simulate the process output with :
ping -t google.com > ping.txt
The ping output is dumped in the ping.txt file.
That file is watched in node for changes.
So, in nodejs i got this code:
var filename = 'ping.txt';
var fs = require("fs");
fs.watch(filename, function (event, filename) {
if (event == 'change') {
console.log('change', filename);
fs.stat(filename, function (err, stats) {
console.log('size ', stats.size);
if(stats.size > 500){
fs.truncate(filename,10,function(){
console.log("truncated");
});
}
})
}
});
But this is my nodejs output:
size 0
change ping.txt
size 192
change ping.txt
size 253
change ping.txt
size 375
change ping.txt
size 436
change ping.txt
size 559
truncated
change ping.txt
size 620
truncated
change ping.txt
size 743
truncated
change ping.txt
size 804
truncated
change ping.txt
size 926
truncated
The file never gets truncated.
I don't want to get a massive file size, because the real process is getting a lot of output.
So that's the main reason to try to get the file truncated.
But it's not working.
Can anyone give me a hand? This is my first nodejs experiment.
Later i'm planning to output that stdout process trought a websocket, but now i'm stucked with this problem.
Thanks in advance ! Best regards!
Edit 1:
The ping process is not the real one that i'm trying to read. The real process is a cryptocoin miner. And it's very resource hungry.
So that's because i took this aproach. To dump the process output in a file and read it with another process.
I'm not 100% happy to have nodejs managing that cryptocoin process, because i don't know if node can handle it.
If is there a better way to do this, i'm very happy to ear it. Thanks!
I did a small snippet using a npm package rotating-file-stream and it seems to work:
// stream.js
const rfs = require('rotating-file-stream');
const stream = rfs('ping.txt', {
size: '500B', // rotate every 500 Bytes written
});
const stdin = process.openStdin();
stdin.pipe(stream);
Run the command in terminal ping google.com | node stream.js
Is there a way to tell require that if file name ends with .jpg then it should return base64 encoded version of it?
var image = require('./logo.jpg');
console.log(image); // data:image/jpg;base64,/9j/4AAQSkZJRgABAgA...
I worry about the "why", but here is "how":
var Module = require('module');
var fs = require('fs');
Module._extensions['.jpg'] = function(module, fn) {
var base64 = fs.readFileSync(fn).toString('base64');
module._compile('module.exports="data:image/jpg;base64,' + base64 + '"', fn);
};
var image = require('./logo.jpg');
There's some serious issues with this mechanism: for one, the data for each image that you load this way will be kept in memory until your app stops (so it's not useful for loading lots of images), and because of that caching mechanism (which also applies to regular use of require()), you can only load an image into the cache once (requiring an image a second time, after its file has changed, will still yield the first—cached—version, unless you manually start cleaning the module cache).
In other words: you don't really want this.
You can use fs.createReadStream("/path/to/file")