I am trying to divide a task in node.js onto several cores (using a i5 I have 4 cores available). So far every explanation I found was to cryptic for me (especially the ones talking about servers, which I have no idea of). Can someone show me on the simple example below how I can split the task onto several cores?
Example:
I just want to split the task, so that each core runs one of the loops. How do I do that?
var fs = require('fs');
var greater = fs.createWriteStream('greater.txt');
var smaller = fs.createWriteStream('smaller.txt');
for (var i=0; i<10000; i++){
var input = Math.random()*100;
if (input > 50){
greater.write(input + '\r\n');
}
}
for (var i=0; i<10000; i++){
var input = Math.random()*100;
if (input < 50){
smaller.write(input + '\r\n');
}
}
greater.end();
smaller.end();
Related
My app will require up to 1000 timers at any given moment. Do timers consume a lot of resources? Is it an accepted practice to deploy multiple timers? Or should I avoid it?
By #jfriend00's suggestion, i made a sample check below, this might be not accurate (cuz of dom manipulations), but hope it gives you concept
// let's use 2 measurings, window.performance.now and console.time
// start console.time, usually it gives same result as window.perf.now
// but window.perf.now uses 10 decimal points
console.time('timer_perf_test');
var now1 = window.performance.now();
// our count, this test really lags when 100,000
// CHANGE THIS
var count = 10000;
// some dom elements for text
var elem = document.querySelector('img');
var counter = document.querySelector('#counter');
var perf = document.querySelector('#perf');
// how smooth our image gonna rotate?
var rotate = function(degree){
elem.style.transform = 'rotate(' + degree +'deg)';
counter.textContent = 'timers executed: ' + degree;
}
// create bunch of timers with different timeout
var timer = function(added_time){
setTimeout(function(){
rotate(added_time);
}, 1000 + (added_time * 10));
}
// test begins
for (var i = 0; i < count; i++) {
timer(i)
}
// check results
var now2 = window.performance.now();
perf.textContent = now2 - now1 + ' MS required to create ' + count + ' timers';
console.timeEnd('timer_perf_test');
<img src="https://km.support.apple.com/library/APPLE/APPLECARE_ALLGEOS/HT4211/HT4211-ios7-activity_icon-001-en.png" width="100" height="100">
<p id="counter"></p>
<p id="perf"></p>
My basic setup I have using the cluster module is: (I have 6 cores)
var cluster = require('cluster');
if (cluster.isMaster) {
var numCPUs = require('os').cpus().length;
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
}else{
//Code here
console.time("Time: ");
var obj = {'abcdef' : 1, 'qqq' : 13, '19' : [1, 2, 3, 4]};
for(var i = 0; i < 500000; i++) {
JSON.parse(JSON.stringify(obj));
}
console.timeEnd("Time: ");
}
If I were to run that test.
It will output:
But... if I run that same exact test inside the cluster.isMaster block, it will output:
1) Why is my code being executed multiple times instead of once?
2) Since I have 6 cpu cores helping me run that test, shouldn't it run that code only once but perform the operation faster?
You're forking os.cpus().length separate processes. So if os.cpus().length === 6, then you should see 6 separate outputs (which is the case from the output you've posted).
No, that's not how it works. Each process would be scheduled on a separate core. It's not about running it faster, but being able to do more processing in parallel.
Is there a limit the length of console.log output in Node.js? The following prints numbers up to 56462, then stops. This came up because we were returning datasets from MySQL and the output would just quit after 327k characters.
var out = "";
for (i = 0; i < 100000; i++) {
out += " " + i;
}
console.log(out);
The string itself seems fine, as this returns the last few numbers up to 99999:
console.log(out.substring(out.length - 23));
Returns:
99996 99997 99998 99999
This is using Node v0.6.14.
Have you tried writing that much on a machine with more memory?
According to Node source code console is writing into a stream: https://github.com/joyent/node/blob/cfcb1de130867197cbc9c6012b7e84e08e53d032/lib/console.js#L55
And streams may buffer the data into memory: http://nodejs.org/api/stream.html#stream_writable_write_chunk_encoding_callback
So if you put reeeaally a lot of data into a stream, you may hit the memory ceiling.
I'd recommend you split up your data and feed it into process.stdout.write method, here's an example: http://nodejs.org/api/stream.html#stream_event_drain
I would recommend using output to file when using node > 6.0
const output = fs.createWriteStream('./stdout.log');
const errorOutput = fs.createWriteStream('./stderr.log');
// custom simple logger
const logger = new Console(output, errorOutput);
// use it like console
var count = 5;
logger.log('count: %d', count);
// in stdout.log: count 5
I have a problem with node.js. The commands of the program doesn't load cronologically and i don't know how to do it.
I'm trying to download some images and text from database and send it with packs of 8. But node.js runs for loop and command after loop at the same time.
Here's my code:
socket.on('background_dinamically', function(data){
connection.query("SELECT * FROM products WHERE id='"+data.cathegory+"'" , function(err, rows, fields){
var count = 0;
var array_elements = [];
if(err){
socket.emit('errorserver');
}else{
for (var i = rows.length - 1, count; i >= 0; i-- & count ++) {
array_elements.push(rows[i]);
if (count == 8) {
socket.emit('image_loading_background', [array_elements, data]);
count = 0;
array_elements = [];
}
};
if(count > 0 && count < 8 && count != 0) {
socket.emit('image_loading_background', [array_elements, data]);
}
}
});
});
Marc, first I would check if synchronisation can be done on the client side. If you force your nodejs app to synchronize before sending data to the client, scalability suffers.
If you cannot do without synchronizing on the server side, you can choose between spaghetti code or a sync lib.
Welcome to the world of asynchronous (not chronological) programming. By default, node will work on I/O operations in parallel as you are seeing. To get other behaviors including chronological (in serial), parallel batches, as well as error handling helpers, have a look at one of the many flow control libraries available. Specifically, I recommend caolan/async.
I'm deciding on the best way to store a lot of timeseries data in memory and I made a simple benchmark to compare buffers vs simple arrays:
var buffers = {};
var started = Date.now();
var before = process.memoryUsage().heapUsed;
for (var i = 0; i < 100000; i++) {
buffers[i] = new Buffer(4);
buffers[i].writeFloatLE(i+1.2, 0);
// buffers[i] = [i+1.2];
}
console.log(Date.now() - started, 'ms');
console.log((process.memoryUsage().heapUsed - before) / 1024 / 1024);
And the results are as follows:
Arrays: 22 'ms'
8.391242980957031
Buffers:
123 'ms'
9.9490966796875
So according to this benchmark arrays are 5+ times faster and take 18% less memory. Is this correct? I certainly expected buffers to take less memory.
There's an overhead (in time and space ) for each Buffer you create.
I expect you'll get better space (and maybe time) performance if you compare
buffers[i] = new Buffer(4*1000);
for(k=0;j<1000;++j)
{
buffers[i].writeFloatLE(i+k+1.2, 4*j);
}
With
buffers[i] = [];
for(k=0;j<1000;++j)
{
buffers[i].push(i+k+1.2);
}